Author Topic: GDAL Conversion Tips  (Read 32105 times)

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1143
    • http://webgis.wr.usgs.gov
    • Email
GDAL Conversion Tips
« on: August 03, 2009, 12:48:36 PM »
GDAL. This open source raster library (used in many apps) has some support for map projected PDS v3, ISIS2, and ISIS3 reading. It can export raw or tons of other formats. It maintains projection information and some metadata to output formats that allow it. The GDAL utility applications have stretching (scaling options), map projection support, clipping, resampling and mosaic capabilities. There are binaries available for Mac, Windows, and Linux, although, I prefer to build the latest version on linux myself. For windows you will want FWTOOLS or OSGeo4W. For Mac you will want to install the GDAL and related Frameworks from Kyng Chaos.

Installing

prebuilt GDAL (Windows):
http://fwtools.maptools.org/ (getting outdated) or http://wiki.osgeo.org/wiki/OSGeo_Win32_Installer (latest)

Linux (FWtools experimental): http://fwtools.maptools.org/linux-experimental.html
update (Oct 2013): USGS 64bit version with Jpeg2000 support: ftp://pdsimage2.wr.usgs.gov/pub/pigpen/GDAL/FWTools-linux-x86_64-3.0.6f-usgs.tar.gz

Mac binary Frameworks: http://www.kyngchaos.com/software:frameworks
   *Mac notes: install bundled frameworks (just open and click). Also get ECW framework for Jpeg2000 support. Add this line your Csh shell .chsrc (if not found create /Users/YourName/.cshrc ). If you use bash you will need to edit your .profile and the path statement will be defined a little differently).
        set path = ( $path /Library/Frameworks/GDAL.framework/Programs/ )


for the main gdal_translate help: http://gdal.org/gdal_translate.html

to see what formats are currently supported:
>gdal_translate --formats

convert input bit type = same output type in Tiff (ISIS2/3 cub example ):
>gdal_translate -of GTiff input.cub output.tif
    - GTiff is the default format so you can actually skip "-of GTiff" in this case

for Surfer (binary - maintains null data)
>gdal_translate -of GSBG input.cub output.grd
to resize for files too large for Surfer:
>gdal_translate -of GSBG -outsize 50% 50% input.cub output.grd

16bit Tiff (truncate floats):
>gdal_translate -of GTiff -ot Int16 input.cub output.tif

16bit UInt Tiff (truncate negative and floats):
gdal_translate -ot UInt16 -of GTiff -a_nodata 0 input.cub output.tif

scaled 16bit Uint Tiff so use (change pixel values across 16bit range):
gdal_translate -ot UInt16 -of GTiff -a_nodata 0 -scale 767 2296 1 65536 moon11s308_8.cub moon.tif
--output range has been scaled to: 1 65536

output raw (ENVI compatible):
>gdal_translate -of ENVI  input.cub out.raw
        -This will create a ENVI header (*.hdr) with parameters spelled out

output lossless geoJpeg2000
>gdal_translate -of JP2KAK -co quality=100 input.cub out.jp2
--best if convert to 8bit in ISIS or other before conversion (use scale flag shown above to stretch yourself)
update: see next thread to help with the conversion from 32bit cube to an 8bit cube prior to a GDAL format conversion.

output lossless geoJpeg2000 using gdal to create GeoTiff and free kakadu_binaries (even 16bit compatible for DEMs - great for MOLA, LOLA)
>gdal_translate -of GTIFF input.cub temp_geo.tif
>kdu_compress -i temp_geo.tif -o out_geo.jp2 Creversible=yes Clayers=16 -num_threads 2
Kakadu binaries for download: http://www.kakadusoftware.com/index.php?option=com_content&task=view&id=26&Itemid=22

stretch to PNG 8bits and save out PNG worldfile (recommended):
>gdal_translate -of PNG -scale -co worldfile=yes input.cub out.png

stretch to JPG 8bits and save out JPG worldfile (recommended):
>gdal_translate -of JPEG -scale -co worldfile=yes input.cub out.jpg

manual stretch to PNG 8bits (no GIS worldfile):
>gdal_translate -of PNG -scale in_min in_max out_min out_max input.cub out.png
  *notes I use output as 1 to 255 so that 0 is maintained as NULL - example:
>gdal_translate -of PNG -scale  0.0232  0.12  1  255  -a_nodata 0 input.cub out.png

scale (resize) to PNG 8bits at 10% original size for browse image:
>gdal_translate -of PNG -scale -size 10% 10% input.cub out_browse.png

scale (resize) to JPG 8bits at 100x200 pixels for web image (aspect not maintained):
>gdal_translate -of JPEG -scale -size 100 200 input.cub out100x200.jpg

ESRI ASCII Grid format:
>gdal_translate -of AAGrid input.cub output.asc

XYZ format (script may not be available for all binary releases)
>gdal2xyz.py input.cub out.xyz
or a new output type (GDAL v1.8+ ):
>gdal_translate -of XYZ input.cub output.xyz

information about image:
>gdalinfo input.cub

stats about image:
>gdalinfo -stats input.cub

min/max range:
>gdalinfo -mm input.cub

To project to new projection use the program gdalwarp.

----simple example Csh Script to force output scale to 1 to 255.
mag{90}> more /usgs/cdev/contrib/bin/to8bit_gdal_tif.csh
Code: [Select]
#!/bin/csh
#usage: to8bit_gdal_tif.csh input.ext output.tif
#Note the use of $3 in gdal_translate which just allows you to add in other options. By default it will just be blank.
set in=$1
set out=$2

set stats = `gdalinfo -stats $in > /tmp/xxxtemp_stats.junk`
set min = `cat /tmp/xxxtemp_stats.junk | grep MINIMUM | awk -F= '{print $2}' | sed 's/ //g'`
set max = `cat /tmp/xxxtemp_stats.junk | grep MAXIMUM | awk -F= '{print $2}' | sed 's/ //g'`

gdal_translate -ot byte -of GTIFF -co compress=lzw -ot Byte $3 -a_nodata 0 -scale $min $max 1 255 $in $out
/bin/rm -f /tmp/xxxtemp_stats.junk

related:
http://isis.astrogeology.usgs.gov/IsisSupport/viewtopic.php?t=372

-Trent
« Last Edit: October 24, 2013, 10:37:56 AM by thare »

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1143
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #1 on: September 08, 2009, 11:06:49 AM »
update: ISIS 3.3.x now has a percentage option in "stretch" and a "bit2bit" program which can help with the conversion from 32bit cube to an 8bit cube. Although don't use bit2bit just yet until clipped values are handled differently.

ISIS3 tips to get 8bit:

If you notice above I state to use ISIS3 to get to an 8bit range since it can give you more options. Here are some directions which should get easier as I think a specific program for ISIS3 is coming (see update above).

So to review, you can simply apply a linear stretch in GDAL:
>gdalinfo -mm input.cub  --> this will give you "in_min" and "in_max" values for the cube
now run gdal translate with
>gdal_translate -of JP2KAK -ot byte -co quality=100 -a_nodata 0 -scale in_min in_max 1 255 input.cub out.jp2

---------------------

OR for ISIS you can run stretch.

Stretch method 1
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs="0:1 100:254" null=0 lis=0 lrs=0 his=255 hrs=255

This allows you to specify input percentages for the mapping pairs. Thus when USEPERCENTAGES=true is set pairs="0:1 100:254" means:
map 0% to 1 (or the file's min value to 1) and 100% to 254 (file's max value).

Stretch method 2
This also means you can apply a recommended 0.5% clip to remove the potential extraneous lows and highs like:
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs="0:1 0.5:1 99.5:254 100:254" null=0 lis=0 lrs=0 his=255 hrs=255

-Trent
« Last Edit: January 11, 2013, 09:59:09 AM by thare »

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1143
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #2 on: November 23, 2009, 08:00:08 AM »
There have been changes to the command-line parser for ISIS3. It appears at least for tcsh you don't need the backslash escape characters. But it looks like there was another change for the general "to" parameter. If you choose a lower bit type it seems it forces you to add in a output range there... I will alert the the ISIS team to this issue.

So if you redundantly specify the stretch pair it seems to work. This makes sense for other programs but not for stretch. Maybe this means we need an independent program to convert types and will help the user with the in_min and in_max values. It would be good to have a few stretches to pick from (e.g. linear, std dev (#), log, guassian, etc.).

so try this - notice the output range or stretch pair is also on the "to" parameter:
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 pairs="0.092769:1 0.18348:254" null=0 lis=1 lrs=0 his=255 hrs=255

-Trent
« Last Edit: January 11, 2013, 09:57:25 AM by thare »

tscharff

  • Horus (New Member)
  • *
  • Posts: 2
Re: GDAL Conversion Tips
« Reply #3 on: November 24, 2009, 01:21:12 AM »
Confirmed: It works in bash with on a real image with non-escaped quotes and the range specifier appended to the "to" argument.

Many thanks for you help, Trent.  I would never have figured this out on my own.

Ted.

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1143
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #4 on: October 11, 2011, 05:33:53 PM »
Here is another tutorial called, Basic conversion tutorial for planetary data. Mostly for ISIS to ArcMap 10 but other good information in there for other apps too.
http://webgis.wr.usgs.gov/pigwad/tutorials/arcgis10.Basic_converting_data.pdf
parent page: http://webgis.wr.usgs.gov/pigwad/tutorials/planetarygis

from: https://isis.astrogeology.usgs.gov/IsisSupport/index.php/topic,3310.msg12914.html

-Trent
« Last Edit: October 11, 2011, 05:36:16 PM by thare »

wfarrand

  • Osiris (Active Member)
  • **
  • Posts: 12
    • Email
Re: GDAL Conversion Tips
« Reply #5 on: October 24, 2011, 08:13:27 AM »
OK, I was able to get the Jpeg2000 file generated with isis2std and downloaded isis3world.pl and got an appropriate world file and *.prj file generated so that I could read the CTX image into ArcMap just fine.  I'm completely unfamiliar with the GDAL stuff, but I might have to look into it.  Thanks!

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1143
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #6 on: December 06, 2012, 03:37:16 PM »
Just an update for this thread with my currently favorite method to convert to 8bit (before using gdal to convert to geoTiff or GeoJpeg2000 or other format). I can't recommend the ISIS program bit2bit since it pushes valid data which is clipped into NoDATA (not good for viewing the image).

Stretch method 1
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs="0:1 100:254" null=0 lis=1 lrs=1 his=255 hrs=255

This allows you to specify input percentages for the mapping pairs. Thus when USEPERCENTAGES=true is set pairs="0:1 100:254" means:
map 0% to 1 (or the file's min value to 1) and 100% to 254 (file's max value).

Stretch method 2
This also means you can apply a recommended 0.5% clip to remove the potential extraneous lows and highs like:
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs="0:1 0.5:1 99.5:254 100:254" null=0 lis=1 lrs=1 his=255 hrs=255

-Trent

Note: that all the special pixel values are getting mapped to 0 or 255.

If there is speckle in the shadows of your image it might be from LIS or LRS pixels.  You can try to map those to 1 instead of 0 (e.g. lis=1 lrs=1). But Nulls should always be mapped to 0.
more see: http://isis.astrogeology.usgs.gov/IsisWorkshop/index.php/Special_Pixels#How_many_special_pixel_values_exist_in_Isis.3F



« Last Edit: December 28, 2013, 11:21:25 AM by thare »

Valentina

  • ☿ PhD Student
  • Isis User Group
  • Osiris (Active Member)
  • **
  • Posts: 10
  • Studying the tectonics of Mercury
Re: GDAL Conversion Tips
« Reply #7 on: April 17, 2014, 08:34:23 AM »
Dear Trent,

What happens if I stretch a multi band image?
With monochrome images I usually use the stretch command in ISIS3 as you suggested and it works perfectly. When I tried to do the same with a multiband image two of the three bands resulted like being "overexposed" and the final GDAL converted image had a funky color (oversaturated cyan). Should I stretch each band separately in some way?

Thank you.
V