Author Topic: GDAL Conversion Tips  (Read 41414 times)

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
GDAL Conversion Tips
« on: August 03, 2009, 12:48:36 PM »
GDAL. This open source raster library (used in many apps) has some support for map projected PDS v3, ISIS2, and ISIS3 reading. It can export raw or tons of other formats. It maintains projection information and some metadata to output formats that allow it. The GDAL utility applications have stretching (scaling options), map projection support, clipping, resampling and mosaic capabilities. There are binaries available for Mac, Windows, and Linux, although, I prefer to build the latest version on linux myself. For windows you will want FWTOOLS or OSGeo4W. For Mac you will want to install the GDAL and related Frameworks from Kyng Chaos.

Installing

prebuilt GDAL (Windows):
http://fwtools.maptools.org/ (getting outdated) or http://wiki.osgeo.org/wiki/OSGeo_Win32_Installer (latest)

Linux (FWtools experimental): http://fwtools.maptools.org/linux-experimental.html
update (Oct 2013): USGS 64bit version with Jpeg2000 support: ftp://pdsimage2.wr.usgs.gov/pub/pigpen/GDAL/FWTools-linux-x86_64-3.0.6f-usgs.tar.gz

Mac binary Frameworks: http://www.kyngchaos.com/software:frameworks
   *Mac notes: install bundled frameworks (just open and click). Also get ECW framework for Jpeg2000 support. Add this line your Csh shell .chsrc (if not found create /Users/YourName/.cshrc ). If you use bash you will need to edit your .profile and the path statement will be defined a little differently).
        set path = ( $path /Library/Frameworks/GDAL.framework/Programs/ )


for the main gdal_translate help: http://gdal.org/gdal_translate.html

to see what formats are currently supported:
>gdal_translate --formats

convert input bit type = same output type in Tiff (ISIS2/3 cub example ):
>gdal_translate -of GTiff input.cub output.tif
    - GTiff is the default format so you can actually skip "-of GTiff" in this case

for Surfer (binary - maintains null data)
>gdal_translate -of GSBG input.cub output.grd
to resize for files too large for Surfer:
>gdal_translate -of GSBG -outsize 50% 50% input.cub output.grd

16bit Tiff (truncate floats):
>gdal_translate -of GTiff -ot Int16 input.cub output.tif

16bit UInt Tiff (truncate negative and floats):
gdal_translate -ot UInt16 -of GTiff -a_nodata 0 input.cub output.tif

scaled 16bit Uint Tiff so use (change pixel values across 16bit range):
gdal_translate -ot UInt16 -of GTiff -a_nodata 0 -scale 767 2296 1 65536 moon11s308_8.cub moon.tif
--output range has been scaled to: 1 65536

output raw (ENVI compatible):
>gdal_translate -of ENVI  input.cub out.raw
        -This will create a ENVI header (*.hdr) with parameters spelled out

output lossless geoJpeg2000
>gdal_translate -of JP2KAK -co quality=100 input.cub out.jp2
--best if convert to 8bit in ISIS or other before conversion (use scale flag shown above to stretch yourself)
update: see next thread to help with the conversion from 32bit cube to an 8bit cube prior to a GDAL format conversion.

output lossless geoJpeg2000 using gdal to create GeoTiff and free kakadu_binaries (even 16bit compatible for DEMs - great for MOLA, LOLA)
>gdal_translate -of GTIFF input.cub temp_geo.tif
>kdu_compress -i temp_geo.tif -o out_geo.jp2 Creversible=yes Clayers=16 -num_threads 2
Kakadu binaries for download: http://www.kakadusoftware.com/index.php?option=com_content&task=view&id=26&Itemid=22

stretch to PNG 8bits and save out PNG worldfile (recommended):
>gdal_translate -of PNG -scale -co worldfile=yes input.cub out.png

stretch to JPG 8bits and save out JPG worldfile (recommended):
>gdal_translate -of JPEG -scale -co worldfile=yes input.cub out.jpg

manual stretch to PNG 8bits (no GIS worldfile):
>gdal_translate -of PNG -scale in_min in_max out_min out_max input.cub out.png
  *notes I use output as 1 to 255 so that 0 is maintained as NULL - example:
>gdal_translate -of PNG -scale  0.0232  0.12  1  255  -a_nodata 0 input.cub out.png

scale (resize) to PNG 8bits at 10% original size for browse image:
>gdal_translate -of PNG -scale -size 10% 10% input.cub out_browse.png

scale (resize) to JPG 8bits at 100x200 pixels for web image (aspect not maintained):
>gdal_translate -of JPEG -scale -size 100 200 input.cub out100x200.jpg

ESRI ASCII Grid format:
>gdal_translate -of AAGrid input.cub output.asc

XYZ format (script may not be available for all binary releases)
>gdal2xyz.py input.cub out.xyz
or a new output type (GDAL v1.8+ ):
>gdal_translate -of XYZ input.cub output.xyz

information about image:
>gdalinfo input.cub

stats about image:
>gdalinfo -stats input.cub

min/max range:
>gdalinfo -mm input.cub

To project to new projection use the program gdalwarp.

----simple example Csh Script to force output scale to 1 to 255.
mag{90}> more /usgs/cdev/contrib/bin/to8bit_gdal_tif.csh
Code: [Select]
#!/bin/csh
#usage: to8bit_gdal_tif.csh input.ext output.tif
#Note the use of $3 in gdal_translate which just allows you to add in other options. By default it will just be blank.
set in=$1
set out=$2

set stats = `gdalinfo -stats $in > /tmp/xxxtemp_stats.junk`
set min = `cat /tmp/xxxtemp_stats.junk | grep MINIMUM | awk -F= '{print $2}' | sed 's/ //g'`
set max = `cat /tmp/xxxtemp_stats.junk | grep MAXIMUM | awk -F= '{print $2}' | sed 's/ //g'`

gdal_translate -ot byte -of GTIFF -co compress=lzw -ot Byte $3 -a_nodata 0 -scale $min $max 1 255 $in $out
/bin/rm -f /tmp/xxxtemp_stats.junk

related:
http://isis.astrogeology.usgs.gov/IsisSupport/viewtopic.php?t=372

-Trent
« Last Edit: October 24, 2013, 10:37:56 AM by thare »

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #1 on: September 08, 2009, 11:06:49 AM »
update: ISIS 3.3.x now has a percentage option in "stretch" and a "bit2bit" program which can help with the conversion from 32bit cube to an 8bit cube. Although don't use bit2bit just yet until clipped values are handled differently.

ISIS3 tips to get 8bit:

If you notice above I state to use ISIS3 to get to an 8bit range since it can give you more options. Here are some directions which should get easier as I think a specific program for ISIS3 is coming (see update above).

So to review, you can simply apply a linear stretch in GDAL:
>gdalinfo -mm input.cub  --> this will give you "in_min" and "in_max" values for the cube
now run gdal translate with
>gdal_translate -of JP2KAK -ot byte -co quality=100 -a_nodata 0 -scale in_min in_max 1 255 input.cub out.jp2

---------------------

OR for ISIS you can run stretch.

Stretch method 1
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs="0:1 100:254" null=0 lis=0 lrs=0 his=255 hrs=255

This allows you to specify input percentages for the mapping pairs. Thus when USEPERCENTAGES=true is set pairs="0:1 100:254" means:
map 0% to 1 (or the file's min value to 1) and 100% to 254 (file's max value).

Stretch method 2
This also means you can apply a recommended 0.5% clip to remove the potential extraneous lows and highs like:
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs="0:1 0.5:1 99.5:254 100:254" null=0 lis=0 lrs=0 his=255 hrs=255

-Trent
« Last Edit: January 11, 2013, 09:59:09 AM by thare »

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #2 on: November 23, 2009, 08:00:08 AM »
There have been changes to the command-line parser for ISIS3. It appears at least for tcsh you don't need the backslash escape characters. But it looks like there was another change for the general "to" parameter. If you choose a lower bit type it seems it forces you to add in a output range there... I will alert the the ISIS team to this issue.

So if you redundantly specify the stretch pair it seems to work. This makes sense for other programs but not for stretch. Maybe this means we need an independent program to convert types and will help the user with the in_min and in_max values. It would be good to have a few stretches to pick from (e.g. linear, std dev (#), log, guassian, etc.).

so try this - notice the output range or stretch pair is also on the "to" parameter:
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 pairs="0.092769:1 0.18348:254" null=0 lis=1 lrs=0 his=255 hrs=255

-Trent
« Last Edit: January 11, 2013, 09:57:25 AM by thare »

tscharff

  • Horus (New Member)
  • *
  • Posts: 2
Re: GDAL Conversion Tips
« Reply #3 on: November 24, 2009, 01:21:12 AM »
Confirmed: It works in bash with on a real image with non-escaped quotes and the range specifier appended to the "to" argument.

Many thanks for you help, Trent.  I would never have figured this out on my own.

Ted.

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #4 on: October 11, 2011, 05:33:53 PM »
Here is another tutorial called, Basic conversion tutorial for planetary data. Mostly for ISIS to ArcMap 10 but other good information in there for other apps too.
http://webgis.wr.usgs.gov/pigwad/tutorials/arcgis10.Basic_converting_data.pdf
parent page: http://webgis.wr.usgs.gov/pigwad/tutorials/planetarygis

from: https://isis.astrogeology.usgs.gov/IsisSupport/index.php/topic,3310.msg12914.html

-Trent
« Last Edit: October 11, 2011, 05:36:16 PM by thare »

wfarrand

  • Osiris (Active Member)
  • **
  • Posts: 12
    • Email
Re: GDAL Conversion Tips
« Reply #5 on: October 24, 2011, 08:13:27 AM »
OK, I was able to get the Jpeg2000 file generated with isis2std and downloaded isis3world.pl and got an appropriate world file and *.prj file generated so that I could read the CTX image into ArcMap just fine.  I'm completely unfamiliar with the GDAL stuff, but I might have to look into it.  Thanks!

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #6 on: December 06, 2012, 03:37:16 PM »
Just an update for this thread with my currently favorite method to convert to 8bit (before using gdal to convert to geoTiff or GeoJpeg2000 or other format). I can't recommend the ISIS program bit2bit since it pushes valid data which is clipped into NoDATA (not good for viewing the image).

Stretch method 1
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs="0:1 100:254" null=0 lis=1 lrs=1 his=255 hrs=255

This allows you to specify input percentages for the mapping pairs. Thus when USEPERCENTAGES=true is set pairs="0:1 100:254" means:
map 0% to 1 (or the file's min value to 1) and 100% to 254 (file's max value).

Stretch method 2
This also means you can apply a recommended 0.5% clip to remove the potential extraneous lows and highs like:
>stretch from=input.cub to=output_8bit.cub+8bit+1:254 USEPERCENTAGES=true pairs="0:1 0.5:1 99.5:254 100:254" null=0 lis=1 lrs=1 his=255 hrs=255

-Trent

Note: that all the special pixel values are getting mapped to 0 or 255.

If there is speckle in the shadows of your image it might be from LIS or LRS pixels.  You can try to map those to 1 instead of 0 (e.g. lis=1 lrs=1). But Nulls should always be mapped to 0.
more see: http://isis.astrogeology.usgs.gov/IsisWorkshop/index.php/Special_Pixels#How_many_special_pixel_values_exist_in_Isis.3F



« Last Edit: December 28, 2013, 11:21:25 AM by thare »

Valentina

  • PhD Student
  • Isis User Group
  • Osiris (Active Member)
  • **
  • Posts: 19
  • Studying the tectonics of Mercury☿
    • Email
Re: GDAL Conversion Tips
« Reply #7 on: April 17, 2014, 08:34:23 AM »
Dear Trent,

What happens if I stretch a multi band image?
With monochrome images I usually use the stretch command in ISIS3 as you suggested and it works perfectly. When I tried to do the same with a multiband image two of the three bands resulted like being "overexposed" and the final GDAL converted image had a funky color (oversaturated cyan). Should I stretch each band separately in some way?

Thank you.

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #8 on: April 22, 2014, 06:38:34 PM »
Hmm - I haven't tried that yet. I guess you could stretch each band if you need too but I would assume the stretch program would do this for you? You might need to get a good looking color stretch manually in qview and write down the min/max values it calculates for each band. To run one band at a time with a min/max stretch you can add "+1" for band 1, "+2" for band 2, etc.:

> stretch from=input.cub+1 to=output_8bit_band1.cub+8bit+1:254 pairs="MIN:1 MAX:254" null=0 lis=1 lrs=1 his=255 hrs=255
   where MIN and MAX are from qview or other method
> stretch from=input.cub+2 to=output_8bit_band2.cub+8bit+1:254 pairs="MIN:1 MAX:254" null=0 lis=1 lrs=1 his=255 hrs=255
> stretch from=input.cub+3 to=output_8bit_band3.cub+8bit+1:254 pairs="MIN:1 MAX:254" null=0 lis=1 lrs=1 his=255 hrs=255

You can recombine the colors in ISIS or GDAL

ISIS:
> ls output_8bit_band*.cub > list
> cubeit fromlist=list to=output_8bit_rgb.cub

GDAL
> gdal_merge.py  -separate -o output_8bit_rgb.tif   output_8bit_band*.cub

I'm sure there is a better method...?
-Trent

Also in ArcMap, you do have control over each band (even if not 8bit) - see jpeg. The histogram settings in this example will actually take a DEM duplicated over 3 bands and colorize with a rainbow-ish stretch. Kind-of a strange method to colorize a DEM but it works.

The histogram settings show:
# Red stretch pairs. The pattern is percentage:DN
0:75 10:0 40:0 55:204 70:255 85:255 99:150 100:255

# Green stretch pairs. The pattern is percentage:DN
0:0 10:0 30:151 40:204 55:204 70:102 85:0 99:75 100:255

# Blue stretch pairs. The pattern is percentage:DN
0:102 10:204 25:151 40:0 99:0 100:255


« Last Edit: April 26, 2014, 06:38:58 PM by thare »

sruthi

  • Ra (Power Member)
  • *****
  • Posts: 103
Re: GDAL Conversion Tips
« Reply #9 on: June 13, 2014, 03:43:17 AM »
All the above information works very well but when i try to translate a number of cubes to Tiff using Gdal_translate i am always unsuccesfull.
Can you suggest how it can work well or the best way of conversion.Its really a tidious job to translate each cube.

Thanks in advance,
Sruthi

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #10 on: June 14, 2014, 09:43:21 AM »
maybe try this, change options for what you want. This will result in a double extension which is a little odd. Using "basename" or sed can be used to strip the extentsion in linux. There are tricks in the dos environment to remove extensions too.

batch (*nix, using csh or tcsh):
foreach i (*.cub)
foreach> gdal_translate $i  $i.tif
foreach> end

remove extension (note direction of single quote)
foreach i (*.cub)
foreach> set outname = `basename $i .txt`
foreach> gdal_translate $i  $outname.tif
foreach> end


batch (dos):
for %i in (*.cub) do  gdal_translate %i  %i.tif

remove extension in dos (I think):
for %i in (*.cub) do  gdal_translate %i  %~ni.tif

help...?
-Trent
« Last Edit: June 14, 2014, 09:54:45 AM by thare »

sruthi

  • Ra (Power Member)
  • *****
  • Posts: 103
Re: GDAL Conversion Tips
« Reply #11 on: June 18, 2014, 01:58:36 AM »
I tried all of the methods issue is i dont know where i m going wrong but i always see the syntax error at (*.cub) whenever i try to run my #.sh.
Is it because i am giving the text input wrong or my csh shell not reading the paranthesis?

Help me  :(

---
Sruthi
« Last Edit: June 18, 2014, 09:48:01 PM by sruthi »

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #12 on: June 18, 2014, 07:37:55 AM »
A couple issues.

(1) My example emulates what is typed on the command-line and not really a script to be run and
(2) it is a CShell/TShell "csh" script not Bash "sh" script.

I attached both types of scripts below, first download, then try:

---convert to executable
chmod +x cub_to_tif.sh
---now run
./cub_to_tif.sh

same with the "csh" script if you like Cshell. If you are coming from a windows edited script file you might have to run this first:
dos2unix cub_to_tif.sh

Note this doesn't change the ISIS bit type, typically 32bit. The Tiff will also be 32bit. For just drawing on the image 32bit is overkill and I first recommend converting the isis cubes to 8bit first. See the ISIS "stretch" command above.

-Trent

« Last Edit: June 18, 2014, 07:23:05 PM by thare »

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #13 on: October 18, 2014, 10:34:07 AM »
My new favorite environment for GDAL and works on probably all OSes. Install Anaconda python environment and then run "conda install gdal".
https://store.continuum.io/cshop/anaconda/

-Trent


Update: JohnVV asks, how well does "Anaconda python" work with Vision workbench and ASP  and isis ?
« Last Edit: January 08, 2015, 05:34:49 PM by thare »

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1187
    • http://webgis.wr.usgs.gov
    • Email
Re: GDAL Conversion Tips
« Reply #14 on: October 28, 2014, 09:45:37 AM »
ISIS should not have any issues since it doesn't use Python or GDAL. For the Pipeline, I would install Anaconda with version 2.7 of Python which should be fine for their 2.6 version Python scripts. If not, you can change the Python version in Anaconda. Now I have not tested the integration of Anaconda with GDAL since the Pipeline uses GDAL also. I bet it will work but you can also keep the run-time environments independent.

Anyway, I will try to setup a VM with all and see how it works (and report back here). For GDAL apps/scripts in Linux we currently still use a USGS-version of FWtools ("portable" GDAL/python environment) but I think Anaconda should work also. I like portable environments since they do not mess with the current environment and you don't have to be root to install. But you also don't have to be root to get Anaconda installed.

-Trent

Anaconda: https://store.continuum.io/cshop/anaconda/
USGS-version of FWTools: ftp://pdsimage2.wr.usgs.gov/pub/pigpen/GDAL/
« Last Edit: October 28, 2014, 01:26:18 PM by thare »