Author Topic: GIS-ready CTX images - sort-of  (Read 9458 times)

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1217
    • http://webgis.wr.usgs.gov
    • Email
GIS-ready CTX images - sort-of
« on: June 14, 2014, 10:35:05 AM »
While HiRISE, HRSC, THEMIS, MOLA Teams release their products in formats which are map projected (GIS-ready with some caveats), this is not true for CTX images. HiRISE are released as geoJpeg2000 (tip), HRSC are released as well-labelled map projected PDS images and GeoTiff!, THEMIS IR released as a derived mosaics controlled and uncontrolled (and you can process them in THMPROC or POW).  MOLA DEMs released in map projected PDS formats.

But for CTX you have process them in ISIS or POW. Or do you...? Obviously there is Jmars GIS to view them, but if you want them in your own GIS, with a little scripting help, you can get them almost fully processed from ASU's Mars Space Flight Facilities' CTX search website (and very nicely stretched). Although not all images will work in all GISs because of a potential longitude system issue (see below). http://viewer.mars.asu.edu/planetview/inst/ctx#start

UPDATE, Oct. 2015: ASU is slowly rolling out properly labeled GeoJpeg2000 CTX images. It makes this post unnecessary - which is great!




So let's try this image. This is a method from loooonggg ago that I actually had to revive. But it seems to work.

(1) download both the jpeg2000 and ISIS header from the ASU site. If you load the (yes map projected) Jp2 file there is NO geospatial header in the header - ahhh -- whaattt.... So let's "fix" it.
http://viewer.mars.asu.edu/planetview/inst/ctx#/planetview/inst/ctx/B07_012195_1750_XN_05S222W

(2) download my isis3world.exe or isis3world.pl (*nix machines) script into the same directory. See the date on those apps -- 2009! ftp://pdsimage2.wr.usgs.gov/pub/pigpen/Perl/

(3) run this in a command line on the ISIS3 header.  tip: in Windows, shift right click your directory and click on "open command window here".
> isis3world -J -prj B07_012195_1750_XN_05S222W.scyl.isis.hdr
in linux
> perl isis3world.pl -J -prj B07_012195_1750_XN_05S222W.scyl.isis.hdr
-- that is a capital "-J" not lowercase. This makes an old-school GIS worldfile and projection file. But you need to attach the projection file to the image so.

(4.1) In ArcCatalog, stand-alone app or the pop-up window in ArcMap, right click on the B07_012195_1750_XN_05S222W.jp2 , select properties. Under "Spatial Reference" click edit...  Now under the globe icon pull-down select "import" and select the newly created file B07_012195_1750_XN_05S222W.prj. While you are in the image's properties, select the NoDATA "edit..." and set band1 to "0".

(4.2) in GDAL, you can attach the prj virtually (this VRT file should also work for ArcMap)
> gdal_translate -of VRT -a_srs "ESRI::B07_012195_1750_XN_05S222W.prj" -a_nodata 0  B07_012195_1750_XN_05S222W.jp2 B07_012195_1750_XN_05S222W.vrt
Now use the VRT file to open the image in QGIS, Arc, etc. (any GDAL supported GIS). Or translate to a GeoTiff in GDAL. gdal_translate B07_012195_1750_XN_05S222W.vrt B07_012195_1750_XN_05S222W.tif"

lonsys issue: If you drag the image into you GIS and it complains about the image's extent (or doesn't draw it), that is because the image is not using the lonsys recommendations below. There is a method to move it using the georef toolbar, shift button, and radius * PI (but that is for another time). BTW, in VRT you can just edit the X Offset.

I also have an old ArcMap toolbox batch method to attach many *.prj file to many images but I haven't tested in a long time.

Write to ASU... ;-)  and ask for geoJpeg2000 files!    We did and they now support geoJpeg2000!


-Trent


tip -- again GIS rule for processing:
if lonsys = 180 then set clon = 0 (or center of image)  --- recommended for ArcMap
or
if lonsys = 360 then set clon = 180 (or center of image).  --- works in ArcMap not Arcglobe or Google
« Last Edit: October 27, 2015, 10:07:47 AM by thare »

SJConway

  • Osiris (Active Member)
  • **
  • Posts: 6
Re: GIS-ready CTX images - sort-of
« Reply #1 on: August 01, 2014, 09:44:02 AM »
Hello Trent and everyone,

We've made a python script here at Open Uni that makes the projection and georeference information for the CTX JP2 images for ArcGIS 10 (and QGIS) in one step. All you need to do is put the python script in the same directory as all your JP2 and associated scyl.isis.hdr files and then run it.

You can download it from here:
http://www.physics.open.ac.uk/sjconway/datasets.php

No guarantee that it works on all systems/setups, but feel free to change and/or update.

Cheers,
Susan

EDIT: I've updated the code on our website to include Trent's modifications below - thanks Trent!
« Last Edit: January 24, 2015, 08:33:04 AM by thare »

rhemmi

  • Osiris (Active Member)
  • **
  • Posts: 17
    • Email
Re: GIS-ready CTX images - sort-of
« Reply #2 on: August 03, 2014, 01:55:37 AM »
Dear Susan,

It does work fine;)
This is exactly what we've wanted!

Best regards,

Ryodo

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1217
    • http://webgis.wr.usgs.gov
    • Email
Re: GIS-ready CTX images - sort-of
« Reply #3 on: August 05, 2014, 12:18:32 PM »
Perfect. I wish I knew about this before writing the painful steps above!   :o

I did update the code a little since you said it would be okay:
  • the parameter string parsing (you said it was ugly code so I replaced it.)
  • added argparse for command-line options so the user doesn't get asked a question at the end of the run. You can add "--del" to do the same. You can also use "-h" for help.
  • Since it is assumed ocentric lats will be used I change the polar file to use the SemiMinor radius (hack to get around ocentric/ographic issues).
  • define NoDATA = 0 into the aux.xml file
  • other minor tweaks

current usage:
download CTX *.jp2 and *.hdr from ASU and run:
> python ctx_project.py
-- to delete the isis header after running
> python ctx_project.py --del
-- tiny bit of help
> python ctx_project.py -h

new version on original site:
http://www.physics.open.ac.uk/sjconway/datasets.php
or ftp://pdsimage2.wr.usgs.gov/pub/pigpen/Python/ctx_project.py
Windows binary version (created with cx_Freeze):
ftp://pdsimage2.wr.usgs.gov/pub/pigpen/Python/ctx_project.exe

thanks for sharing the code!
-Trent
« Last Edit: October 30, 2015, 10:06:39 AM by thare »

rhemmi

  • Osiris (Active Member)
  • **
  • Posts: 17
    • Email
Re: GIS-ready CTX images - sort-of
« Reply #4 on: August 07, 2014, 06:58:04 PM »
Dear Trent,

I have some questions (or request) about the CTX (.jp2 format) outputs produced by ctx_project.py.
When rendering them on ArcMap 10.2.1, most of the blackest marginal areas (defined as NoDATA = 0) cannot be shown; however, black pixels around each CTX's footprints have still remained.
This problem also happens in HiRISE image.
Do you have any good idea to remove them on ArcMap window (or any processing before rendering)?

In addition, NoDATA = 0 removes simple shade caused by sunlight (e.g., parts of inside of crater) in each image, which works too much for me. How can I avoid this?

Probably, to solve both, (1) create shapefile along each CTX's footprint, and then (2) crop each image using each shapefile are realistic steps, I think, but they take much time for several tens of CTX images (my goal is JMARS-like rendering (but not online-loading) of multiple, map-projected CTX images on ArcMap display). I'd like to know easier and faster methods, if any.

With best regards,

Ryodo

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1217
    • http://webgis.wr.usgs.gov
    • Email
Re: GIS-ready CTX images - sort-of
« Reply #5 on: August 25, 2014, 11:45:23 AM »
Ryodo,
  Corey F. did some testing last week with some good results by playing around with a mosaic datatype. See the tutorial from Raster Riser to create a "mosaic" datatype: http://resources.arcgis.com/gallery/file/arcobjects-net-api/details?entryID=70A6CFB9-1422-2418-3473-F97D1590AFF1

So during the footprint creation you can have it automatically clip the lossy fringe of data around the image. I think we create footprints in 2 steps.
(1) create mosaic datatype footprints with "Radiometry" using ~8 vertices and Ignore (NODATA) as 0. This tries to outline the edge.
(2) No rerun using "None" so it doesn't recreate the footprint but will modify it. Here we clip-back the footprint 50 meters. Basically a negative buffer on the footprint.

Now go the mosaic properties, ArcCatalog, and have it clip to the footprint. Add to Arcmap, then within the layer's properties, in the mosaic tab, set "seamline" and blend. Right click and export to a new raster file to save out a burned-in mosaic with these settings defined.

Optional: to remove inside NoData pixels you can export the mosaic and then stretch all NoDATA to 1 (Raster calculator, Con and IsNull). Now using the moasic datatype "boundary polygon", you can reset the outside and gaps back to NoData (0 in this case). I can try to explain this more if needed.

-Trent
« Last Edit: August 25, 2014, 07:52:20 PM by thare »

mchoj

  • Osiris (Active Member)
  • **
  • Posts: 15
Re: GIS-ready CTX images - sort-of
« Reply #6 on: December 06, 2015, 09:44:54 AM »
Hi Trent,

I am glad this was resolved and ASU fixed the labels on their GeoJpeg2000 CTX images.   As you say, these are very nicely stretched - better than the CTX products we process in-house here at UA.     

Is there a way to submit a list of CTX images to ASU and automatically download those JP2s and .hdr files and avoid manually slogging through all the images.   

I know POW sort of does this by processing from scratch, but I seem to recall someone saying the stretch isn't as nice?    We could write a python script but didn't want to reinvent the wheel....   

Also, this old post reply of yours https://isis.astrogeology.usgs.gov/IsisSupport/index.php/topic,3089.msg12043.html#msg12043 stated " Unfortunately, those images are just jpeg2000 compressed versions of the raw EDRs. " of ASU CTX image - which I am pretty sure is no longer true?  Please confirm. 

Thanks!
Matt

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1217
    • http://webgis.wr.usgs.gov
    • Email
Re: GIS-ready CTX images - sort-of
« Reply #7 on: December 06, 2015, 11:03:14 AM »
If you are running ArcMap, check out this toolbox script. There is a newer version written for the new "Geo"Jp2 files by ASU. Now ASU is running these one more time since a small subset were actually not getting stretched correctly. It takes a full month to rerun all these images which I think start last week! It should be okay to use this script while these files are getting updated but you might get a mix of new and old. If you see a poorly stretch image, try again in a couple weeks.

https://github.com/USGS-Astrogeology/ArcGIS_Plugins/tree/master/CTX_CreateMosaic_fromASU

-Trent
« Last Edit: December 06, 2015, 02:40:22 PM by thare »