Author Topic: Trouble Using Kaguya TC Mosaic Tiles  (Read 3988 times)

astrostu2

  • Hathor (Active Member)
  • ***
  • Posts: 31
    • Email
Trouble Using Kaguya TC Mosaic Tiles
« on: March 29, 2016, 11:28:24 AM »
I've been downloading some of the 10 mpp 3°x3° Kaguya terrain camera mosaics.  Problem is getting the projection correct.

When I expand the files and import the raw IMG file into ArcMap, it's almost a random placement with respect to longitude.  For example, a tile that's supposed to be 270-273° longitude maps to 90-93° (so maybe it's a 0-360 vs ±180° issue?), but then the next tile that's supposed to be 273-276° longitude maps to 87-90°.  ???

So I ran PDS2ISIS and went to use MAP2MAP to get it into a -180° to +180° lunar coordinate system.  Problems came up.  It said, basically, there was no mapping group information.  So I created a maptemplate from scratch using the file name to get min/max lat/lon, and then I forced the maptemplate onto the CUB file with maplab.  No errors in ISIS.  Tried bringing the CUB into ArcMap, and it lists #QNAN as the extent and won't show it.

Help?

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1217
    • http://webgis.wr.usgs.gov
    • Email
Re: Trouble Using Kaguya TC Mosaic Tiles
« Reply #1 on: March 30, 2016, 02:05:53 PM »
We have been fortunate to work with the Kaguya Team and will try to help them release some of their data. A project just wrapped up last year to pull the TC map projected tiles together and release them in larger pieces. You can find very sub-sampled versions here: http://astrogeology.usgs.gov/search/results?q=kaguya+terrain
But for the full-res TC images you will find a link on those pages to zipped up tiles. These should work in ArcMap without hassle (besides setting a good stretch). Note, these are not fully controlled mosaics and not yet controlled to LOLA so there will be some offsets to LOLA.

Also Univ. of Hawaii has gathered some of the Kaguya MI tiles together and that should appear here relatively soon.

For a quick view of the "best-of" TC mosaic (merge), we also have it available as a WMS layer (full-res). To make sure it is projected to the Moon in ArcMap just add-in a map projected file first and then add this WMS (you can find it in the "add data from ArcGIS Online" pull-down). ArcMap will force the WMS layers to the Moon: http://planetarygis.blogspot.com/2014/09/tips-to-interact-with-astros-wms-maps.html
LROC Team has a great WMS also. If you zoom-in enough and wait for a few seconds you can view many of the LROC NACs: http://lunaserv.lroc.asu.edu/about.html

-Trent

Make sure to refer to the JAXA Conditions for material usage page ( http://jda.jaxa.jp/en/service.php ) and please credit JAXA and the SELENE/Kaguya data. ©JAXA/SELENE
« Last Edit: April 01, 2016, 09:36:06 PM by thare »

astrostu2

  • Hathor (Active Member)
  • ***
  • Posts: 31
    • Email
Re: Trouble Using Kaguya TC Mosaic Tiles
« Reply #2 on: March 30, 2016, 02:18:04 PM »
Hey Trent, thanks for the info.  I had seen that before, but the whole "64ppd" meant I didn't even click.  The link to the ZIPs is buried, but I found it: http://pdsimage.wr.usgs.gov/downloads/Kaguya_TC

I'm trying a "tiny" 19GB tc_mor_v04_354_360.cub.zip and will let you know.  Out of curiosity, do you know if these are 8-, 16-, or 32-bit, and if there's a nice 'n' easy GDAL command to (down-sample if 32- and) convert to JP2, one of my new favorite image formats in Arc?

And yes, I know these aren't controlled to LOLA, but I'm hoping that they will be close enough to to WAC/LOLA where I need them such that it won't be a significant issue.

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1217
    • http://webgis.wr.usgs.gov
    • Email
Re: Trouble Using Kaguya TC Mosaic Tiles
« Reply #3 on: March 30, 2016, 02:57:59 PM »
They are 16bit. Conversion to Jpeg2000 should be fine. For the WMS mosaic I also stretched to 8bit and then merged. Below are the linux commands (and ISIS stretch command with stretch pairs listed). I had to use the linux 7z routine since the older unzip available in my Linux box could not handle these larger zip files. The gdal_translate shown here has the Kakadu encoder built-in. You could just batch convert the 8bit cubes in ArcMap too or build pyramids for the cubes and use them directly.

-Trent

Code: [Select]
7z e tc_ortho_v02_000_030.cub.zip
stretch from=tc_ortho_v02_000_030.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:2 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_000_030.jp2

7z e tc_ortho_v02_030_060.cub.zip
stretch from=tc_ortho_v02_030_060.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_030_060.jp2

7z e tc_ortho_v02_060_090.cub.zip
stretch from=tc_ortho_v02_060_090.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_060_090.jp2

7z e tc_ortho_v02_090_120.cub.zip
stretch from=tc_ortho_v02_090_120.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_090_120.jp2

7z e tc_ortho_v02_120_150.cub.zip
stretch from=tc_ortho_v02_120_150.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_120_150.jp2

7z e tc_ortho_v02_150_180.cub.zip
stretch from=tc_ortho_v02_150_180.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_150_180.jp2

7z e tc_ortho_v02_180_210.cub.zip
stretch from=tc_ortho_v02_180_210.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_180_210.jp2

7z e tc_ortho_v02_210_240.cub.zip
stretch from=tc_ortho_v02_210_240.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_210_240.jp2

7z e tc_ortho_v02_240_270.cub.zip
stretch from=tc_ortho_v02_240_270.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_240_270.jp2

7z e tc_ortho_v02_270_300.cub.zip
stretch from=tc_ortho_v02_270_300.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_270_300.jp2

7z e tc_ortho_v02_300_330.cub.zip
stretch from=tc_ortho_v02_300_330.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_300_330.jp2

7z e tc_ortho_v02_330_360.cub.zip
stretch from=tc_ortho_v02_330_360.cub to=stretch.cub+Lsb+Tile+Attached+UnsignedByte+2:255 pairs="0:1 30:254" null=0 lis=1 lrs=1 his=255 hrs=255
gdal_translate -of JP2KAK -co quality=100 stretch.cub tc_ortho_v02_330_360.jp2
« Last Edit: March 30, 2016, 03:01:05 PM by thare »

astrostu2

  • Hathor (Active Member)
  • ***
  • Posts: 31
    • Email
Re: Trouble Using Kaguya TC Mosaic Tiles
« Reply #4 on: March 30, 2016, 08:05:44 PM »
The JP2KAK command isn't recognized by my GDAL install ... JP2000 runs, but then aborts:

Code: [Select]
Assertion failed: (ret == cnt), function mem_write, file jas_stream.c, line 1044.
This is despite including a "large_ok=yes" in there.  Any ideas?

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1217
    • http://webgis.wr.usgs.gov
    • Email
Re: Trouble Using Kaguya TC Mosaic Tiles
« Reply #5 on: March 31, 2016, 11:38:13 AM »
Kakadu is the best encoder out there but it is not longer available with most GDAL builds (proprietary). Currently there are no good open source Jpeg2000 encoders although some are getting better. See this GDAL page for our Linux binaries. https://isis.astrogeology.usgs.gov/IsisSupport/index.php?topic=2172.0

Also on this page are instructions to first convert to GeoTiff and then to JP2 using Kakadu binaries.
Quote
output lossless geoJpeg2000 using gdal to create GeoTiff and free kakadu_binaries (even 16bit compatible for DEMs - great for MOLA, LOLA)
>gdal_translate -of GTIFF input.cub temp_geo.tif
>kdu_compress -i temp_geo.tif -o out_geo.jp2 Creversible=yes Clayers=16 -num_threads 2
Kakadu binaries for download: http://kakadusoftware.com/downloads/

And as a test I am temporarily supporting a mosaic of the Kaguya Ortho merge weighing in at only 180 GB! Without compression is would be ~500GB. I would use wget to download. I need to check if it is lossy or lossless (but it would be hard to tell either way).
http://planetarymaps.usgs.gov/mosaic/Lunar_Kaguya_TC_Ortho_Global_4096ppd_v02.jp2
http://planetarymaps.usgs.gov/mosaic/Lunar_Kaguya_TC_Ortho_Global_4096ppd_v02.jp2.html

-Trent

astrostu2

  • Hathor (Active Member)
  • ***
  • Posts: 31
    • Email
Re: Trouble Using Kaguya TC Mosaic Tiles
« Reply #6 on: March 31, 2016, 01:33:05 PM »
Okay, I'll look for other binaries.

Regarding the 180GB download, if that's for me, I'm not going to be able to try that until the end of April (home for the next few days then 2-week vacation in Spain).  I'm also more interested in the morning/evening maps than the regular ortho.

ANahm

  • Osiris (Active Member)
  • **
  • Posts: 12
    • Email
Re: Trouble Using Kaguya TC Mosaic Tiles
« Reply #7 on: May 09, 2016, 04:35:43 AM »
It took me several days to download the 4096 ppd jp2, but I got it. However, when I open it in ArcGIS, it is one big black box. The range in values is 0 to 0, so stretching doesn't seem to help. Do you have any suggestions besides waiting another week for the download again?

Thanks.
Amanda

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1217
    • http://webgis.wr.usgs.gov
    • Email
Re: Trouble Using Kaguya TC Mosaic Tiles
« Reply #8 on: May 09, 2016, 05:56:42 PM »
First what version of ArcMap are you using? I am testing this file in 10.3.1 and it is working fine (but slow). I am suspicious that the download is bad...?  ArcMap should not show 0 to 0 but default to 0 to 255 (8bit). How to test. Right click on layer name in the table of contents and "zoom to raster resolution". Once zoomed-in, the map will draw very fast. If still all black then something is wrong with the file or ArcMap is having issues.

Other ways to check the image.

1.) kdu_show can open this image also. You can get this small viewer free from Kakadu. http://kakadusoftware.com/downloads/  Note that kdu_show isn't on all platforms but it is on Windows (which I assume you have since you are using ArcMap).

2.) check image with kdu_jp2info
Code: [Select]
> kdu_jp2info.exe -i Lunar_Kaguya_TC_Ortho_Global_4096ppd_v02.jp2

<JP2_family_file>
  <ftyp name="file-type box" header="8" body="12" pos="12">
    <brand> "jp2_" 0x6A703220 </brand>
    <minor_version> 0 </minor_version>
    <compatible_brand> "jp2_" 0x6A703220 </compatible_brand>
  </ftyp>
  <jp2h name="JP2-header box" header="8" body="37" pos="32">
    <ihdr name="image-header box" header="8" body="14" pos="40"></ihdr>
    <colr name="colour box" header="8" body="7" pos="62"></colr>
  </jp2h>
  <asoc name="association box" header="8" body="4,346" pos="77">
    <lbl_ name="label box" header="8" body="9" pos="85"></lbl_>
    <asoc name="association box" header="8" body="1,818" pos="102">
      <lbl_ name="label box" header="8" body="18" pos="110"></lbl_>
      <xml_ name="xml box" header="8" body="1,784" pos="136"></xml_>
    </asoc>
    <asoc name="association box" header="8" body="2,495" pos="1,928">
      <lbl_ name="label box" header="8" body="18" pos="1,936"></lbl_>
      <xml_ name="xml box" header="8" body="2,461" pos="1,962"></xml_>
    </asoc>
  </asoc>
  <uuid name="UUID box" header="8" body="486" pos="4,431"></uuid>
  <jp2c name="contiguous-codestream box" header="8" body="rubber" pos="4,925">
    <codestream>
      <width> 1474593 </width>
      <height> 737297 </height>
      <components> 1 </components>
      <tiles> 296 </tiles>
    </codestream>
  </jp2c>
</JP2_family_file>

3.) Crop out a section to test file. This command says decompress a limited region of the original image (starts 50% down and 50% in from left, extends for 2% of the original height and 1% of the original width). Now I'm not sure the geoTiff header updated correctly. For this would need to use gdal_translate (with Jp2 support - not always easy to find).
Code: [Select]
> kdu_expand -i Lunar_Kaguya_TC_Ortho_Global_4096ppd_v02.jp2 -o region_test.tif -region {0.5,0.5},{0.02,0.01}

Using this image:

Once you can figure out if the file is good, there are methods to make this file more usable. So Jpeg2000 is suppose to have built-in "pyramids" but for files this large I think it is still hard for any application to use. So what to do. 

1.) Build pyramids for Jp2 file (leave for the night since it will probably take all night). You can do this in ArcCatalog or ArcMap toolbox or using "gdaladdo".

2.) Convert to BigTiff using free kdu_expand routine and then build pyramids. This will be 500GB in size and pyramids will be huge too. Note gdal_translate can clip out small regions too.
$ kdu_expand -i Lunar_Kaguya_TC_Ortho_Global_4096ppd_v02.jp2 -o outFullImage_geo.tif



If you can't get those to work, there are the smaller tiles available: http://pdsimage.wr.usgs.gov/downloads/Kaguya_TC/TC_Ortho/version_02/

And remember this full-res layer is available as a live WMS layer. It is best to add in a lunar image or shapefile to initialize ArcMap as on the Moon and the map projection you want to use. Now in ArcMap, use "add data from ArcGIS Online", type in "WMS Lunar" into the search window, and click "add" under "WMS Lunar Server, Astrogeology". Note all layers will be turned on. Just leave "Kaguya TC Ortho Mosaic" layer on. It is fairly fast, even online, since it has pyramids built on the server.

let us know,
Trent
« Last Edit: May 09, 2016, 06:09:45 PM by thare »

ANahm

  • Osiris (Active Member)
  • **
  • Posts: 12
    • Email
Re: Trouble Using Kaguya TC Mosaic Tiles
« Reply #9 on: May 10, 2016, 01:27:35 AM »
Hi Trent-

Thank you for your quick and detailed reply!

I am using 10.3.1. I zoomed in to the raster resolution and it was just white. So I conclude that the download was bad. I'm downloading it again just to make sure, but that will take about a week to see the results.

I have connected the WMS lunar server to my map project. I didn't know it existed! It is working perfectly. I still intend to try to get the downloaded version to work, so that I can have a little more control over how it's displayed (like the stretch etc).

Thank you again and I will update when and if I can get the jp2 to work.
Amanda