Author Topic: isis2world on THMPROC BTR QUB  (Read 18144 times)

katef

  • Guest
isis2world on THMPROC BTR QUB
« on: September 21, 2006, 09:38:26 AM »
I ordered a 32 bit, Band 9 ,Polar-projected BTR cube from THMPROC and received a file with a .QUB extension.  It looked fine in the preview on thmproc.  However, when I try to generate a world file to put it into ArcMap 9 using isis2world.pl (with -e), the raw file is ridiculously small, and the world file is nonsense.  It is in totally the wrong place in Arcmap, and the temperature values (even after running it through USGS Image Toolbox at 16bits) are nonsense.  I have ISIS 3 but not the isis2gisworld.pl script (perl dowload site is down today).  

I have tried to open the file in qview, and I basically get a projected, white rectangle. There's no information in the pixels, but it is clearly polar projected. So I guess that ISIS recongizes the file as an ISIS cube, but is not showing any data. Not sure, especially since the file size is large . . .

Also, I have re-ordered the file from THMPROC to make sure that they didn't send me a faulty file. I have made sure to order a polar-projected, 32-bit ISIS cube, channel 9 BTR. I have also changed the longitudes to the -180:180 system since I thought that might help with my worldfile problems. I have not received the new file yet, but when I do, I will test it out with qview and isis2world.pl again and let you know what happens.

Incidentally, the old file in Arcmap is not only just in the wrong place, but it is bent. Weird.

Any ideas what might be wrong here?  I have followed the exact instructions of a colleague who has done this successfully.  

--Kate

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1145
    • http://webgis.wr.usgs.gov
    • Email
isis2world on THMPROC BTR QUB
« Reply #1 on: September 26, 2006, 10:24:01 AM »
Just a couple comments:

 
1.) The Qub files from ASU's thmproc are are ISIS2-like but not a 100 percent exact (i.e. using a 16bt NULL in a 32bit file).

2.) isis2gisworld.pl is only for the ISIS2 system.  There isn't a similar program built into ISIS3.  ISIS3 only has, isis2std, which has the ability to write out 8bit png, tif, or jpeg (depending on your QT install - which is going to cause some confusion).  If the file has mapping labels, it appears will automatically write out a worldfile.

3.) Make sure to always check "Unrectify" so that the projection is re-applied after all the corrections are made.

4.) From ASU slide about UDDW (Undrift-Dewobble) and temperatures: "Care should be taken during dusty seasons, at high elevations, and crossing the polar vortex."

5.) Lastly, these polar files are huge without compression!

I processed three polar images through thmproc.  I used POLA with a clon=0 and clat =90.  After 24 hours it has finished only the "tsurf" (surface temperature QUBS).  "Surface/target temperature—measure radiance curve, compute brightness temperature at each point, the maximum brightness temperature approximates the surface temperature." -notes by N. Barlow.  Also these images have to be daytime to get valid surface temps.  Anyway, I brought those down, un-gzipped and ran:
Dos> for %i in (*.QUB) do isis2world -e %i

Then I used the "batch set to Null 16bit".  They appear to be correct.  My tsurf images look a mess but maybe valid, ranges from 0 to 181.398.  Spatially, they are correct.

Let me know if the next batch worked for you,
Trent

katef

  • Guest
Polar Projections Strike Again!
« Reply #2 on: September 27, 2006, 01:19:26 AM »
Thanks for the help, Trent.

I emailed Josh Banfield to find out whether or not I might have done something wrong when I ordered from THMPROC.  I knew to check "Unrectify", so I knew that wasn't the problem.  Also, I have already created a brightness temperature map with this image using an RDR processed in ISIS3, output to png, and then converted to temps in IDL.  So I knew that the data itself was okay (though admittedly not perfect).  However, there wasn't a straightforward way to the project the png from IDL and create a worldfile for it.  So I decided to try THMPROC.  

Anyway, Josh's answer was basically that polar files are huge, and there's the null value 16bit vs. 32 bit problem (hence the white box in qview).  For the null values, it's fine, because I can use the USGS Image Toolbox on it in ArcMap.  And I think that my Arcmap projection problem was that I didn't specify the correct projection parameters.

The new products that I had ordered from THMPROC were asking too much, because the polar files are so big (and the request was never completed).  So . . .

So from what I understand from your and Josh's answers, I need to order from THMPROC a Band 9, 32bit ISIS cube, BTR, polar-projected, with meridian = 0, clat = 90, lonsys= -180:180, and cropped to a smaller latitude range.  I just ordered this, it went extremely fast, and it looks good in THMPROC preview.

Then I am confused.  Do I now run the resultant QUB file through isis3world.pl (which I downloaded) or through isis2std to get a world file and image that I can read into ArcMap?
 :?:

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1145
    • http://webgis.wr.usgs.gov
    • Email
isis2world on THMPROC BTR QUB
« Reply #3 on: September 27, 2006, 08:06:12 AM »
Great. So it sounds like if you put in a lat/lon constraint the polar images will work.

Remember the ASU QUB is still an ISIS2 formatted file.  ISIS3 files are a completely new (and incompatible) format.  Although I think ISIS3 can read/convert ISIS2 files to the new format.

I guess it depends on what you what to do; get into ISIS3 or get into a GIS/RS package.

GIS:
So since these are ISIS2 files, you will still need to run isis2world.pl standalone PERL or EXE).  For ArcMap I would suggest the "-e" ERDAS raw switch.  If you are running ISIS2 then the built-in "isis2gisworld.pl" script should work on these files too (It is basically the same script but uses ISIS routines to read the label).  ISIS2's "dform" might not work well because of the 16bit null issue.

ISIS3:
hmm, since ISIS3 can read ISIS2 files maybe ISIS3's "isis2std" would work?  I think I heard the conversion from ISIS2 to ISIS3 might lose some of the projection label information???  I'll try to find out.  And you still might have issues with the 16bit NULL data.  I'm not even sure how to convert that value into a NULL value using ISIS3.  I am still new to it also.

Trent

katef

  • Guest
Still not working . . .
« Reply #4 on: September 28, 2006, 08:32:36 AM »
Hello again.

So I ran the new QUB that I ordered from THMPROC (as described above) through the isis2world.pl script (I don't have ISIS2).  And I still get a worldfile that doesn't work (see below) and a raw file that is only 1kb in size.  So I am still confused. I feel like I am missing something very obvious here since I know other people have done this without much problem.  Or is it that polar projection rearing its ugly head again!?

Thanks again!

WORLD FILE:
103.963
0.0
0.0
-103.963
-848182.1355
509262.7555

P.S. I attached the header of the QUB file in case that helps.

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1145
    • http://webgis.wr.usgs.gov
    • Email
isis2world on THMPROC BTR QUB
« Reply #5 on: September 28, 2006, 11:21:49 AM »
But it is working!  :wink:

The ERDAS "raw" file is just a detached text file that points to the *.QUB.  Thus ArcMap will read the *.raw to access the raster in the *.QUB. So all three files ( .QUB, .raw and the .rww) need to be in the same directory to be accessed by ArcMap or other app.

Using your header.txt the *.raw file will look like:
Code: [Select]
IMAGINE_RAW_FILE
WIDTH 4755
HEIGHT 3632
NUM_LAYERS 1
FORMAT BSQ
DATA_TYPE F32
BYTE_ORDER LSB
PIXEL_FILES header.txt   ** This line will point to your *.QUB **
DATA_OFFSET 33792



The *.rww is the worldfile, which is the most simplest method to georeference an image.  The 1st and 4th numbers are the cellsize in X and Y.  The 5th and 6th numbers tell the image where to start in the X, Y Cartesian plane defined by the projection (i.e. Polar Stereographic) and the projection's parameters.  For more: http://isis.astrogeology.usgs.gov/IsisSupport/viewtopic.php?t=328 ArcMap will still complain it doesn’t know the projection, but it should be positioned correctly relative to the other images.


You will still need to run "batch set 16bit Null" on the *.raw files to remove the border pixels from the displayed image range.


Tip: if you have changed your worldfile after ArcMap has viewed it, there is the potential that the registration information now resides in an extra file ArcMap created call the *.aux file.  This mystery file can hold image statistics and other raster information.  So if the image info (i.e. worldfile) changes delete this *.aux so that ArcMap will refresh it.  If the image changes (say you run a different stretch in ISIS), then it is a good idea to also delete the *.rrd file, which is a reduced resolution version of the file.


Trent

-----------
Another format like the ERDAS raw is the PCI *.aux file (not to be confused with the ArcMap *.aux file).  If you run "isis2world -p" on your *.QUB a PCI *.aux will be created.  This detached header also just points to the original *.QUB.  This allows PCI Geomatica and programs like "gdal_translate" to read the ISIS2 file also.  Using the PCI *.aux, gdal_translate can convert the ISIS2 image into many of formats like a minimal geotiff.

katef

  • Guest
Actually, still not working
« Reply #6 on: September 29, 2006, 03:18:22 AM »
Yes, it's me again.  Well, I did stupidly forget to put the QUB file in that directory too.  But after I did that, while the file is geographically in the right place, I still can't get any data out of it.  I run the "16 bit null" thing and still only get a few pixels of data.  I tried it for 32 bit too.  I'm getting vales like 1e-038 to 1e038 to begin with and -3278 (or whatever the default "Value" is in the "16 buit null" tool) to 1e038 after running the tool.  I even tried erasing all the files and putting them in again and running the "16 bit null" again.  Still no love.  Since I have created a temperature map on this image before using IDL, I also tried to run "16 bit null" but constrained the values to the actual temperature constraints.  After that, I got almost no data at all (maybe a scattering of pixels).  So I don't know what the deal is.  I would give up on this image except that it looks fine in THMPROC preview and works just fine in IDL.  The data is scratchy but it's not awful.   Sorry to be such a pain, but I just can't seem to get this to work (even though you've been a great help).

--kate

Incidentally, to clarify -- the ArcMap tool does remove the black border pixels, but the image itself is left as a scattering of very few pixels with messed-up values.

madepablo

  • Guest
Re: isis2world on THMPROC BTR QUB
« Reply #7 on: August 19, 2009, 04:52:31 PM »
Hi Katef,

Could you provide a step-by-step procedure to have BTR files into arcmap from THMPROC?
I am something lost with all the possibilities and options, and i am never sure that the result provide real surface BTR values, and how to include it into a GIS (armap). I don't have a vailable ISIS in my computers, so to use scrips such as isis2world should be perfect.

I suposse that it could be useful for a lot of people.

Thanks for your time,

katef

  • Guest
Re: isis2world on THMPROC BTR QUB
« Reply #8 on: August 20, 2009, 08:57:54 AM »
Okay, here is what I have done in the past.  I haven't done this in a while (e.g., haven't tested it with ArcGIS 9.3, only 9.2), so some steps might have changed.  Also, you need ISIS, and since it is free, I would recommend downloading it.  You're going to need ISIS for so many other projects anyway that it just makes sense to have it.  The BTR values from THMPROC are real and valid.  I have tested it by producing my own BTR maps in IDL and get the same values.  I hope this helps . . . .

IMPORTING  THEMIS BAND 9 TEMPERATURE MAPS INTO ARCGIS:

1.   Order projected, Band 9 ISIS cube from THMPROC (THEMIS Processing Web Interface)
a.   Go to http://thmproc.mars.edu/index.php (you can also get to it from http://themis.asu.edu/thmproc).
b.   You will need to create a username and password.  These are used to log in and check on the status of your order and to retrieve your order when it is finished.
c.   Click “New Batch”.
d.   Enter the specific THEMIS IR images that you want to process.
e.   Enter a job description that will jog your memory about what you ordered.
f.   Under “Standard Processing”, make sure to click ALL the boxes (including “Unrectify”).  If you don’t click “Unrectify”, your projection will be messed up.
g.   Enter the projection you want, and remember that ArcGIS works under a -180:180 longitude system.
h.   Cropping is a good (and sometimes necessary) idea, especially if you are ordering polar projected cubes, which end up being huge files.  If you don’t crop, your order might not be processed.
i.   For “Output”, you generally want just “Brightness Temperature” (ordering more will seriously delay the processing time).  Under “Brightness Temperature”, click “32 bit ISIS cube”.  Then choose “Band 9” (don’t worry about “Backplanes”).
j.   Click “Submit Query”.  The rest is straightforward.
k.   You will have to check on the progress of your order by logging back in periodically (your order name and status will appear when you log in, and you just click on it to see the product).  You won’t receive any kind of notification when the order is processed.  When the order is complete, you will be able to preview it to see if it looks okay (this allows you to skip the same check in ISIS qview).
l.   Save your cube (.QUB) somewhere that is accessible by ISIS.

2.   Create a raster file and worldfile from your ISIS cube.
a.   To do this, you need the isis2world.pl perl script from the USGS.
•   Go to the USGS PIGWAD ftp site http://webgis.wr.usgs.gov/pigwad/tutorials/scripts/perl.htm .
•   Download isis2world.pl.  It will bring up the script in the browser window.  Copy-paste it to a text file and make sure to save it with the .pl extension (no .txt extension).
•   Obviously, you need perl up and running on your machine, so talk to your sys admin if you don’t know how to do this (too much extra detail for this post).
b.   In the ISIS environment, run the perl script on the cube.
•   >> perl isis2world.pl –e filename.QUB (the –e tag creates a raw file and is the best option for import into ArcGIS).
c.   The perl script will generate 2 files: filename.raw and filename.rww (the world file).  filename.raw points to the .QUB for its data.
d.   Open the .raw file in a text editor and make sure that the byte order is LSB (not MSB).  Change it to LSB if necessary and re-save.  If you don’t so this, the USGS tool you need below won’t work.

3.   Import the raw file into ArcMap.
a.   Create a file in ArcCatalog called themisimagenumber_dataset (obviously you put in the image number and dataset).
b.   Copy the .raw, .rww, and .QUB files into this folder (you need the .QUB since the raw file points to .QUB for the data).
c.   Add the .raw file to your ArcMap project.  It may look really strange at this point, but it should be in the right place.
d.   You will need the USGS Image Toolbox v1 for ArcMap 9 to make the raster not look strange anymore.
•   Search for this toolbox and download it from the ESRI scripts website (http://arcscripts.esri.com).  Save the script with your other ArcMap toolboxes.
•   Add the toolbox to ArcMap by right-clicking on “ArcToolbox” in the toolbox area of ArcMap ? choose “Add Toolbox”.
e.   Run the “Covert 32bit NODATA” tool from the USGS Image Toolbox to covert the weird values.  Leave the “Expression” box at its default value.  Save the “output raster” in the themisimagenumber_dataset folder you created above.
f.   Voila!  Now it should look like a normal, B/W raster with a scale that corresponds to temperature.  You can of course change the color scale.  And you can get rid of the black space around it by setting the “nodata” pixels to clear (Under the “Display” tab in the “Layer Properties” box, which you get by double-clicking on the layer).




« Last Edit: June 25, 2010, 08:07:07 AM by thare »

thare

  • GIS Support Team
  • Isis (Extreme Power Member)
  • *****
  • Posts: 1145
    • http://webgis.wr.usgs.gov
    • Email
Re: isis2world on THMPROC BTR QUB
« Reply #9 on: August 20, 2009, 09:37:14 AM »
awesome - great summary.

I thought I might mention a warning. The last time I checked, not for a while now, the ASU generated ISIS2 cubes incorrectly used a 16bit NULL data value for their 32bit generated ISIS2 Qubes. This means to set the NULL value for ArcMap you have to run the "Convert 16bit NODATA" instead of the "Convert 32bit NODATA" (which is used on more normal ISIS2 32bit files).

Test one image both ways and see which one works.

-Trent

madepablo

  • Guest
Re: isis2world on THMPROC BTR QUB
« Reply #10 on: August 20, 2009, 04:57:59 PM »
Thanks so much Kate for the detailed explanation, and also Trent for the warning.

I will test it this weekend and post here the the results.
Thanks a lot!
« Last Edit: January 25, 2012, 07:48:54 AM by thare »

madepablo

  • Guest
Re: isis2world on THMPROC BTR QUB
« Reply #11 on: August 21, 2009, 12:38:03 AM »
Hi.

I tryed to process the BTR as Kate said (i missed some steps, because, in fact i already have an account in THMPROC, i already have the USGS toolbox, and i don't have installed perl in my laptop; so i used the exe version of the scripts), and mostly everything runs perfectly.

The unique problem was the isis2world, that sometimes creates *.raw files without the necessary data inside... but i repeated the process that it was there, so...
I also had problems with arcmap, because the raw file was in the incorrect place. It was not a problem with the projection that i selected in THMPROC. But what i did was to change a little bit the call to the script to force it to create a prj file, in that way:
isis2world.pl -e -prj {filename.QUB}
it created {filename.prj} file.
Then, in arcmap, by the use of USGS Image Toolbox, i used first the tool: "Batch Set Image Projection from PRJ" what assign the creater prj file to the raw one.
And finally i used the "Convert 16bit NODATA" tool from the USGS toolbox as proposed Trent, because i also tried with the 32bit and it produce a bad result (although it maintains the pixel value). And i produced a img file (i don't like so much the esri grid). So finally i have the image correctly proyected, in the correct place, with the correct values of the pixels.!

So thanks so much to both. It is nice the patience of Kate to write this manual. I think that it is enough detailed to correct it with the changes proposed by Trent and save as a pdf manual some were in the ftp server for everybody. Thanks kate and thanks Trent.

Just only one more question. In THMPROC also allow to obtain surface temperature products. Why is better BTR than surface temperature? Ok, it is not corrected the atmosphere effect, but for not so dusty days, may be then difference is pretty small....
In fact, i already processes by THMPROC the same file but both options, BT and Surface temperature. Ones in arcmap, there are not differences between them... I calculated the difference between them by raster calculator, and the result is 0.

Thanks!
« Last Edit: August 21, 2009, 01:58:05 AM by madepablo »

ejones

  • Osiris (Active Member)
  • **
  • Posts: 10
    • E Jones
    • Email
Re: Actually, still not working
« Reply #12 on: May 10, 2012, 08:52:50 PM »
Hi Kate,  did you find a solution to this problem?  I am having the same problem also - but only for multiband qubs not single band ones!


Yes, it's me again.  Well, I did stupidly forget to put the QUB file in that directory too.  But after I did that, while the file is geographically in the right place, I still can't get any data out of it.  I run the "16 bit null" thing and still only get a few pixels of data.  I tried it for 32 bit too.  I'm getting vales like 1e-038 to 1e038 to begin with and -3278 (or whatever the default "Value" is in the "16 buit null" tool) to 1e038 after running the tool.  I even tried erasing all the files and putting them in again and running the "16 bit null" again.  Still no love.  Since I have created a temperature map on this image before using IDL, I also tried to run "16 bit null" but constrained the values to the actual temperature constraints.  After that, I got almost no data at all (maybe a scattering of pixels).  So I don't know what the deal is.  I would give up on this image except that it looks fine in THMPROC preview and works just fine in IDL.  The data is scratchy but it's not awful.   Sorry to be such a pain, but I just can't seem to get this to work (even though you've been a great help).

--kate

Incidentally, to clarify -- the ArcMap tool does remove the black border pixels, but the image itself is left as a scattering of very few pixels with messed-up values.