hijitter and noproj on hirise images produces DN differences from last release
I'm seeing differences (in some cases significant) between cubes processed with ISIS 3.5.0 and 3.4.13 hijitter and noproj apps. I don't know if it's a bug or some underlying software change? Neither app lists any updates between versions. As an example of what is happening, I've uploaded a screen shot of a difference cube (3.4.13 subtracted from 3.5.0) of output from hijitter. There is some weird blockiness and you can see underlying surface features. I'm not sure what impact this would have on DTM production, but it doesn't seem good.
Let me know what other information I can provide.
#3 Updated by Trent Hare over 1 year ago
To add in a comment here. When the difference was applied and you begin to see "structure", this seems to be indicative of a single pixel shift across the different images. This looks to be the case in the visible blocks (tiles) more than the rest of the image though which is odd. Obviously we would expect it to be same across the image. Thus of course our "tiling" scheme is something to look at for changes. I will assume you are using the same resampling method across both which could have similar "structure" effects. But something else for us to look into also.
#6 Updated by Audrie Fennema over 1 year ago
Here are examples of the command lines I used. I've run several different HiRISE images through, with similar results. Let me know if you need anything else.
noproj to=/tmp/PSP_010453_1675_pnodex:6343/PSP_010453_1675_RED0.noproj.cub source=frommatch from=/tmp/PSP_010453_1675_pnodex:6343/PSP_010453_1675_RED0.balance.cub match=/tmp/PSP_010453_1675_pnodex:6343/PSP_010453_1675_RED5.balance.cub
hijitter regdef=/opt/usgs/isis3data/mro/calibration/hijitreg.p1745.s3070.def tolist=outlist_pnodex:6616.txt jitterck=/HiRISE/Users/audrie/pipe_dev_sandbox/HiRISE/Data/HiJACK/PSP/ORB_010400_010499/PSP_010453_1675//PSP_010453_1675.jittery.bc fromlist=inlist_pnodex:6616.txt jitter=/HiRISE/Users/audrie/pipe_dev_sandbox/HiRISE/Data/ResolveJitter/PSP/ORB_010400_010499/PSP_010453_1675//PSP_010453_1675_jitter.txt
#11 Updated by John Shinaman over 1 year ago
Hi again Audrie,
I just want to clarify my last posting. The file I need is the table of pixel offsets used in the "jitter" parameter. You can either upload the actual file here, or if that becomes a problem, copy and paste the contents here in a post. We're getting ready to work on this problem and we appreciate your assistance.
#13 Updated by Audrie Fennema over 1 year ago
The site won't let me upload or post the contents of the file. The upload says it doesn't like the file extension. I've tried none, .txt, .dat. Is there an acceptable file extension? When I try to post the contents copy and pasted I get an error.
Any other ideas?
#16 Updated by John Shinaman over 1 year ago
I'm trying to reproduce your error and I'm running into procedural issues in getting accurate results. In specific, I can't run "hijitter" after "noproj" because the latter is creating a new idealized "Instrument" group and moving the "CcdId" keyword out of it and into a new group called "OriginalInstrument". "hijitter" wants to see this keyword in the main "Instrament" group as a data identifier. Otherwise, it see's the data no as non-MRO and invalid. I'm not sure how you're getting past this step without similar issues. I suspect I'm missing something in the processing steps.
Are you running the ISIS3 released version of "hijitter" and "noproj" or hybrids? I presume from your post that you're using released versions, but I need to make sure. How are you producing the "balance" cubes? Is there another process between "noproj" and "hijitter". Can you please supply a detailed list of your processing steps from ingestion up through "hijitter" along with any pertinent ancillary input data aside from the original .img's? I've downloaded and ingested the entire CCD array of PSP_010453_1675_RED*.img data into ISIS, but I need to compare the processing steps from there.
#17 Updated by Audrie Fennema over 1 year ago
When processing images you either run noproj or hijitter, but not both. Noproj is run on observations that don't have much jitter and therefore don't need to be jitter-corrected. Hijitter is run on observations that do need to be jitter-corrected.
I am running the released versions. The steps to create the balance cubes are long and complicated. I'll email you a link where you can download some balance cubes and ancillary files you will need to run hijitter.
Let's start with hijitter. Here are the steps I use:
1. Run spiceinit on each balance cube:
spiceinit spkrecon=TRUE shape=SYSTEM spkpredicted=FALSE cksmithed=TRUE from=PSP_010453_1675_XXXX.balance.cub spksmithed=TRUE
hijitter regdef=/opt/usgs/isis3data/mro/calibration/hijitreg.p1745.s3070.def tolist=PSP_010453_1675_outlist.txt jitterck=PSP_010453_1675.jittery.bc fromlist=PSP_010453_1675_inlist.txt jitter=PSP_010453_1675_jitter.txt
I run all the adjacent cubes through hijitreg and then mosaic all the dejittered cubes together using the average sample/line translation with handmos. I'm guessing you won't need to do that? But let me know if you do and I can give more detail.
For noproj I'll give you a different observation so it doesn't get confusing, PSP_009521_2065.
1. Run spiceinit on each balance cube:
spiceinit spkrecon=TRUE shape=SYSTEM spkpredicted=FALSE cksmithed=TRUE from=PSP_009521_2065_XXXX.balance.cub spksmithed=TRUE
Run noproj on each balance cube using RED5 as the match cube:
noproj from=PSP_009521_2065_REDX.balance.cub to=PSP_009521_2065_REDX.noproj.cub source=frommatch match=PSP_009521_2065_RED5.balance.cub
I run all the adjacent cubes through hijitreg and then mosaic all the noproj cubes together using the average sample/line translation with handmos. Same as before, I'm guessing you won't need to do that? But let me know if you do and I can give more detail.
Let me know if something is unclear or you need more info.
#22 Updated by John Shinaman about 1 year ago
- Assignee set to Tyler Wilson
We can recreate the problem. I'm curious, do you test each ISIS version update, or is this the first time you've done the comparison? Either way, it's definitely a difference. The DN values are very small so it may be "in the noise" of the processing, but in any event, we'll track it down.
Thanks for your input!
**** INTERNAL NOTES ****
data working directory: /work/users/jshinaman/Support/RM_4741 (this is an extensive directory structure, but it includes all the processed data, print files, and comparison data)
results data directory: /work/users/jshinaman/Support/RM_4741/RESULTS (this contains the FX difference cubes)
The anomaly is spread across several CCD cubes but you can use /work/users/jshinaman/Support/RM_4741/PSP_009521_2065_RED6.balance.cub as test data for expediancy, and refer to Sample 13664.7 Line 39488.1, LAT -12.5331 LON 178.054 in the final "jittered" and FX difference data.
There are two main sub working directories containing the processed data representing the two different versions of ISIS:
Please see me for questions and information on processing steps and working data.
#35 Updated by Jesse Mapel 10 months ago
- Assignee changed from Ian Humphrey to Jesse Mapel
I have been continuing the investigation by Jac and Ian on this problem. Here's a brief update on what I've found so far.
This is not caused by a code change. I tested this on Fedora 21 with version 3.4.13 and 3.5.0 and the results were identical. They matched the results for 3.5.0 on other systems.
The tiling is caused by how noproj processes images. It operates on 128x128 pixel tiles and then breaks them down into potentially 8x8 tiles.
There is a very small difference between different OS versions when it comes to going from a ground point to a pixel location on the edge of a HiRISE ccd, on the order of 10-8 pixels in ESP_050349_1980_RED5. The difference appears to depend upon the resolution of the image. This manifests in a slight shift for a processing patch in noproj. This is why the differences of the outputs looks like a patchy difference filter.
I am still looking into exactly where the difference going from ground to image coordinates starts. I will post another update here once I find it.