Project

General

Profile

Bug #4741

hijitter and noproj on hirise images produces DN differences from last release

Added by Audrie Fennema 8 months ago. Updated 8 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Applications
Target version:
Impact:

No changes were made

Test Reviewer:
Story points:
4

Description

I'm seeing differences (in some cases significant) between cubes processed with ISIS 3.5.0 and 3.4.13 hijitter and noproj apps. I don't know if it's a bug or some underlying software change? Neither app lists any updates between versions. As an example of what is happening, I've uploaded a screen shot of a difference cube (3.4.13 subtracted from 3.5.0) of output from hijitter. There is some weird blockiness and you can see underlying surface features. I'm not sure what impact this would have on DTM production, but it doesn't seem good.

Let me know what other information I can provide.

Thanks,
Audrie

Screen Shot 2017-04-07 at 3.03.12 PM.png View (353 KB) Audrie Fennema, 2017-04-07 03:05 PM


Related issues

Blocks ISIS - Bug #4606: socetframesettings fails on Galileo SSI images Blocked

History

#1 Updated by Tammy Becker 8 months ago

  • Status changed from New to Acknowledged

#2 Updated by Tammy Becker 8 months ago

  • Target version set to 3.5.1 (Sprint 1)

#3 Updated by Trent Hare 8 months ago

To add in a comment here. When the difference was applied and you begin to see "structure", this seems to be indicative of a single pixel shift across the different images. This looks to be the case in the visible blocks (tiles) more than the rest of the image though which is odd. Obviously we would expect it to be same across the image. Thus of course our "tiling" scheme is something to look at for changes. I will assume you are using the same resampling method across both which could have similar "structure" effects. But something else for us to look into also.

#4 Updated by Stuart Sides 8 months ago

  • Subject changed from ISIS 3.5.0 hinoproj and hijitter diffences to ISIS 3.5.0 hinoproj and hijitter differences

#5 Updated by Stuart Sides 8 months ago

Audrie,
Can you please supply the exact command lines you are using?
Stuart

#6 Updated by Audrie Fennema 8 months ago

Stuart,

Here are examples of the command lines I used. I've run several different HiRISE images through, with similar results. Let me know if you need anything else.

Thanks,
Audrie

noproj to=/tmp/PSP_010453_1675_pnodex:6343/PSP_010453_1675_RED0.noproj.cub source=frommatch from=/tmp/PSP_010453_1675_pnodex:6343/PSP_010453_1675_RED0.balance.cub match=/tmp/PSP_010453_1675_pnodex:6343/PSP_010453_1675_RED5.balance.cub

hijitter regdef=/opt/usgs/isis3data/mro/calibration/hijitreg.p1745.s3070.def tolist=outlist_pnodex:6616.txt jitterck=/HiRISE/Users/audrie/pipe_dev_sandbox/HiRISE/Data/HiJACK/PSP/ORB_010400_010499/PSP_010453_1675//PSP_010453_1675.jittery.bc fromlist=inlist_pnodex:6616.txt jitter=/HiRISE/Users/audrie/pipe_dev_sandbox/HiRISE/Data/ResolveJitter/PSP/ORB_010400_010499/PSP_010453_1675//PSP_010453_1675_jitter.txt

#7 Updated by Stuart Sides 8 months ago

  • Subject changed from ISIS 3.5.0 hinoproj and hijitter differences to hinoproj and hijitter differences

#8 Updated by John Shinaman 8 months ago

Hi Audrie,

Can you please supply the "jitter" input .txt file you used in your proceesing?

Thanks,

Jac

#9 Updated by John Shinaman 8 months ago

  • Test Reviewer set to John Shinaman

#11 Updated by John Shinaman 8 months ago

Hi again Audrie,

I just want to clarify my last posting. The file I need is the table of pixel offsets used in the "jitter" parameter. You can either upload the actual file here, or if that becomes a problem, copy and paste the contents here in a post. We're getting ready to work on this problem and we appreciate your assistance.

Thanks!

Jac

#12 Updated by Audrie Fennema 7 months ago

Sorry Jac. Hadn't checked in a while. Uploading didn't work. I'll post contents.

Thanks,
Audrie

#13 Updated by Audrie Fennema 7 months ago

Jac,

The site won't let me upload or post the contents of the file. The upload says it doesn't like the file extension. I've tried none, .txt, .dat. Is there an acceptable file extension? When I try to post the contents copy and pasted I get an error.

Any other ideas?

Audrie

#14 Updated by John Shinaman 7 months ago

Hi Audrie,

You can send me the file at jshinaman@usgs.gov

Jac

#15 Updated by Tyler Wilson 7 months ago

  • Story points set to 4
  • Assignee set to Tyler Wilson

#16 Updated by John Shinaman 7 months ago

Hi Audrie,

I'm trying to reproduce your error and I'm running into procedural issues in getting accurate results. In specific, I can't run "hijitter" after "noproj" because the latter is creating a new idealized "Instrument" group and moving the "CcdId" keyword out of it and into a new group called "OriginalInstrument". "hijitter" wants to see this keyword in the main "Instrament" group as a data identifier. Otherwise, it see's the data no as non-MRO and invalid. I'm not sure how you're getting past this step without similar issues. I suspect I'm missing something in the processing steps.

Are you running the ISIS3 released version of "hijitter" and "noproj" or hybrids? I presume from your post that you're using released versions, but I need to make sure. How are you producing the "balance" cubes? Is there another process between "noproj" and "hijitter". Can you please supply a detailed list of your processing steps from ingestion up through "hijitter" along with any pertinent ancillary input data aside from the original .img's? I've downloaded and ingested the entire CCD array of PSP_010453_1675_RED*.img data into ISIS, but I need to compare the processing steps from there.

Thanks!

Jac

#17 Updated by Audrie Fennema 7 months ago

Hi Jac,

When processing images you either run noproj or hijitter, but not both. Noproj is run on observations that don't have much jitter and therefore don't need to be jitter-corrected. Hijitter is run on observations that do need to be jitter-corrected.

I am running the released versions. The steps to create the balance cubes are long and complicated. I'll email you a link where you can download some balance cubes and ancillary files you will need to run hijitter.

Let's start with hijitter. Here are the steps I use:
1. Run spiceinit on each balance cube:
spiceinit spkrecon=TRUE shape=SYSTEM spkpredicted=FALSE cksmithed=TRUE from=PSP_010453_1675_XXXX.balance.cub spksmithed=TRUE

  1. Run hijitter:
    hijitter regdef=/opt/usgs/isis3data/mro/calibration/hijitreg.p1745.s3070.def tolist=PSP_010453_1675_outlist.txt jitterck=PSP_010453_1675.jittery.bc fromlist=PSP_010453_1675_inlist.txt jitter=PSP_010453_1675_jitter.txt

  2. I run all the adjacent cubes through hijitreg and then mosaic all the dejittered cubes together using the average sample/line translation with handmos. I'm guessing you won't need to do that? But let me know if you do and I can give more detail.

For noproj I'll give you a different observation so it doesn't get confusing, PSP_009521_2065.
1. Run spiceinit on each balance cube:
spiceinit spkrecon=TRUE shape=SYSTEM spkpredicted=FALSE cksmithed=TRUE from=PSP_009521_2065_XXXX.balance.cub spksmithed=TRUE

  1. Run noproj on each balance cube using RED5 as the match cube:
    noproj from=PSP_009521_2065_REDX.balance.cub to=PSP_009521_2065_REDX.noproj.cub source=frommatch match=PSP_009521_2065_RED5.balance.cub

  2. I run all the adjacent cubes through hijitreg and then mosaic all the noproj cubes together using the average sample/line translation with handmos. Same as before, I'm guessing you won't need to do that? But let me know if you do and I can give more detail.

Let me know if something is unclear or you need more info.

Thanks,
Audrie

#18 Updated by Ian Humphrey 6 months ago

  • Status changed from Acknowledged to Assigned

#19 Updated by Trent Hare 6 months ago

  • Related to Bug #4606: socetframesettings fails on Galileo SSI images added

#20 Updated by John Shinaman 6 months ago

  • Status changed from Assigned to Testing

#21 Updated by Tyler Wilson 6 months ago

  • Assignee deleted (Tyler Wilson)

#22 Updated by John Shinaman 6 months ago

  • Assignee set to Tyler Wilson

Hi Audrie,

We can recreate the problem. I'm curious, do you test each ISIS version update, or is this the first time you've done the comparison? Either way, it's definitely a difference. The DN values are very small so it may be "in the noise" of the processing, but in any event, we'll track it down.

Thanks for your input!

Jac

**** INTERNAL NOTES ****

data working directory: /work/users/jshinaman/Support/RM_4741 (this is an extensive directory structure, but it includes all the processed data, print files, and comparison data)

results data directory: /work/users/jshinaman/Support/RM_4741/RESULTS (this contains the FX difference cubes)

The anomaly is spread across several CCD cubes but you can use /work/users/jshinaman/Support/RM_4741/PSP_009521_2065_RED6.balance.cub as test data for expediancy, and refer to Sample 13664.7 Line 39488.1, LAT -12.5331 LON 178.054 in the final "jittered" and FX difference data.

There are two main sub working directories containing the processed data representing the two different versions of ISIS:
/work/users/jshinaman/Support/RM_4741/3.4.13
/work/users/jshinaman/Support/RM_4741/3.5.0

Please see me for questions and information on processing steps and working data.

#23 Updated by John Shinaman 6 months ago

  • Status changed from Testing to Acknowledged

#24 Updated by Tyler Wilson 6 months ago

  • Assignee deleted (Tyler Wilson)

#25 Updated by Audrie Fennema 6 months ago

Hi Jac,

At HiRISE we test every ISIS version update extensively. This is the first time I've ever seen unexpected differences with hijitter and hinoproj.

Thanks for all your help. We appreciate the support.

Audrie

#26 Updated by Ian Humphrey 6 months ago

  • Status changed from Acknowledged to Assigned
  • Assignee set to Ian Humphrey

#27 Updated by Stuart Sides 4 months ago

  • Target version changed from 3.5.1 (Sprint 1) to 3.5.2 (2017-01-31 Jan)

#28 Updated by Audrie Fennema 4 months ago

Hi Jac,

Just checking in to see if there are any updates or anything else you need from me?

Thanks,
Audrie

#29 Updated by Makayla Shepherd 4 months ago

  • Status changed from Assigned to In Progress

#30 Updated by Ian Humphrey 4 months ago

Hi Audrie,

We are still researching and attempting replicate the issue with a smaller data set.

I will keep you posted with any issues we identify related to this ticket.

-Ian

#31 Updated by Makayla Shepherd 3 months ago

  • Related to deleted (Bug #4606: socetframesettings fails on Galileo SSI images)

#32 Updated by Makayla Shepherd 3 months ago

  • Blocks Bug #4606: socetframesettings fails on Galileo SSI images added

#33 Updated by Makayla Shepherd 3 months ago

  • Status changed from In Progress to Acknowledged

#34 Updated by Jean Backer 2 months ago

  • Subject changed from hinoproj and hijitter differences to hijitter and noproj on hirise images produces DN differences from last release
  • Description updated (diff)

#35 Updated by Jesse Mapel 25 days ago

  • Assignee changed from Ian Humphrey to Jesse Mapel

I have been continuing the investigation by Jac and Ian on this problem. Here's a brief update on what I've found so far.

  1. This is not caused by a code change. I tested this on Fedora 21 with version 3.4.13 and 3.5.0 and the results were identical. They matched the results for 3.5.0 on other systems.

  2. The tiling is caused by how noproj processes images. It operates on 128x128 pixel tiles and then breaks them down into potentially 8x8 tiles.

  3. There is a very small difference between different OS versions when it comes to going from a ground point to a pixel location on the edge of a HiRISE ccd, on the order of 10-8 pixels in ESP_050349_1980_RED5. The difference appears to depend upon the resolution of the image. This manifests in a slight shift for a processing patch in noproj. This is why the differences of the outputs looks like a patchy difference filter.

I am still looking into exactly where the difference going from ground to image coordinates starts. I will post another update here once I find it.

#36 Updated by Jesse Mapel 25 days ago

  • Status changed from Acknowledged to In Progress

#37 Updated by Jesse Mapel 8 days ago

  • Status changed from In Progress to Resolved
  • Impact updated (diff)

Also available in: Atom PDF