Project

General

Profile

Bug #4657

cnettable is outputting extraneous information beyond the header

Added by Tyler Wilson 9 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Applications
Target version:
Impact:

Extra trailing commas have been removed for some cases. Postprocessing of output files from cnettable may be affected.

Software Version:
Test Reviewer:

Description

In both apps tests allowErrors and append, data in the comma-delimited text file is being output
past where the header ends. This is not the case for the default test case. A fix for this issue should
figure out why this is happening as well as verify that the data in the columns matches what should
be there from the column headers.

The header for the output is generated by three functions in this order:

  1. cnettable::IsisMain()
  2. ControlMeasure::PrintableClassData()
  3. CameraPointInfo::GetPointInfo(...)

History

#1 Updated by Tammy Becker 9 months ago

  • Status changed from New to Acknowledged

#3 Updated by Lynn Weller 8 months ago

  • Test Reviewer set to Lynn Weller

Tyler, could you provide an example of the issue you are seeing or verify what I see is the same problem?

My data are under /work/users/lweller/Isis3Tests/CnetTable/

I have mimicked the app tests and named output appropriately:

allowErrors: cnettable fromlist=images.lis cnet=testNet.net flatfile=out_allwErrors.csv allowerrors=true

append: cnettable fromlist=images.lis cnet=testNet.net flatfile=out_append.csv append=true

default: cnettable fromlist=images.lis cnet=testNet.net flatfile=out_default.csv

and additional test append and allowErrors: cnettable fromlist=images.lis cnet=testNet.net flatfile=out_append_allowErrors.csv append=true allowerrors=true

In default mode there appears to be an extra comma after the last field, including the header (which seems to be opposite what you are suggesting). I see the same behaviour for the append test. The allowErrors and append_allowErros output look fine to me - no extra comma at the end.

Is this what you are talking about?

On another note, the test data is sorely outdated (net file is PVL and from 2009!) and does not contain all of the fields that may be included in the output. All of the missing fields have to do with the bundle adjustment keywords. I think this is a major oversight that needs to be corrected. My test data are too big for the app tests so I will create a new network that contains two clementine images (similar to the test data, but not the same files) and a network of similar size that contains data for the missing fields. I'll add another note when that data are available.

#4 Updated by Lynn Weller 8 months ago

New images and a network are available to replace the current isis app test data. These completely mimic the current app data in that there are two clementine images and the network contains 10 points/20 measures. Please replace the current app data with the new and up to date data and recreate truth data.

Files can be found here: /work/users/lweller/Isis3Tests/CnetTable/NewAppTestData/
Input images are lub4618n.342_tr.cub lub4666n.342_tr.cub
Input network is testNet.net

#5 Updated by Stuart Sides 8 months ago

  • Status changed from Acknowledged to Assigned
  • Assignee set to Kaj Williams

#6 Updated by Kaj Williams 8 months ago

  • Status changed from Assigned to In Progress

#7 Updated by Kaj Williams 8 months ago

Tyler Wilson wrote:

In both apps tests allowErrors and append, data in the comma-delimited text file is being output
past where the header ends. This is not the case for the default test case. A fix for this issue should
figure out why this is happening as well as verify that the data in the columns matches what should
be there from the column headers.

The header for the output is generated by three functions in this order:

  1. cnettable::IsisMain()
  2. ControlMeasure::PrintableClassData()
  3. CameraPointInfo::GetPointInfo(...)

#8 Updated by Kaj Williams 8 months ago

New data was added to the "append" test case but old data was left for the other two test cases for backwards compatibility.

#9 Updated by Kaj Williams 8 months ago

  • Status changed from In Progress to Resolved
  • Impact updated (diff)

#10 Updated by Lynn Weller 7 months ago

  • Status changed from Resolved to Feedback

I've just tested the changes and there don't seem to be any. My output from /work/projects/isis/latest/m04657/isis/ is identical to the output created using /usgs/pkgs/isis3production2017-03-28/isis/. Other than the new app test, was anything else done to this program?

Hi Lynn, I think the way the bad behavior was triggered was when the append option was being used.
Can you test that one?

#11 Updated by Kaj Williams 7 months ago

Lynn, yes, cnettable.cpp was modified.

#12 Updated by Lynn Weller 7 months ago

Sorry, I'm not seeing any changes at all between/usgs/pkgs/isis3production2017-04-25/isis and /work/projects/isis/latest/m04657/isis.
To clarify, when I say I'm not seeing any changes, I mean specifically in the output files that are generated.

I see the problem even in default mode where append=false and allowerrors=false.
Here are my tests (same as in previous post) with a couple more to test append=true differently
(commands in bold result in an extra comma at the end of every line; only when allowerrors=true does the comma go away):

cnettable fromlist=images.lis cnet=testNet.net flatfile=out_allwErrors.csv allowerrors=true
cnettable fromlist=images.lis cnet=testNet.net flatfile=out_append.csv append=true
cnettable fromlist=images.lis cnet=testNet.net flatfile=out_default.csv
cnettable fromlist=images.lis cnet=testNet.net flatfile=out_append_allowErrors.csv append=true allowerrors=true

via fix version:
setisis /work/projects/isis/latest/m04657/isis/
cnettable fromlist=images.lis cnet=testNet.net flatfile=out_m04657fix_allwErrors.csv allowerrors=true
cnettable fromlist=images.lis cnet=testNet.net flatfile=out_m04657fix_append.csv append=true
cnettable fromlist=images.lis cnet=testNet.net flatfile=out_m04657fix_default.csv
cnettable fromlist=images.lis cnet=testNet.net flatfile=out_m04657fix_append_allowErrors.csv append=true allowerrors=true

not clear if I'm using append=true properly, so some more tests:
cp out_default.csv orig_out_default.csv
cnettable fromlist=images.lis cnet=testNet.net flatfile=orig_out_default.csv append=true
(additional lines added; still has extra comma at the end)

cp out_m04657fix_default.csv orig_out_m04657fix_default.csv
cnettable fromlist=images.lis cnet=testNet.net flatfile=orig_out_m04657fix_default.csv append=true
(additional lines added; still has extra comma at the end; identical to orig_out_default.csv production version)

It would be useful if the original poster chimed in as it was never clear to me what they identified as the problem. As one who uses ISIS regularly and generates many flat files (csv), a comma after the last column doesn't make sense, so that's what I identified as the problem.

#13 Updated by Kaj Williams 7 months ago

The modified file (cnettable.cpp) is in
/work/projects/isis/latest/m04657/isis/src/control/apps/cnettable

I see the changes when I do the following:
vsdiff cnettable.cpp /usgs/pkgs/isis3/isis/src/control/apps/cnettable

Also, doing a "svn status" within tsts, I see the following modified files:
M allowErrors/Makefile
? allowErrors/cnettable.csexe
M append/Makefile
? append/cnettable.csexe
? append/input
? append/truth
M default/Makefile
? default/cnettable.csexe
? default/input
? default/truth

#14 Updated by Kaj Williams 7 months ago

Ok, Lynn, please try it again!

#15 Updated by Lynn Weller 7 months ago

Thanks Kaj - looks more like I was expecting. The extra comma at the end of each line is now gone (for the specific conditions I previously noted).

You might want to set the Status back to Resolved to get Tyler's attention. As the author of the post it is ultimately up to him to review and close the post.
Everything looks good from my perspective though.
Thank you!

#16 Updated by Kaj Williams 7 months ago

  • Status changed from Feedback to Resolved

#17 Updated by Tyler Wilson 7 months ago

Hello Kaj,

The original bug for this ticket appears to have been solved by updating the test data. The trailing comma bug was something I did not know about, but am glad to see it was addressed. Sorry about
the delay in getting back to both you and Lynne. The email notifications for this ticket were getting dumped into a folder they shouldn't have been getting dumped in because one of my email filters had a mistake in it.

Anyway, great job! I am closing this ticket now.

#18 Updated by Tyler Wilson 7 months ago

  • Status changed from Resolved to Closed

#19 Updated by Stuart Sides 4 months ago

  • Target version changed from 3.5.1 (Sprint 1) to 3.5.1 (2017-08-08 Aug)

Also available in: Atom PDF