Control network memory usage calculator/estimator
No impact on any ISIS objects or applications.
It would be useful to have some mechanism (algorithm/script/application) by which a user can estimate how much memory control network applications require to operate on any given network and associated images. This is essential if/when internal users utilize interactive sessions to the nebula hpc and need to designate the amount of memory to access the proper resources (and not have their session killed due to insufficient memory allocated). Larger networks (hundreds of thousands points and millions of measures) in particular use a large amount of memory (my Themis IR global network uses over 30G of ram and I believe the messenger network uses over 200G) but even the more moderate sized networks (tens of thousands of points and hundreds of thousands of measures) require more than a couple G of ram (which currently is the default allocated for a cluster job). I have no idea how to guesstimate how much memory an application like cnetstats, cnetextract, qnet or jigsaw require when running these programs and more often than not exceed allowable memory on the cluster resulting in a cancelled job and lost time. Is there a way a user can estimate memory needs to avoid this sort of thing and ask for appropriate resources?
Not sure what Tracker or Category this request falls under (please change if there is something more appropriate), but I have selected High Priority because this is an ongoing issue that is not going to go away and which currently affects a number of users (not just on the cluster - astrovm4 just about crashed when over 75% of its memory was being used by jigsaw on a portion of the messenger network; the user just simply did not know this could happen) and no doubt others in the near future.
It's possible jigsaw requires a slightly different calculation to include the number of images in the network, but I honestly don't know. Help please!
Running on new OS using latest version of isis (isis3production2017-04-25) though old OS and older versions of isis demonstrate the issue as well.
I'll be happy to point a developer to a larger network if/when necessary.
#8 Updated by Lynn Weller over 1 year ago
Below are some varying sized networks to choose from. You should be able to copy the network and its associated image file list to your working directory without having to grab images. All of the images in the lists provided below reside in the same location.
Moderately sized network (but by no means huge):
Pretty big network (1/4 of a planet):
Very big network (global):
I know that jigsaw may use about 2Gb of memory for the moderately sized network; I have to allot about 5G of memory for cnetedit for the pretty big network; and I need to allot about 30G of memory to run jigsaw (without updating images...which I think doubles the memory) on the global network. Also noted that I need 15Gb of memory to extract from the very big network.
#9 Updated by Lynn Weller over 1 year ago
To start by simplifying things, you might try seeing how much memory is required to run cnets stats on the Pretty big network (the middle one in the list).
Here's a command you could use:
cnetstats fromlist=Themis_DayIR_Merged_GroundNet_2017Apr17_Longitude180to270.lis cnet=Themis_DayIR_Merged_GroundNet_2017Apr17_Longitude180to270.net create_image_stats=yes image_stats_file=ImgStats_Themis_DayIR_Merged_GroundNet_2017Apr17_Longitude180to270.csv
Note, this might take 10+ minutes to run. I'm not sure, but it should be less than 30 minutes.
And for something to be ultimately useful, we would sort to need to have a way of guesstimating memory based on the size of the network is possible (is there a way to guess memory need for certain programs based on the number of points and measures?).
#10 Updated by Lynn Weller over 1 year ago
Here's another very big network you can test with your calculator:
It's been a while since I've worked with this one, but I believe it will need something to the tune of 30Gb of memory.
#18 Updated by Adam Paquette about 1 year ago
The script for running this is in /usgs/cdev/contrib/bin/ and is called cnetcalculator.py. To run the script use:
python cnetcalculator.py file1, file2, ... ,filen
You can optionally include
--offset X, where X is the value in Gigabytes to offset the estimation. This is defaulted to 2 to be on the safe side.