FINDIMAGEOVERLAPS Consumes Huge Quantities of RAM
I'm trying to use FINDIMAGEOVERLAPS on about 750 polar CTX images. I could've sworn I had seen something in the documentation about a known issue with the program that it would fail where there were large numbers of overlaps, but I can't find that now, so perhaps it's a false memory or that issue has been fixed -- will let you know in a separate bug report if not. But right now, it's been running overnight, has so-far produced a 20.7 MB overlaps file, and is currently consuming about 25GB of RAM. It's only going up, and I know from past experience cutting up this region that I should expect at least an overlaps file 3x larger ... meaning RAM usage might go up to 75GB, maybe.
I'm reporting this because I'm unsure if this is a bug, and if not, you might want to look into how the program operates. I am not nearly the level of programmer that you have working on this software, but I don't understand why large amounts of RAM should be necessary for this program: Shouldn't it load one image, compare with all other images and output the info, then go to the next, compare, write, next, compare, write, etc.? As in, why should it need to store huge amounts of data into memory?
#2 Updated by Stuart Sides 7 months ago
- Target version set to N/A
The program does indeed use large amounts of memory, and for large data sets it will run a long time. The algorithms in the program are somewhat brute force, but were changed from exactly what you described so it would be faster.