Going by the information supplied on the processing involved with the HiRISE images, (hi2isis, hical, histitch, spiceinit, spicefit, noproj, hijitreg, and handmos), I know for certain that findimageoverlaps will not work following 'handmos'. The serial number is constructed from keywords in the Instrument Group on the input image labels...when any ISIS3 image is mosaicked using 'handmos', 'automos', and 'mapmos'...the Instrument Group is removed from the output labels, which is why you are seeing the 'invalid serial number' message.
Internally, we have had success with findimageoverlaps with HiRISE data following 'histitch, spiceinit and spicefit'. If even possible, I don't think that we have established a procedure for images post noproj and hijitreg.