Project

General

Profile

Bug #991

Optimize 'Blend'

Added by Tammy Becker over 5 years ago. Updated 9 months ago.

Status:
Acknowledged
Priority:
High
Assignee:
-
Category:
Applications
Target version:
-
Impact:
Software Version:
Test Reviewer:

Description

Currently 'blend' is very slow when applied to a large list of images. The cause has been diagnosed on the original post (refer to Mantis #866).

This post will reflect the optimization effort.

Two people will be working on this Janet Barrett and Orrin Thomas.


Related issues

Related to ISIS - Recommendation #866: Look at blend program to see why it is so slow on large image lists and see how much time it would take to add improvements Closed

History

#1 Updated by Janet Barrett over 5 years ago

After consulting with Orrin on ways to improve the blend program, I have modified the blend program to read in its image data differently in order to reduce the I/O time. The blend program was originally using the Chip class to read in the image data one pixel at a time and doing a cubic convolution interpolation on each pixel. Orrin suggested that I use the Buffer class to read in the image data of the overlapping area with one read and bypass the interpolation of the data. The data does not need to be interpolated because it is already map projected.

I ran a test of the new blend program and it takes 1/3 the time that the old blend took (9.5 minutes versus 28.5 minutes on a list of 47 files).

The new blend program can be tested by setting your ISIS to /work/projects/isis/latest/m00991/isis. It would be good to evaluate the output from the new version of blend to see if it is still giving reasonable results.

I still have time left to work on adding other improvements that Orrin suggested starting with changing the way the weighting is being calculated (use the distance from the level1 image instead of the distance from the edge of the level2 image). I will continue working on this aspect of the program.

#2 Updated by Lynn Weller over 5 years ago

Janet,

I'm testing this right now on about 1000 themis images of mine. I will have to run the system version of blend in order to make a time and mosaic comparison of the output files to your updated version. Looks like I cleaned up some directories and removed the one test of blend I ran on themis data. Sometimes I'm too quick to clean up! I'll let you know when I have some results.

Lynn

#3 Updated by Lynn Weller over 5 years ago

Here are the time results of testing the same dataset using the system version of blend versus the updated version:

isis3production2012-07-31
ConnectTime = 26:09:49.0
CpuTime = 23:42:47.1

/work/projects/isis/latest/m00991/isis/bin
ConnectTime = 15:33:54.0
CpuTime = 07:09:34.5

I'm not sure why the connect time is so long for the updated version. I started the process around 5pm on Monday, but even that late in the day it's possible that the system I was on was being hammered.

It will take about 1.5 hours to create the mosaic. If the two don't match for some reason I will report back, otherwise assume the mosaics are the same.

#4 Updated by Lynn Weller over 5 years ago

I have been able to finally create and view mosaics and have found the system version of blend and the updated version of blend vary quite a bit. In each case I passed the program the same list of images to start with (projected, sigmastreched, same order as well) and used a stop=100. Then I ran automos on each using the same image order, with priority=ontop and let the range default.
I linked and synced in qview and made sure the stretches were identical and blinked.

I can't decide if one is any better than the other, but clearly they are different.

My data are under
/work/projects/themis_control/lweller/Lunae_Palus/NightIR/BlendTest/
In the top level directory are the blended files created using /work/projects/isis/latest/m00991/isis/bin along with a mosaic named LunaePalus_NightIR_Controlled_Blend.cub.

There is a subdirectory called System/ where isis3production blend was run. The blend files from that run are there as well as the mosaic LunaePalus_NightIR_Controlled_Blend_System.cub.

I renamed the system blend files to filename....system.blend.cub so that the individual files could be compared and it would be clear who is who. If you take a look at I17308026RDR for each version you will notice quite a difference (I pulled this one up randomly). A quick glance at the particular file leans me towards the system version of blend, if I had to pick one. Take a look to see what I mean.

I ran cathist on these files to see if I can pick up any difference and the only thing I noted was that blend is not making it into the history (for either version). We probably should address that. My print files verify that everything is the same.

Sorry the news wasn't better.

Lynn

#5 Updated by Janet Barrett over 5 years ago

Lynn,

Thank you for doing so much testing on the new version of blend. The products you are getting from the old/new versions of blend have the same blend algorithm run on them. The main difference is that the new blend program is no longer doing interpolation on the overlap regions before the blending occurs. Orrin and I ran some tests and created a blended file with the new version of blend and then ran the sharpen program on the result. We compared this result with the old blend process and we thought that the new result was better. However, I think we lucked out with the image that we used for testing because it wasn't nearly as noisy as the 17308026RDR image.

I would like to modify the sharpen program to allow the user to choose the type of interpolation to run on a file so we can further investigate the viability of separating the interpolation process from the blending process. The "sharpen" program is very limited in its capability because it only allows for a high pass averaging of pixels in a user defined boxcar. The program would be more useful if it allowed the user to specify the type of interpolation to be performed: bilinear, cubic convolution, nearest neighbor. I will check first with Orrin to see if he thinks this is a reasonable idea.

Janet

#6 Updated by Janet Barrett about 5 years ago

I have spent quite a bit of time trying to get the interpolation built back into the new version of blend. This has been more difficult than I originally thought it would be. The interpolation in ISIS3 is tied very closely to the Cube class and I tried to extract it so that it would not be as I/O intensive. I tried to wrap up the project last week, but I now have to move on to this year's priorities. I wasn't sure if this project is still covered under PDS this year or not.

I was able to come up with a test case of 4 images that isolate the problem. I have a version of blend built in the /work/projects/progteam/jbarrett/blendwithchip/isis/src/base/apps/blend directory with a file list containing the 4 images. The problem can be demonstrated by running the blend in this directory and using a fromlist=file_list.sub. I built the system version of blend at /work/users/jbarrett/blend so that I could compare images and find differences.

#8 Updated by Anonymous over 4 years ago

  • Target version deleted (150)

#9 Updated by Tammy Becker 9 months ago

  • Status changed from Assigned to Acknowledged
  • Assignee deleted (Janet Barrett)

Unassigned Janet Barrett

Also available in: Atom PDF