specfix Documentation

specfix - Removes noise spikes from spectral cubes
This program removes noise spikes from spectral cubes.
There are several steps taken during this process.

The first step is to examine the entire cube to locate
and flag low-average spectra; that is, spectra in
the cube which, when the values for each pixel in the
spectrum are summed and divided by the number of pixels
in the spectrum, are less than some tolerance (ASETOL).
Each of these low-energy spectra are flagged using a
backplane set up for this purpose (backplane name is
SPECFIX).  Locations in the backplane with -2 indicate
the spectrum was flagged as low energy. These
spectra will not be 1) filtered or 2) used for any
statistical calculations in further steps.

The next step is to filter the valid spectra in the
cube. This is done in the following manner. A brick is
convolved through the entire cube. The dimensions of
the brick are defined via the DIMS parameter (e.g.,
DIMS=(7,5,20) indicates the brick will be 7 samples
by 5 lines by 20 bands). For each brick, the average
(G) of each spectrum in the brick is calculated. Using
the above example for DIMS there would be 35 (7x5) G
values, each calculated using the 20 corresponding
pixels in the spectrum. Next, each spectrum is
normalized using it's associated average G. This
procedure generates a normalized brick, which in turn
is used to calculate a normalized average (H) and
standard deviation (SIGMA) of each band in the brick.
Again, for the above example of DIMS there would be
20 normalized averages and standard deviations.

At this stage the statistics are in place to determine
if there are spikes in the target spectrum. The target
spectrum is typically the spectrum at the center of the
brick (not true for target spectra near the edge of a
cube).

Each pixel (A) of the target spectrum is compared
against the corresponding H and SIGMA in the following
manner:

	DIFF = | A - G * H |
	TOL1 = | G * Q * SIGMA |
	TOL2 = P * Ptab

where

	Q and P are user input tolerances and
	Ptab is an optional noise spectrum
	obtained via the PFILE parameter. (If
	PFILE is not specified, then Ptab=1.0
	is used.)

If DIFF exceeds both TOL1 and TOL2 the pixel A is
considered a spike and will be replaced with either
G * H or with a NULL special pixel value (depending
upon the value of REPNULL).

Each time a pixel is replaced in a spectrum, the
corresponding location in the SPECFIX backplane is
incremented. Thus the values in the backplane will
represent either low energy spectrum (-2) or a
number indicating the number of spikes found in the
spectrum.  In addition, if the spectrum is not marked
as a low energy spectrum, then the backplane value
for a target pixel location is incremented each time
its brick is found to not have enough valid pixels.
(See VFRAC parameter.)

Also, if the recursive option is selected, each time
a pixel is replaced in a spectrum, the values for G,
H, and SIGMA will be recalculated.

Programmer: Jeff Anderson, U.S.G.S., Flagstaff, AZ

Input cube file name
  (default extension is .cub)

NONE

Output cube file name
  (default extension is .cub)

NONE

Brick Dimensions
  (Samples,Lines,Bands)

NONE

Average Spectra energy tolerance

NONE

Minimum valid brick fraction

0.5

Brick increment in band
  direction

--

Standard deviation tolerance

4

Absolute tolerance

NONE

File listing of filter pixels
  (no default extension)

"specfix.prt"

Replace spikes with NULL
  (YES or NO)

YES

Use filter pixels in next
  calculations (YES or NO)

YES

P tolerance data file
  (no default extension)

--

ADDITIONAL NOTES:

Parm	Description
FROM	Input cube file name. If the file extension is omitted, then ".cub" will be assumed.
TO	Output cube file name. If the file extension is omitted, then ".cub" will be assumed.
DIMS	Filter dimensions (samples,lines,bands) of the brick used for statistics, where samples and lines are odd integers between 3 and 9, inclusive, and bands is between 3 and MAXBANDS, inclusive. The normalization of each spectrum is done over bands. Example DIMS=(3,5,15).
ASETOL	Average spectra energy tolerance. Spectra with average energy below this tolerance will not be included when performing noise removal. That is, they will not be 1) filtered or 2) used for statistical purposes. If the spectrum at a spatial location is below the average spectra energy tolerance, then the corresponding location in the SPECFIX backplane is set to -2.0.
VFRAC	Fraction of brick that must be valid in order for filtering to occur. Invalid values can be caused by 1) low average energy spectra or 2) special pixel values. Valid values for VFRAC are between 0.0 and 1.0 and the default is 0.5 which equates to 50% of the brick. When examining a pixel, the corresponding location in the SPECFIX backplane is incremented by 1000.0 if the pixel's brick does not have enough valid values (provided a backplane value of -2.0 does not already mark the spectrum as being below the average spectra energy tolerance ASETOL).
KDEL	Defines the movement rate of the brick in the band direction. The brick convolves over the entire cube in the line and sample directions. However, it moves through the bands based on KDEL. KDEL must be no more than the brick size in the band dimensions (DIMS[3]). If KDEL is specified as the Null TAE parameter value then the brick movement increment will be the same as the brick size in the band dimension, i.e., DIMS(3).
Q	The Q test checks against a standard deviation tolerance. A pixel will only be replaced if it differs from the mean of the brick by more than Q sigma. Note that both P and Q tests must be satisfied before a pixel is replaced. Q should be smaller than: SQRT (DIMS(1) * DIMS(2) - 1) and is recommended to be about 1 less than this value.
P	The P test checks against an absolute difference between the mean of the brick and the pixel. A pixel will be replaced only if it differs from the mean by more than P times the corresponding value from PFILE. (If no PFILE is specified, then a PFILE value of 1.0 is used.) Note that both P and Q tests must be satisfied before a pixel is replaced.
LISTFILE	Listing file for information about changed/filtered pixels. If set to the Null TAE parameter value then the file will not be created. Otherwise the following information will be written to the file: samp, line, band, ovalue, nstd, A-GH Where samp, line, and band represent the location of the filtered noise ovalue represents the original value at the location give by samp, line, and band nstd represents the number of standard deviations the original pixel value was away from the mean (what the Q value was for each pixel). This value will always be greater than the value entered for Q. A-GH represents the difference between the original value (A) and the replacement value (GH).
REPNULL	This parameter specifies whether noise pixels are replaced by NULL pixel values (REPNULL=YES) or are replaced by an interpolated value (REPNULL=NO).
RECURSIV	This option is used to select the filtering method, either recursive (RECURSIV=YES) or non-recursive (RECURSIV=NO). That is, a recursive filter will use "fixed" (corrected) pixel values for further tests and calculations, whereas a non-recursive filter will only use the original pixels from the input cube.
PFILE	This is used to specify a file that contains a tolerance value for each band when using the P parameter. The file must have exactly the same number of tolerance values as there are bands in the cube. The beginning of the file can contain comments that will be ignored until a line with C_END occurs. Note however a line containing C_END must occur in the file. The tolerance values must start on the next line and must be an integer value followed by a real value. The two values must be seperated by white space. The integer value represents the band number (which is currently ignored) and the real value will be used for the P tolerance check for each band. Again there must be the same number of tolerance values as bands in the cube. The value given by P will then be multiplied against each tolerance value given in the file. If this file is not entered (Null parameter value is specified) then only the value P will be used when tolerance checking occurs for each band.