Overview of ISIS Architecture
Isis 2 Documentation
Overview of ISIS Architecture
ISIS combines elements of both classical image processing and spectral analysis, as well as new techniques developed specifically for the analysis of image cubes. Standard image processing techniques can be applied to the individual spatial-spatial planes of image cubes. Spectral analysis techniques can be applied to the individual spectra of the cubes. In addition, a new set of analysis programs, designed to take advantage of the combination of image and spectral information, can be used to analyze image cubes. ISIS will also continue to incorporate the functionality of digital cartography found in the Planetary Image Cartography System (PICS).
Various other pages of the web-based documentation contain Notes on ISIS Documentation (including instructions on producing a hard-copy program reference manual), Notes on Use of the IDL Display Programs (including required set-up), a Display Program Summary, summary lists of currently available application programs (which consist of the program name and a one-line description), a summary of available spectral library data, suggestions on techniques for exchanging ISIS data with other software systems, etc. The program list is available as an alphabetical list and also as a list organized by functional categories. Clicking on a program name will display a page that contains the complete documentation for the program. Additional documentation web pages describe overall image processing procedures. The Release Notes pages describe the changes in the latest and previous releases of ISIS.
When ISIS is properly installed, the ISISVERSION environment variable contains the date of the ISIS release in the format YYYY-MM-DD.
The main document that provides detailed information for ISIS application programmers is the ISIS System Design document (ISD), which describes the system design and the programming environment that is provided for developing application programs.
% tae
- or -
% taetm
This will give you the "TAE>" prompt, indicating that TAE is
ready to accept a command. TAE commands are not case sensitive, although
the file names sent to ISIS programs are case sensitive and the names of
ISIS programs must be entered as lower case. TAE commands can also be
abbreviated using only enough characters needed to uniquely specify the
command.
TAE>t ratio
TAE will then display a list of the parameters for the program,
along with a brief description of each parameter and its initial
default value (if any). The "?" prompt at the bottom of the screen
indicates that TAE is ready for a tutor mode command. Since most
programs have more parameters than can be displayed on one "page", typing
the RETURN key will display the next page of parameters. You can set the
value for a particular parameter by typing the name of the parameter
(which may be abbreviated) and the desired value, which must be separated by
an "=". For example, to specify a file called test.cub as the input
file for the "ratio" program, type the following:
?FROM = test.cub
Alternatively, typing the down-arrow or up-arrow keys will select
the "next" or "previous" parameter for interactive editing of the
current parameter value.When you have specified all the parameter values, typing "RUN" (or simply "r") will run the program with the current parameter values. If you decide you don't want to run the program, typing "EXIT" will return to the "TAE>" prompt.
TAE>dims = 200,300,40 -or- TAE>dims = (200,300,40)
A string parameter value can usually be specified by typing the value without quotes. However, double quotes are required if the string value contains a comma (so that it will not look like an array of values). For example:
TAE>from = myfile.cub -or- TAE>from = "myfile.cub"
Some string parameters have a defined list of valid values. In this situation, it is only necessary to specify enough characters to uniquely identify the value, and the characters are not case-sensitive. For example, the MODE parameter in the "dsk2dsk" program has valid values "GET_RANGE" and "SET_RANGE". This parameter can be set as follows:
TAE>mode = GET_RANGE -or- TAE>mode = get -or- TAE>mode = g
?> parameter = --
?SAVE test
will save the parameters in a file called "test.par". The RESTORE
command (with an optional file name) will restore a set of
previously-saved parameter values.When you RUN a program in tutor mode, the current parameter values are automatically saved in a file called last.par. If an error is detected, you can then tutor the program and use the "REST last" command to restore the complete set of parameter values. You can then correct a parameter value without having to re-specify all the other parameter values.
TAE>labels from=test.cub print=all
This command will run the program with the specified values for the
FROM and PRINT parameters. The default values will be used for
any parameters that are not specified. If parameters are specified
in the same order as the parameter definitions for the program, then
the parameter names may be omitted. Thus, the following would be
equivalent to the above command:
TAE>labels test.cub all
TAE>ush grep word file.txt
When in tutor mode, typing "HELP" will display general help information on using tutor mode. Typing "HELP *" will display the overall description of the current program that is being tutored. (This is the same as the program description that is contained in this User's Manual.) Typing "HELP parm" will display help information on the specified parameter. Typing "EXIT" will exit from the display of the help information.
Note that there may well be conflicts with ISIS applications and system programs and applications with the same names. For example, we know that $ISISEXE/convert conflicts with /usr/X11R6/bin/convert on some Linux systems.
For Linux systems, the /proc file system *MUST* be in use in order for this feature to be utilized. See your Linux system adminstrator if you have problems or questions about this.
When running ISIS applications from the shell, you will need to be conscious of shell substitutions. The easiest way to avoid problems is to enclose arguments in single quotes so shell substitution is not performed. Here is an example of how to run an ISIS application from the shell:
% zero to=zero.cub dims='(100,100,1)'
In this example, the ISIS "zero" application creates a floating point ISIS cube file with 100 samples, 100 lines and 1 band in Band SeQuential (BSQ) storage order and fills it with 0.0. (The "%" should not be typed as it is the Unix prompt.)
All ISIS applications exit with status indicating success or failure. The shell can test the exit status of any ISIS application and take appropriate action. An exit code of 0 indicates success. Error conditions are normally returned as negative values.
The "isisparam" program provides the capability to pass values back from an ISIS application to a Perl or shell script. Although this program is not invoked through TAE, there is a special TAE PDF file that contains a description of the program:
TAE> t isisparam
?> help *
When TAE is started, the file $TAEPDF/slogon.pdf is executed. One of the things that this file does is to define a list of directories to be searched when the user runs a program. TAE searches these directories until a ".pdf" file for the program is located. (This file contains a description of the parameters and the documentation for the program.) TAE then runs the executable program from the same directory in which it found the ".pdf" file. (If the ".pdf" file is a procedure PDF file, then there is no corresponding executable and the commands in the file are executed.) The directory search list starts with the current working directory and then includes several other directories. Typing the "show-lib" command at the TAE prompt will list the current search directory list. The following is the list set up in the standard ISIS installation:
TAE>show-lib
User Library ($USERLIB):
.
Application Libraries ($APLIB):
$USERTAEBIN
$DEVISISEXE
$ISISEXE
$PICEXE
$ISISR/local/exe
$TAEBIN
System Library ($SYSLIB):
$TAEPDF
This allows TAE to find the standard ISIS programs in the $ISISEXE directory.
However, note that the $DEVISISEXE directory is searched before $ISISEXE. This
allows $DEVISISEXE to be used for developmental versions of programs. Programs
in this directory are not included in the normal ISIS distribution, but this
directory could be used for holding local developmental programs at an ISIS
installation site. If you plan to use this feature, note that you will need
to create the $DEVISISEXE directory and set its access privileges according
to the needs of your installation.
In addition, the $USERTAEBIN directory is searched before $DEVISISEXE. The standard ISIS installation procedures do not define the $USERTAEBIN environment variable, so the user can define this to be the directory that contains special programs being used for a particular project. Note that different directories could be used for holding platform-specific versions of the programs. Setting $USERTAEBIN to the appropriate directory could then select the set needed for running on a particular platform.
Note that the last thing that slogon.pdf does is execute the user logon PDF, ulogon.pdf. This PDF is also located in $TAEPDF and it is currently empty. If a user would define $USERTAEBIN to be a directory that contains a customized ulogon.pdf file, then this version would be executed at startup. This would allow greater flexibility for users and programmers.
When ISIS programs are invoked directly from the Unix shell prompt, a similar search of a list of directories is performed. In this case, the $PATH environment variable contains the list of directories to be searched. Note that $USERTAEBIN must be defined prior to execution of the isispgmdef initialization file in order for the $USERTAEBIN directory to be included in the search path. See the ISIS Installation Guide for details.
Since UNIX TAE does not provide true batch capabilities, an ISIS program called "submit" has been written to help you start the UNIX batch once the TAE "batch" file is created. For instructions on how to create a TAE "batch" file for running a single ISIS program or for running a procedure PDF that invokes a sequence of ISIS progorams, tutor the "submit" program:
TAE>t submit
Most ISIS programs include the USERNOTE TAE parameter, which does nothing other than allow the user to specify an arbitrary comment that will be recorded in the session log file and the cube file history entry.
The default file name for the session log file is the file "print.prt" located in the current default directory. If desired, the user can select a different file by defining the UNIX environment name ISIS_LOG_FILE to be the desired file name. This can be done by typing a command such as the following in the C shell:
% setenv ISIS_LOG_FILE session.log
or for the Borne shell:
% ISIS_LOG_FILE=session.log
% export ISIS_LOG_FILE
Note that this is an operating system command that is typed before
starting the TAE session. The specified file name may optionally
include a full path specification. Multiple concurrent sessions
are not permitted to use the same session log file. Thus, one of the
main reasons for specifying a different session log file is to allow
multiple concurrent sessions, e.g., two interactive sessions or an
interactive session along with a batch run.The session logging can be turned off by setting the log file name to be the UNIX null device, i.e., "/dev/null" (C shell).
% setenv ISIS_LOG_FILE /dev/null
Note that the "errors" program can be used to locate program runs in
the session log file that encountered errors.
To turn off printing to the terminal screen AND ISIS log file, the (C or T shell) user can issue the following command at the shell prompt:
% setenv ISIS_TERM_REPORT OFF
% setenv ISIS_LOG_REPORT OFF
This will suppress all but a record of the start and completion of each
program both on the terminal and in the log file. An example of what
this generates for output on both output text streams may be:
OBJECT = PLANSINU
END_OBJECT = PLANSINU
OBJECT = GEOM
END_OBJECT=GEOM
Note that by setting these environment variables to any other value, no value,
or if they are not defined at all will result in the normal
complete printing of all text to the target device. It is
perfectly acceptable to turn the terminal output off and log file
output on. This scenario is accomplished by the following setting
of the variables: (C Shell)
% setenv ISIS_TERM_REPORT OFF
% setenv ISIS_LOG_REPORT ON
This will generate full listings in the log file but suppress
output to the screen. Note that "OFF" and "ON" values may be
any combination of upper or lower case characters.NOTE that the usual reporting of program percentage completed will not be affected by these settings. They will also have no effect on those cases where applications write directly to the output streams without using the standard ISIS text output routine.
To report the values of the FROM and TO parameters, the user can issue the following shell command (C Shell):
% setenv ISIS_PARM_REPORT "FROM, TO"
Each parameter name may be separated by spaces, tabs, commas,
semicolons or colons, and the entire list of parameter names must be enclosed
by double quotes. A typical session log with the previous settings
would look something like:
OBJECT = PLANSINU
FROM = "/work2/user/file1.cub"
END_OBJECT = PLANSINU
OBJECT = GEOM
FROM = "/work2/user/file1.cub"
TO = "$WORK2/outfile1.cub"
END_OBJECT=GEOM
Second, when a program detects an error and calls the error facility, the first call will generate a dump of all the program input (TAE) parameters (housed in the HISTORY entry generated by each application program). This essentially is equivalent to exactly what the user would see if reporting to the terminal and log file where turned on. Subsequent calls to the error facility will simply record the current error message.
In most cases, the normal operating system conventions are applicable for determining the location of a data file, i.e., an explicit directory can be specified, but the file is assumed to be located in the current default directory if the user does not specify a directory.
% setenv NIMSDATA /work3/user/nimsdata
% taetm
TAE>labels FROM=$NIMSDATA/test.cub
for the Borne shell:
% NIMSDATA=/work3/user/nimsdata
% export NIMSDATA
% taetm
TAE>labels FROM=$NIMSDATA/test.cub
The environment variable cannot be defined after entering TAE. Defining an
environment variable before entering TAE will maintain the definition across
multiple TAE sessions run from the same C or Borne shell session.Another way to use environment variables is to define an environment variable in which the definition is an actual cube file name. The environment variable can then be entered for the TAE parameter value that specifies a cube file name in an ISIS program. For example in C shell:
% setenv MYCUBE /work2/user/temp/cubs/biglongcubefilename.cub
% taetm
TAE>labels FROM=$MYCUBE
The setenv command can use another environment variable in the definition.
For example, if MOONDATA were defined to be /work3/user/moon,
then the following setenv command could be used (C shell):
% setenv MYCUBE $MOONDATA/biglongcubname.cub
The ISIS I/O routines will use a two-step process when trying
to determine the actual disk filename to use. First, an intermediate
string will be formed by assuming that the supplied string is an environment
variable and by translating it into its defined value. If the user-supplied
string is not a defined environment variable, then the intermediate
string will be just the original user-supplied string. In the second
step, the I/O routines will check the intermediate string to see if
it has an extension, i.e., a ".<something>" on the end. If there is
no extension, then the default extension (".cub", ".isl", or ".tbl")
will be added. This will allow the user to omit the file extension on
the setenv command, e.g., the following will have equivalent results
(C shell):
% setenv MYCUBE /work3/user/filename.cub
% setenv MYCUBE /work3/user/filename
Note that the directory can be omitted from the
definition of the environment variable, e.g.:
% setenv MYCUBE filename.cub
When this environment variable is used, the current default directory will
be used as the location of the file.Also note that a directory can be combined with the reference to an environment variable. e.g. (C shell):
% setenv MYCUBE filename.cub
% taetm
TAE>labels from=/work3/user/$MYCUBE
When ISIS records the TAE parameters in the HISTORY entry and the
Session Log file, the exact string specified by the user will be
recorded. In addition, most programs will also record the translated
string that gives the actual disk file name that is to be used. This
will be recorded as a comment line delimited by "/*" and "*/". (The
translated string that is recorded will not include an assumed default
file extension.)
Some ISIS applications will read the ISISBUFMEM environment variable in order to determine the memory utilization. This variable should initially be determined by the system manager. (The amount of I/O intermediate work buffers assigned by the system manager is a function of the I/O memory available on a computer and the number of users using the system). However, if this is not enough memory for an application, the user can alter the allocation amount by setting the ISISBUFMEM environment variable as follows (C Shell):
% setenv ISISBUFMEM xxxxxxxx
Where xxxxxxxx = number of bytes to be used for the I/O
intermediate work buffers for an application. If the
ISISBUFMEM environment variable has not been set up, then
a default of 4 mega-bytes will be used.
% setenv ISISPROGRESS inc
where 'inc' is the increment you want (only values 1 thru 20 are valid).
% setenv ISISETA time
where 'time' is the amount of time in seconds to wait before
printing the time left (only values 15 thru 60 are valid). If the
program run in less than the number of seconds set then no
time status will be output.
A PDS or ISIS labeled file consists of two basic parts. The first part is the label, which is located at the beginning of the file. The remainder of the file is the data area, which contains one or more data objects. The label describes the structure and content of the file. Information in the label is stored in a "keyword=value" text format. Appendix A contains an example label for an ISIS cube file.
The QUBE files that are supported by the ISIS I/O routines are used for storing multi-dimensional data arrays. The maximum number of supported dimensions is six. A CUBE file is a special case of a qube file in which the number of dimensions in the data array is three. (See the ISD for more information on the generalized qube files.) Cube files are normally used for storing data produced by imaging spectrometers. Thus, two of the dimensions correspond to spatial coordinates and the other dimension corresponds to the spectral (wavelength) coordinate. The contents of the data values in a standard cube file can of course consist of things other than observations measured by an imaging spectrometer.
Each cube file contains at least two data objects, a HISTORY object and a QUBE object. A cube file may optionally contain "extra" data objects in addition to the HISTORY object and QUBE object. The standard ISIS cube-processing programs will propagate these "extra" data objects unchanged from an input file to an output file.
All ISIS programs automatically maintain the history information. When an application program reads data from an input file and writes processed data to an output file, it will copy the input file history information to the output file and then append a new history entry for the current program.
If there are several program runs that operate on a cube file without creating a new file, the label and/or history area can become full, causing a program failure. There are several ways to avoid this problem:
Some of the label keywords are contained within GROUPs. The most important of these is the BAND_BIN group, which describes the spectral (wavelength band) axis of the cube. These include a list of wavelengths (BAND_BIN_CENTER) and a list of the original instrument band numbers that correspond to the file band numbers (BAND_BIN_ORIGINAL_BAND). This keyword group might optionally include a list of instrument grating positions and detector numbers (BAND_BIN_GRATING_POSITION and BAND_BIN_DETECTOR).
______________________ .
/ / /| . ^
/ backplane / / | } suffix . /
/_ _ _ _ _ _ _ _ _/_ _/ | } region . /
/ / /| | . / d
/ / / |c | . / n
/ / / |o | . / a
/ core / / |r | . / B
/ / / |n | . /
/_________________/___/ e |e | . /---------------->
| | | d e|r | . | Sample
| | | i n | /| l . |
s | | |s a |/ / a . | L
p | | | l / / r . | i
a | core | |p /|/ t . | n
t | | | / / c . | e
i | | | / / e . |
a | | | / / p . |
l |_________________|___|/ / s . v
| bottomplane | | / } suffix .
|_________________|___|/ } region .
spatial ^^^ .
suffix region .
The AXES keyword specifies the number of dimensions (axes) in the core. For standard cube files, this value is always three. The CORE_ITEMS keyword has an array of integer values that specify the length of each axis of the core. These values are expressed as the number of pixels. The AXIS_NAME keyword has an array of literal values that specify the names of the axes. (A literal keyword value is similar to a string value except that it is either enclosed by single quotes or not enclosed by quotes at all. String keyword values are always enclosed by double quotes.) The order of the values in CORE_ITEMS and AXIS_NAME corresponds to the physical storage order of the data. (See below for details on physical storage orders.) The core data values are further described by the required application-owned keywords CORE_NAME and CORE_UNIT. (These are called "application-owned" because they are maintained by the applications code of the programs rather than by the ISIS system routines. The applications programs can set these keywords to any desired values.) The label also includes a number of keywords that describe the format of individual core pixel values. These are described in detail later in this document.
The SUFFIX_ITEMS keyword has an array of integer values that specify the number of suffix planes that exist along each of the three axes. The order of the values in SUFFIX_ITEMS corresponds to the physical storage order of the data file, i.e., the values in the array correspond to the values in the AXIS_NAME keyword. (See below for details on physical storage orders.)
In a manner similar to the core description keywords, the name of each suffix plane and the units of the data values are described in suffix description keywords of the form axis_SUFFIX_NAME and axis_SUFFIX_UNIT, where "axis" is the name of the axis that is extended by the suffix data, e.g., BAND_SUFFIX_NAME and BAND_SUFFIX_UNIT for suffixes that are backplanes. The "axis" value must correspond to one of the axis names listed in the AXIS_NAME keyword. The suffix description keywords all have array values, where the number of values corresponds to the number of suffix planes on the axis. This allows each suffix plane to have a description that is independent of all the other suffix planes. Note that the name of each suffix plane is required to be unique within each suffix region in a cube file. Additional suffix description keywords are described later in this document.
In a standard three-dimensional cube file, the names of the three axes are always SAMPLE, LINE, and BAND. The AXIS_NAME keyword has an array of values that list the names of the axes in the cube. The order of the names specifies the cube storage order in the file. The first axis is the fastest varying, and the third axis is the slowest varying. The standard ISIS cube-processing programs support the following three storage orders:
(SAMPLE, LINE, BAND) - Band Sequential (BSQ)
(SAMPLE, BAND, LINE) - Band Interleaved by Line (BIL)
(BAND, SAMPLE, LINE) - Band Interleaved by Pixel (BIP)
The CORE_ITEMS and SUFFIX_ITEMS keywords have array values. The order of these values corresponds to the order of the axes listed in the AXIS_NAME value.
In the physical file storage, suffix pixel data (if present) is "scrambled" in with the associated core pixel data. For example, in a BSQ storage order file, the physical cube storage in the file begins with the pixels in the first (top) line of the spatial-spatial image plane at the first wavelength band. This is followed by the sideplane pixel values that extend this line of core pixels. Next are the core pixels for the second line, followed by the sideplane pixels for the second line. After the last line of this first core image plane (and its associated sideplane pixels) come the bottomplane pixels associated with the first band. This is then repeated for the second through last bands. Finally, all the backplane data are stored after all the core data and associated sideplane and bottomplane pixels. Thus, in a BSQ file, all of the backplane pixels are stored together in a single contiguous block at the end of the cube data object. This means that backplanes can be added to a BSQ file without creating a new file because it only requires adding new data at the end of the file. (This is true only if the qube object is the last object in the file, which is usually the case since any "extra" data objects usually are located between the history object and the qube object.) However, adding backplanes to a BIL or BIP file requires creating a new file.
In general, almost all application programs are capable of processing all of the storage orders. The storage orders that can be processed and the efficiency of processing usually depend upon whether the program does its processing in terms of spatial planes or in terms of spectra. For example, the "ratio" program divides each spatial plane by a given spatial plane. This program is most efficient when processing a BSQ file, but BIL and BIP files can also be processed. A warning will be given if an application will be substantially slower because of storage order, but will continue with processing.
Note that the "transpose" program will convert a standard cube file from one physical storage order to any other physical storage order.
It is important to note that the standard ISIS cube-processing programs assume that values stored in a file are stored in formats that are native to the host computer on which the software is currently running. Thus, for example, there is no capability for "on the fly" conversion of "foreign" floating point formats to the native floating point format. This conversion would need to be performed by the "convert" program, which converts a file from a "foreign" format to the standard ISIS cube file format for the current host computer.
A single integer type code value is used to identify the various supported data types, i.e., combinations of CORE_ITEM_TYPE and CORE_ITEM_BYTES keyword values. This is described in more detail below.
--------------------------------------------------------------- | CONCEPTUALLY, A CORE PIXEL IS ALWAYS CONSIDERED TO HAVE | | A "REAL" NUMBER VALUE. | ---------------------------------------------------------------In standard ISIS cube files, these core pixel values can be represented in three different ways, corresponding to different numbers of bytes used to store the values in the file. When 4-byte format is used, the stored number is in the format of a native (for the current host computer) floating-point number that directly represents the "real" value of the pixel. The other two formats store a pixel value using 1-byte or 2-byte storage items in the file. For these, the stored value is considered to be an integer value and type conversion parameters (given by the CORE_BASE and CORE_MULTIPLIER label keywords) are used to convert the stored integer values into the "real" pixel values that are being represented. The "real" value being represented is determined as follows:
"real_value" = CORE_BASE +
(CORE_MULTIPLIER * REAL(stored_value))
From the point of view of the user, core pixels always appear to
have these "real" values, i.e., the user can never directly see the
1-byte or 2-byte file values before the application of the type
conversion parameters. In most cases, an application program will
automatically convert 1-byte or 2-byte input pixel values to floating
point and then perform its computations using these floating point
values. It will then automatically convert the computed values to
1-byte or 2-byte format for writing to an output file. Thus, most
application programs are capable of processing files that use any
of the three different formats.However, there are several aspects of the different storage formats that are important to the user. First, different data types result in different disk file sizes. Second, different data types allow different precision (number of significant digits) in the stored pixel values. 4-byte format uses the most disk space and allows the greatest precision. 1-byte format uses the least disk space and allows the least precision. The third important aspect of the different data formats is the range of values that can be represented. With 4-byte format, the representable range is the range of floating point values and is thus nearly unlimited for all practical purposes. For 2-byte or 1-byte format, the representable range varies depending upon the current values of the type conversion parameters. This is discussed in more detail below.
The following is a summary of the three standard pixel representation formats for core pixels:
Type CORE_ITEM_BYTES CORE_ITEM_TYPE Type Conversion
Code Parameters
1 1 UNSIGNED_INTEGER Yes
2 2 INTEGER Yes
3 4 REAL No
For 4-byte format the stored values are floating point values that
directly represent the pixel values. In this case, the CORE_BASE and
CORE_MULTIPLIER values are not used. However, they are still required
to be present in the label and should have values of 0.0 and 1.0.
In a manner similar to the core description keywords, the data type for suffix pixels is described in suffix description keywords of the form axis_SUFFIX_ITEM_BYTES and axis_SUFFIX_ITEM_TYPE, where "axis" is the name of the axis that is extended by the suffix data, e.g., BAND_SUFFIX_ITEM_BYTES and BAND_SUFFIX_ITEM_TYPE for the suffixes that are backplanes. The "axis" name corresponds to one of the axis names listed in the AXIS_NAME keyword. Each suffix plane within a single file can have a different data format. Thus, the values of these keywords are arrays. Each element of the array refers to a separate suffix plane. Type conversion parameters can also be used for some of the standard suffix data types. These are contained in keywords of the form axis_SUFFIX_BASE and axis_SUFFIX_MULTIPLIER, e.g., BAND_SUFFIX_BASE and BAND_SUFFIX_MULTIPLIER for the backplane suffixes. Each suffix can have different type conversion parameters, so these keywords have array values.
When a suffix pixel contains a numerical value, it is expected that in most cases it will be stored in one of three standard formats:
Type axis_SUFFIX_ axis_SUFFIX_ Type Conversion
Code ITEM_BYTES ITEM_TYPE Parameters
4 1 UNSIGNED_INTEGER Yes
5 2 INTEGER Yes
3 4 REAL No
The ISIS application programs will normally preserve the identity of special pixel values when converting pixels from one representation to another. For example, when converting from 2-byte format to 4-byte format, a 2-byte "Null" pixel will be converted to a 4-byte "Null" pixel.
For the 1-byte (8-bit) pixel data type, there are only two special pixel values: "Null" and "High Instrument Saturation." Thus, "Null", "Low Instrument Saturation" and "Low Representation Saturation" in 2-byte or 4-byte format are all converted the the 1-byte "Null" value. "High Instrument Saturation" and "High Representation Ssaturation' in 2-byte or 4-byte format are both converted to the 1-byte "High Instrument Saturation" value.
An exception to this is the "convert" program which allows translating each type of special pixel value into another type of special value or into a specified valid value.
The "specpix" program allows converting user-specified valid pixel values into special pixel values.
When the output pixel data type is 1-byte or 2-byte (OTYPE=1 or OTYPE=2), then the application program converts the internal 4-byte floating point values to the selected output type using the output file's type conversion parameters. In most cases, the user has control over the type conversion parameters that are used. However, rather than directly specifying these parameters, the ORANGE TAE parameter allows the user to specify the range of pixel values that can be represented in the output file. The application program will then compute the type conversion parameters (CORE_BASE and CORE_MULTIPLIER) that will allow representing the desired range of values. In most cases, the TAE default value of ORANGE equals (0.0, 0.0) or -- (the TAE null parameter value), which indicates that the desired range for the output file is the same as the range of values that can be represented in the input file. (If the input file is 4-byte floating point format and the output file is 1-byte or 2-byte format, then the user must specify an explicit range for the output file.)
For output pixel type 3 (32-bit floating point), the ORANGE TAE parameter and the corresponding CORE_BASE and CORE_MULTIPLIER label keywords are not used because each pixel already directly represents the "true" floating point value. In this case, the ORANGE value is ignored and the CORE_BASE and CORE_MULTIPLIER keywords are always set to 0.0 and 1.0 in the output file.
During the conversion of internal 4-byte floating point values to output 2-byte format, some values may be outside the range of values that can be represented in the output format using the current type conversion parameters. These will be converted to either the High or Low "Representation Saturation" special values. For output 1-byte format, values outside the range that can be represented are converted to "Null" and "High Instrument Saturation". These are thus valid output data values that are "lost" because they are outside the specified ORANGE. The application program will report a count of such "lost" output data values. If desired, the user could then re-run the application program with a larger ORANGE in order to reduce or eliminate these lost values. Note that most application programs will propagate a saturated input pixel value into a corresponding saturated output pixel value. These pixels are NOT included in the count of saturated pixels that are created during the output type conversion process.
Thus, if a "poor" ORANGE is selected, the output data may be saturated or needlessly compressed into very few output "bins." For example, when one file is divided by another (using the "ratio" program) the output range may be unpredictable. The user may have to guess what a reasonable ORANGE will be and look at a histogram of the output to decide a better range. The resulting data may be saturated or compressed into only one or two DN values. If the data are saturated, a lower ORANGE(1) and/or a higher ORANGE(2) must be used. If the data are compressed, the ORANGE values should be set closer to the compressed range. Using OTYPE=3 will eliminate the saturation and bin compression problems.
The CORE_BASE (ADD) and CORE_MULTIPLIER (MULT) are determined from the following approximate equations:
ORANGE(2) - ORANGE(1)
MULT = -----------------------------------
(VALID_MAX+0.5) - (VALID_MIN-0.5)
ADD = ORANGE(1) - MULT*(VALID_MIN-0.5)
The inverse equations are:
ORANGE(1) = ADD + MULT*(VALID_MIN-0.5)
ORANGE(2) = MULT*((VALID_MAX+0.5)-(VALID_MIN-0.5))+ORANGE(1)
Where:Note that the VALID_MIN and VALID_MAX values are adjusted by the 0.5 factors so that the specified ORANGE is mapped into the bottom of the lowest output bin and the top of the highest output bin. Also, note that the actual computations done by ISIS are somewhat more complex than indicated by the above simplified equations. This is due to several reasons: 1) need to carefully deal with precision and round-off effects, 2) need to maximize efficiency in the pixel conversion process, and 3) need to store the binary CORE_BASE and CORE_MULTIPLIER values in a character representation in the cube label.
For example, for a core base of 0.0 and a core multiplier of 1.0
For 8 bit
ORANGE(1) = 0.0+1.0*(1.-.5) = .5
ORANGE(2) = 1.0*(254.-1.+1)+.5 = 254.5
For 16 bit
ORANGE(1) = 0.0+1.0*(-32752.-.5) = -32752.5
ORANGE(2) = 1.0*(32767.-(-32752.)+1)-32752.5 = 32767.5
There also exists an ORANGE PDF file that is not used for invoking an actual
ISIS program. Instead, this provides on-line help information that is
more extensive than the help information in each individual program.
At the TAE prompt type:
TAE>t orange
?> help *
?> help orange
For most application programs, the user specifies the input subcube specifier with the SFROM TAE parameter, which has a string value. In many cases, the user can specify the subcube specifier string without enclosing it in double quotes. However, if it contains commas or parenthesis, the TAE syntax requires that it be enclosed in double quotes.
The default value of the SFROM parameter is usually the NULL TAE parameter value, which selects the entire input cube file (both core and suffix) and is thus the most commonly used value.
The application program may or may not actually process or use all the data selected by the input subcube specifier. The part of the selected data that is used will depend upon the definition and intended purpose of the application program. For example, one application program might be defined to process all selected core data and never process selected suffix data. Another program might make use of a suffix plane. In this case, the input subcube specifier would select the set of suffix planes that are passed to the application code, but there would be a separate TAE parameter that selects which suffix plane is to be actually used by the program.
Some programs may be capable of processing a selected subarea of the input core data. Most of these programs use the input subcube specifier to select the input data that are passed to the program and then use other TAE parameters to specify the part that will actually be processed. The exact method of specifying this process selection will vary according to the needs of the individual programs.
For most programs that process an input file and put the results into an output file, the size of the output file will correspond to the size of the part of the input file selected by the input subcube specifier. (In some cases, the output file might be larger than the input subcube specifier due to the addition of new suffix planes. Also, a program that performs a geometric transformation might produce an output file that has a spatial size that is different from the input file.) Selected input data that are not processed will normally be propagated unchanged to the output file. For example, if the definition of a program consists of processing only core data, then any selected input suffix data will be propagated unchanged to the output file.
The SFROM TAE parameter specifies the subcube using a single string for all three dimensions of the cube. The order of the three dimensions is normally "samples:lines:bands". If a dimension is left blank, all the data for that dimension is selected. The default value of NULL for SFROM selects the entire cube.
There also exists an SFROM PDF file that is not used for invoking an actual ISIS program. Instead, this provides on-line help information that is more extensive than the help information in each individual program. At the TAE prompt type:
TAE>t sfrom
?> help *
?> help sfrom
For a more detailed description of SFROM and more examples look in
Appendix D-Subcube Specifier.
Note that this includes additional capabilities that are not
illustrated by the following examples.
BASIC FORMAT:
sfrom = "ss-es(sinc):sl-el(linc):sb-eb(binc)"
where ss = starting sample
es = ending sample
sinc = sample increment
sl = starting line
el = ending line
linc = line increment
sb = starting band
eb = ending band
binc = band increment
EXAMPLE:
INPUT CUBE FILE = 150 samples x 150 lines x 255 bands
(Assuming this cube is band sequential, it also has 12 backplanes)
(Examples of suffix selection are the very last examples)
1) Select from input file a subcube that is 100x100x10, starting
with sample 15, line 20 and band 60
sfrom="15-114:20-119:60-69"
2) Select the same 100 samples and 100 lines as above,
but all 255 bands
sfrom="15-114:20-119:" (samples:lines:bands left blank)
3) Select all 150 samples and 150 lines of input cube,
but just bands 60-69
sfrom="::60-69" (samples and lines left blank)
4) Select same area as example 1, except every-other sample, line,
and band. (The result will actually be 50x50x5)
sfrom="15-114(2):20-119(2):60-69(2)"
5) Select every-other sample, every third line and every 10th band
sfrom="(2):(3):(10)"
SPECIAL CHARACTERS:
6) * = maximum possible value of a dimension (sample,line or band)
Select an entire cube except starting at sample 3, line 4 and
band 5
sfrom="3-*:4-*:5-*"
7) ~ = Exclude values or ranges enclosed by ()
Exclude lines 3 thru 10 and bands 240 thru 255, keep all samples
sfrom=":~(3-10):~(240-255)"
8) # = Given a starting point, a value following the # sign is the
number of values selected
Select 100 bands starting with band 10
sfrom="::10#100"
Select first 10 samples, and 10 lines starting at 25
sfrom="1#10:25#10:"
SUFFIX SELECTION:
9) S or s = Suffix selection
Select all lines and all samples, the first 10 bands,
(every other band), and the first 3 backplanes
sfrom="::1-10(2),S(1-3)"
Select all samples, all lines, all bands, but exclude all
backplanes
sfrom="::S(~(1-*))"
Select all samples, all lines, all bands, specific backplanes
3,5,7,8,10,12 (excluding 9 and 11)
sfrom="::s(3,5,7-12~(9,11))"
10) If a subcube specification is very complex or is to be used
a repeated number of times, then the specification can be
written to a text file, such as subspec.txt.
sfrom="<subspec.txt>"
An ISL file is a special type of cube file that is stored in BIP (band interleaved by pixel) storage order. The actual spectral data are stored in the core of the cube, and the spectral header data are packed into a set of backplanes on the cube. This storage organization allows rapid searching through the list of spectra in a file since each spectrum and its associated header data are stored as a contiguous piece of the physical disk file.
Appendix F contains an example of an ISL label. Appendix G contains a description of the spectral header that is used for entries in an ISL file.
The ISIS Spectral Libraries document describes available binary spectral library data that can be converted to ISL format with the "specpr2isl" and "spam2isl" programs. The document on Exchanging Data with Other Software Systems describes how to import ASCII spectral library data using the "asc2tbl", "hdrkeys", and "tbl2isl" programs. The "islist" program lists spectral header data from an ISL file.
The Region of Interest function in the "cv" display program writes average spectra for a set of arbitrary spatial regions to an ISL file. These data can then be used by the "unmix" program for classification based on spectral unmixing.
In a cube file, the name of the data object that contains the cube data is always "QUBE". However, in a Table file, the name of the data object that contains the table data is arbitrary and can vary from one Table file to another. One reason for this is to allow a single Table file to contain more than one table data object. (Duplicate object names are not permitted within one file.) In some cases, the name of the Table data object is determined by the application program. In other cases, the name of the Table data object can be specified by the user.
Note that a cube file can contain one or more Table objects in addition to the QUBE and HISTORY objects.
The number of columns of data contained in each row of the table is recorded in the COLUMNS label keyword. Each different column is identified by a unique name, which is recorded in the array of values in the COLUMN_NAME keyword. Note that column names are always recorded in the label using upper-case characters. User specified column names are not case-sensitive, i.e. any lower-case characters are converted to upper-case before being used.
Each column can contain either a scalar value or an array of values. (Most Table objects contain only scalar values.) The number of elements in each column is recorded in the COLUMN_ITEMS keyword. The data type of each element of each column is recorded in the COLUMN_ITEM_TYPE and COLUMN_ITEM_BYTES keywords. Most tables use only INTEGER*4 and REAL*4 data types, but a variety of other data types are supported, e.g., INTEGER*2 and CHARACTER*n (character strings of specified length n).
The table object description can optionally include the AUX_ROW_DEFINITION group of keywords. This group contains keyword names of the form name_NOTE, where "name" is the column name that is recorded in the COLUMN_NAME keyword. The values of these keywords are text strings that describe the meanings of the columns.
The "asc2tbl" program translates table data from an ASCII text format into a standard ISIS binary table file. This can be used for converting the output of the "tbl2asc" program back into the standard ISIS binary table format. This allows the use of a text editor for editing values in a table file. The "asc2tbl" program can also be used for translating other types of text table data, e.g., "foreign" text table data that contains no header information.
ISIS includes a keyword called FILE_STATE in every ISIS cube (qube), table, and Instrument Spectral Library (ISL) data file. This keyword will be set to either CLEAN or DIRTY. Each time the cube is opened this keyword will be checked. If the FILE_STATE is equal to CLEAN, then the program will continue on normally. However, if the FILE_STATE is DIRTY, then the application will halt with the appropriate error message.
When a FILE_STATE becomes DIRTY, it indicates that something has gone wrong in a previously run application. ISIS will always set the FILE_STATE to DIRTY when the file is being opened for writing. If the application crashes and does not close the cube properly the FILE_STATE will remain DIRTY.
However, this does not always mean the file is corrupt. To help restore a file from DIRTY to CLEAN, ISIS has an application called "cleanlab". "cleanlab" will modify the FILE_STATE keyword in the label to a CLEAN state. This program should be used with caution as the contents of the file may not be valid when an ISIS file is left in a DIRTY state.
To use this option, simply set the UNIX environment variable ISIS_IGNORE_INTEGRITY to TRUE. This can be done using (C Shell):
% setenv ISIS_IGNORE_INTEGRITY TRUENote that "TRUE" can be upper-case or lower-case characters. This must be done prior to entering TAE and cannot be modified within the TAE environment. For this case, all programs that open the ISIS file will:
Note that this feature is only supported on ISIS cube files, instrument spectral library files, and table files. It has no effect on other file operations. You should be warned that with this featured "turned on", a corrupted file will be processed as if it were valid.
Mosaic operations are implemented only by a few special mosaic programs.
A few programs are only capable of operating in the update mode. These programs include the FROM TAE parameter, which specifies the name of the input file that is to be updated. (These programs do not use the TO parameter since the output file is the same as the input file.)
Some of the programs are only capable of operating in the mode in which a new output file is created. These programs include the FROM and TO TAE parameters to specify the input file and new output file being created.
Many of the programs are capable of either updating the input file or creating a new output file. These programs include both the FROM and TO parameters. If the TO parameter is specified as the blank string (all blanks), then the input FROM file will be updated. If a non-blank TO is specified, then it will be the name of the new file that is created. Note that if TO specifies the same filename and extension as FROM, then the input file will NOT be updated. (See the 10.5 - File Output Environment section below.) Also, note that in most cases an input file cannot be updated if the input subcube specifier is used to select a sub-area of the input file.
% setenv ISIS_FILE_OVERWRITE TRUE
Note that "TRUE" can be upper-case or lower-case characters. If
ISIS_FILE_OVERWRITE does not exist, or has a value other
than "TRUE", then file overwrite is disabled.When an existing file is overwritten, this will result in the immediate (i.e., at time of file creation) loss of existing data within that file. This implies that one cannot simultaneously write a new "version" of a file and read from a previous "version" of the file. All new files are immediately initialized and truncated to the requested size, which by definition, results in the loss of any previously existing data in the file.
Note that this feature is not supported for file creation methods other than ISIS file I/O. In particular note that this feature does not apply to the "transformation data files" (TFILE) used by some programs. (Attempting to overwrite an existing TFILE is always a fatal error.) Furthermore, realize that utilization of this feature can result in irrecoverable data loss, thus caution is urged in the use of this feature.
By using IDL for ISIS display programs, users have access to easy point and click interfaces. ISIS display programs enable users to interactively display images, extract and plot spectra, query pixels, set match points and many other functions.
For more information on the display programs, see the Display Program Summary and the Notes on Use of the IDL Display Programs.
Contact us online at the Isis Support Center: http://isisdist.wr.usgs.gov