USGS

ISIS 2 Documentation


Isis Overview of ISIS Architecture

----

Table of Contents

1.0 - PURPOSE OF ISIS

ISIS (Integrated Software for Imagers and Spectrometers) is a software system for processing and analyzing the data sets produced by imaging spectrometers. Conceptually, these data sets (often referred to as image cubes) consist of stacks of tens or hundreds of images of the same scene, each image recording the reflected and emitted radiation from the scene in a single, narrow wavelength range. A profile through such a data cube from front to back at a given spatial pixel location contains the spectral signature of the corresponding spatial location. This spectral signature, when properly calibrated, can yield information about the chemical composition and physical nature of surface materials and the overlying atmosphere.

ISIS combines elements of both classical image processing and spectral analysis, as well as new techniques developed specifically for the analysis of image cubes. Standard image processing techniques can be applied to the individual spatial-spatial planes of image cubes. Spectral analysis techniques can be applied to the individual spectra of the cubes. In addition, a new set of analysis programs, designed to take advantage of the combination of image and spectral information, can be used to analyze image cubes. ISIS will also continue to incorporate the functionality of digital cartography found in the Planetary Image Cartography System (PICS).

2.0 - ISIS DOCUMENTATION

This document (the Overview of ISIS Architecture) describes the overall information that is needed for using the ISIS system. This includes an overall description of the system design features and general capabilities. Associated with this are five appendices that include examples of the cube file label, the history information that is maintained in ISIS cube and table files, the table file label, a complete description of how to select a sub-area of a cube file, and a summary of environment variables used by ISIS.

Various other pages of the web-based documentation contain Notes on ISIS Documentation (including instructions on producing a hard-copy program reference manual), Notes on Use of the IDL Display Programs (including required set-up), a Display Program Summary, summary lists of currently available application programs (which consist of the program name and a one-line description), a summary of available spectral library data, suggestions on techniques for exchanging ISIS data with other software systems, etc. The program list is available as an alphabetical list and also as a list organized by functional categories. Clicking on a program name will display a page that contains the complete documentation for the program. Additional documentation web pages describe overall image processing procedures. The Release Notes pages describe the changes in the latest and previous releases of ISIS.

When ISIS is properly installed, the ISISVERSION environment variable contains the date of the ISIS release in the format YYYY-MM-DD.

The main document that provides detailed information for ISIS application programmers is the ISIS System Design document (ISD), which describes the system design and the programming environment that is provided for developing application programs.

3.0 - OPERATING ENVIRONMENT

ISIS can be installed and run on several different operating systems. See the Installation Guide for a description of currently supported platforms. Many of the ISIS programs do not involve interactive display. However, if display capabilities are needed, most of the ISIS display programs require the use of the Interactive Data Language (IDL) from Research Systems, Inc. There is also one display program that does not require the use of IDL. ISIS was originally developed on a Vax VMS system and there still exists a version which can utilize that operating system.

4.0 - RUNNING ISIS PROGRAMS

ISIS is implemented as a collection of separate application programs that operate within the Transportable Applications Environment (TAE), developed by the NASA Goddard Space Flight Center. Most ISIS applications can also be directly run from the operating system prompt.

4.1 - TAE

The "Classic" version of TAE that is used by ISIS provides a text terminal interface that allows the user to specify a set of parameter values that are to be used on a given run of an application program. The following is a very brief summary of some of the main features of TAE. However, note that this is a small subset of the available features. Consult the TAE User's Manual for a complete description of the capabilities provided by TAE. We have been granted permission to distribute TAE Version 5.3 (Classic version only) with ISIS.

4.1.1 - STARTING A TAE SESSION

To start a TAE session the user should type the following when presented with the operating system prompt. (The "%" is the operating system prompt and should not be typed by the user.)

    % tae
      - or -
    % taetm
This will give you the "TAE>" prompt, indicating that TAE is ready to accept a command. TAE commands are not case sensitive, although the file names sent to ISIS programs are case sensitive and the names of ISIS programs must be entered as lower case. TAE commands can also be abbreviated using only enough characters needed to uniquely specify the command.

4.1.2 - USING TUTOR MODE TO INVOKE A PROGRAM

The TUTOR command (which can be abbreviated as simply "T" or "t") allows you to interactively specify parameter values for a given run of an application program. For example, to begin specification of parameter values for the "ratio" program, type:

    TAE>t ratio
TAE will then display a list of the parameters for the program, along with a brief description of each parameter and its initial default value (if any). The "?" prompt at the bottom of the screen indicates that TAE is ready for a tutor mode command. Since most programs have more parameters than can be displayed on one "page", typing the RETURN key will display the next page of parameters. You can set the value for a particular parameter by typing the name of the parameter (which may be abbreviated) and the desired value, which must be separated by an "=". For example, to specify a file called test.cub as the input file for the "ratio" program, type the following:

    ?FROM = test.cub
Alternatively, typing the down-arrow or up-arrow keys will select the "next" or "previous" parameter for interactive editing of the current parameter value.

When you have specified all the parameter values, typing "RUN" (or simply "r") will run the program with the current parameter values. If you decide you don't want to run the program, typing "EXIT" will return to the "TAE>" prompt.

4.1.3 - SPECIFYING STRING AND ARRAY PARAMETER VALUES

An array parameter value is specified by separating the values with commas. The values can optionally be enclosed in parenthesis. For example:
	TAE>dims = 200,300,40
		-or-
	TAE>dims = (200,300,40)

A string parameter value can usually be specified by typing the value without quotes. However, double quotes are required if the string value contains a comma (so that it will not look like an array of values). For example:

	TAE>from = myfile.cub
		-or-
	TAE>from = "myfile.cub"

Some string parameters have a defined list of valid values. In this situation, it is only necessary to specify enough characters to uniquely identify the value, and the characters are not case-sensitive. For example, the MODE parameter in the "dsk2dsk" program has valid values "GET_RANGE" and "SET_RANGE". This parameter can be set as follows:

	TAE>mode = GET_RANGE
		-or-
	TAE>mode = get
		-or-
	TAE>mode = g

4.1.4 - NULL PARAMETER VALUE

TAE offers the use of a null parameter value. When you use the null value for a parameter, you are leaving the decision up to the program. A good example of this feature is the parameter OTYPE. If OTYPE is set to "--" or the null parameter, then the program will use the same OTYPE for the output cube as the input cube. If you wish to assign the null value to a parameter you must type:

	?> parameter = --

4.1.5 - SAVING AND RESTORING PARAMETER VALUES

The tutor-mode command SAVE will save the current set of parameter values in a disk file. This would normally be done immediately before typing the RUN command, but it can be used at any time. If no file name is specified, then the file "prog.par" will be used, where "prog" is the name of the program. If desired, you can specify an explicit file name, e.g., typing

    ?SAVE test
will save the parameters in a file called "test.par". The RESTORE command (with an optional file name) will restore a set of previously-saved parameter values.

When you RUN a program in tutor mode, the current parameter values are automatically saved in a file called last.par. If an error is detected, you can then tutor the program and use the "REST last" command to restore the complete set of parameter values. You can then correct a parameter value without having to re-specify all the other parameter values.

4.1.6 - INVOKING A PROGRAM WITHOUT USING TUTOR MODE

Typing the name of a program at the TAE prompt will directly run the program without entering the tutor mode. For example, to run the "labels" program to list the label of the cube file "test.cub", type:

    TAE>labels from=test.cub print=all
This command will run the program with the specified values for the FROM and PRINT parameters. The default values will be used for any parameters that are not specified. If parameters are specified in the same order as the parameter definitions for the program, then the parameter names may be omitted. Thus, the following would be equivalent to the above command:

    TAE>labels test.cub all

4.1.7 - ISSUING SHELL COMMANDS FROM TAE

The Unix shell commands ls, cat, and pr are pre-defined in TAE and can be directly typed at the TAE prompt. Other shell commands can be issued by preceeding the command with "ush ", e.g.:
    TAE>ush grep word file.txt

4.1.8 - OBTAINING HELP INFORMATION

Typing "HELP" at the "TAE>" prompt will display general help information on using TAE. Typing "HELP prog" will display the overall description ot the program "prog".

When in tutor mode, typing "HELP" will display general help information on using tutor mode. Typing "HELP *" will display the overall description of the current program that is being tutored. (This is the same as the program description that is contained in this User's Manual.) Typing "HELP parm" will display help information on the specified parameter. Typing "EXIT" will exit from the display of the help information.

4.1.9 - ENDING A TAE SESSION

To end a TAE session and return to the operating system prompt, type "EXIT" at the "TAE>" prompt.

4.1.10 - TAE PROCEDURES

TAE allows the creation of procedure PDF files that will run a sequence of programs. This allows writing programs to control the processing using the TAE Command Language (TCL). Information on writing procedure PDFs is available in the VICAR documentation at http://www-mipl.jpl.nasa.gov/PAG/public/vug/vug7.html

4.2 - INVOKING PROGRAMS FROM THE UNIX SHELL PROMPT

In addition to invoking ISIS programs with TAE, most programs can be directly invoked from the Unix shell prompt. This allows invoking ISIS programs in shell scripts, Perl scripts and CGI web programming activities.

Note that there may well be conflicts with ISIS applications and system programs and applications with the same names. For example, we know that $ISISEXE/convert conflicts with /usr/X11R6/bin/convert on some Linux systems.

For Linux systems, the /proc file system *MUST* be in use in order for this feature to be utilized. See your Linux system adminstrator if you have problems or questions about this.

When running ISIS applications from the shell, you will need to be conscious of shell substitutions. The easiest way to avoid problems is to enclose arguments in single quotes so shell substitution is not performed. Here is an example of how to run an ISIS application from the shell:

    % zero to=zero.cub dims='(100,100,1)'

In this example, the ISIS "zero" application creates a floating point ISIS cube file with 100 samples, 100 lines and 1 band in Band SeQuential (BSQ) storage order and fills it with 0.0. (The "%" should not be typed as it is the Unix prompt.)

All ISIS applications exit with status indicating success or failure. The shell can test the exit status of any ISIS application and take appropriate action. An exit code of 0 indicates success. Error conditions are normally returned as negative values.

The "isisparam" program provides the capability to pass values back from an ISIS application to a Perl or shell script. Although this program is not invoked through TAE, there is a special TAE PDF file that contains a description of the program:

    TAE> t isisparam
    ?> help *

4.3 - LOCATING ISIS PROGRAMS

When TAE is started, the file $TAEPDF/slogon.pdf is executed. One of the things that this file does is to define a list of directories to be searched when the user runs a program. TAE searches these directories until a ".pdf" file for the program is located. (This file contains a description of the parameters and the documentation for the program.) TAE then runs the executable program from the same directory in which it found the ".pdf" file. (If the ".pdf" file is a procedure PDF file, then there is no corresponding executable and the commands in the file are executed.) The directory search list starts with the current working directory and then includes several other directories. Typing the "show-lib" command at the TAE prompt will list the current search directory list. The following is the list set up in the standard ISIS installation:

    TAE>show-lib
 
    User Library ($USERLIB):
    .

    Application Libraries ($APLIB): 
    $USERTAEBIN
    $DEVISISEXE
    $ISISEXE
    $PICEXE
    $ISISR/local/exe
    $TAEBIN
 
    System Library ($SYSLIB):
    $TAEPDF
This allows TAE to find the standard ISIS programs in the $ISISEXE directory. However, note that the $DEVISISEXE directory is searched before $ISISEXE. This allows $DEVISISEXE to be used for developmental versions of programs. Programs in this directory are not included in the normal ISIS distribution, but this directory could be used for holding local developmental programs at an ISIS installation site. If you plan to use this feature, note that you will need to create the $DEVISISEXE directory and set its access privileges according to the needs of your installation.

In addition, the $USERTAEBIN directory is searched before $DEVISISEXE. The standard ISIS installation procedures do not define the $USERTAEBIN environment variable, so the user can define this to be the directory that contains special programs being used for a particular project. Note that different directories could be used for holding platform-specific versions of the programs. Setting $USERTAEBIN to the appropriate directory could then select the set needed for running on a particular platform.

Note that the last thing that slogon.pdf does is execute the user logon PDF, ulogon.pdf. This PDF is also located in $TAEPDF and it is currently empty. If a user would define $USERTAEBIN to be a directory that contains a customized ulogon.pdf file, then this version would be executed at startup. This would allow greater flexibility for users and programmers.

When ISIS programs are invoked directly from the Unix shell prompt, a similar search of a list of directories is performed. In this case, the $PATH environment variable contains the list of directories to be searched. Note that $USERTAEBIN must be defined prior to execution of the isispgmdef initialization file in order for the $USERTAEBIN directory to be included in the search path. See the ISIS Installation Guide for details.

4.4 - BATCH PROCESSING

ISIS programs (except the interactive display programs) may be run as batch jobs by creating shell and Perl scripts.

Since UNIX TAE does not provide true batch capabilities, an ISIS program called "submit" has been written to help you start the UNIX batch once the TAE "batch" file is created. For instructions on how to create a TAE "batch" file for running a single ISIS program or for running a procedure PDF that invokes a sequence of ISIS progorams, tutor the "submit" program:

	TAE>t submit

4.5 - INVOKING IDL DISPLAY PROGRAMS

The IDL display programs are invoked by starting up IDL and typing the name of the display program along with any required parameters on the IDL prompt line. Unlike the TAE/shell programs, the names of the IDL display programs are not case-sensitive.

4.6 - ISIS SESSION LOG

ISIS automatically maintains a session log file that contains a record of each program that a user runs. This log file is a text file that can be printed or examined with a text editor. When a program is run, a new entry is added to the log file. (If the file does not exist, then it will be created.) The log file entry contains the name of the program and additional information such as the current date and time. The entry also records the values of the TAE parameters that were used for invoking the program. The format and content of the entry is similar to the history entry that is recorded in each cube file. (See below for discussion of cube file history.) In addition, the log file may contain additional information such as error messages or information produced by an application program. If a TAE parameter was set to the null parameter value at runtime, the parameter value will be listed as NULL in the session log file and history entry.

Most ISIS programs include the USERNOTE TAE parameter, which does nothing other than allow the user to specify an arbitrary comment that will be recorded in the session log file and the cube file history entry.

The default file name for the session log file is the file "print.prt" located in the current default directory. If desired, the user can select a different file by defining the UNIX environment name ISIS_LOG_FILE to be the desired file name. This can be done by typing a command such as the following in the C shell:

    % setenv ISIS_LOG_FILE session.log
or for the Borne shell:

    % ISIS_LOG_FILE=session.log
    % export ISIS_LOG_FILE
Note that this is an operating system command that is typed before starting the TAE session. The specified file name may optionally include a full path specification. Multiple concurrent sessions are not permitted to use the same session log file. Thus, one of the main reasons for specifying a different session log file is to allow multiple concurrent sessions, e.g., two interactive sessions or an interactive session along with a batch run.

The session logging can be turned off by setting the log file name to be the UNIX null device, i.e., "/dev/null" (C shell).

    % setenv ISIS_LOG_FILE /dev/null
Note that the "errors" program can be used to locate program runs in the session log file that encountered errors.

4.7 - TEXT REDUCTION

ISIS allows one to tailor the output text volume and content to the screen and the log file. To accommodate this, ISIS provides three environment variables. These are switches that will determine the level of text that is printed to the terminal screen and log file and the names of specific input (TAE) parameters that all programs will print if they use them, regardless of the level of output text selected by the user. Note that this feature does not affect what is recorded into the history of the file.

4.7.1 - SELECTION OF TEXT TO TERMINAL AND LOG FILE

The ISIS shell environment variables, ISIS_TERM_REPORT and ISIS_LOG_REPORT, can be used by the user to turn on or off complete printing of all text generated by ISIS system routines and programs. The default if these environment variables are not set is to print all text.

To turn off printing to the terminal screen AND ISIS log file, the (C or T shell) user can issue the following command at the shell prompt:

        % setenv ISIS_TERM_REPORT OFF
        % setenv ISIS_LOG_REPORT  OFF
This will suppress all but a record of the start and completion of each program both on the terminal and in the log file. An example of what this generates for output on both output text streams may be:

        OBJECT = PLANSINU
        END_OBJECT = PLANSINU

        
        OBJECT = GEOM
        END_OBJECT=GEOM
Note that by setting these environment variables to any other value, no value, or if they are not defined at all will result in the normal complete printing of all text to the target device. It is perfectly acceptable to turn the terminal output off and log file output on. This scenario is accomplished by the following setting of the variables: (C Shell)

        % setenv ISIS_TERM_REPORT OFF
        % setenv ISIS_LOG_REPORT  ON
This will generate full listings in the log file but suppress output to the screen. Note that "OFF" and "ON" values may be any combination of upper or lower case characters.

NOTE that the usual reporting of program percentage completed will not be affected by these settings. They will also have no effect on those cases where applications write directly to the output streams without using the standard ISIS text output routine.

4.7.2 - REPORTING SELECTED INPUT (TAE) PARAMETERS

Users who wish to see more information than the beginning and end of a program run may also wish to use the ISIS environment variable called ISIS_PARM_REPORT, which may contain a list of input (TAE) parameter names that will always be reported to both the terminal and the log file. A common use of this feature is to add the reporting of input and output file names for each program. Once the user sets ISIS_PARM_REPORT to the names of desired input parameters, all ISIS programs, regardless of the report modes selected by ISIS_TERM_REPORT and ISIS_LOG_REPORT, will list the values of these parameters to the terminal and log file.

To report the values of the FROM and TO parameters, the user can issue the following shell command (C Shell):

        % setenv ISIS_PARM_REPORT "FROM, TO"
Each parameter name may be separated by spaces, tabs, commas, semicolons or colons, and the entire list of parameter names must be enclosed by double quotes. A typical session log with the previous settings would look something like:

        OBJECT = PLANSINU
          FROM = "/work2/user/file1.cub"
        END_OBJECT = PLANSINU

        
        OBJECT = GEOM
          FROM = "/work2/user/file1.cub"
          TO = "$WORK2/outfile1.cub"
        END_OBJECT=GEOM

4.7.3 - HANDLING ERROR REPORTING

The following special considerations are applicable when errors occur: First, ISIS will report ALL error messages to both the screen and log file regardless of the reporting levels selected.

Second, when a program detects an error and calls the error facility, the first call will generate a dump of all the program input (TAE) parameters (housed in the HISTORY entry generated by each application program). This essentially is equivalent to exactly what the user would see if reporting to the terminal and log file where turned on. Subsequent calls to the error facility will simply record the current error message.

4.8 - SPECIFYING DATA FILE NAMES AND LOCATIONS

In many cases, a default file extension will be used if the user omits the file extension when specifying a data file name. The most common example of this is the use of the default extension ".cub" for cube data files. The default extension ".isl" will be used for Instrument Spectral Library files, and the default extension ".tbl" will be used for ISIS binary Table files. (See below for more information on Cube files, Instrument Spectral Library files, and Table files.)

In most cases, the normal operating system conventions are applicable for determining the location of a data file, i.e., an explicit directory can be specified, but the file is assumed to be located in the current default directory if the user does not specify a directory.

4.8.1 - USING ENVIRONMENT VARIABLES FOR DATA FILES

There are several ways in which the operating system setenv command can be used for defining UNIX environment variables that can be used in specifying data file names and/or locations. The simplest case is the use of an environment variable for the disk directory in which a file is to be found. For example, to define an environment variable before entering TAE, type for the C shell:

    % setenv NIMSDATA /work3/user/nimsdata
    % taetm
    TAE>labels FROM=$NIMSDATA/test.cub
for the Borne shell:

    % NIMSDATA=/work3/user/nimsdata
    % export NIMSDATA
    % taetm
    TAE>labels FROM=$NIMSDATA/test.cub

The environment variable cannot be defined after entering TAE. Defining an environment variable before entering TAE will maintain the definition across multiple TAE sessions run from the same C or Borne shell session.

Another way to use environment variables is to define an environment variable in which the definition is an actual cube file name. The environment variable can then be entered for the TAE parameter value that specifies a cube file name in an ISIS program. For example in C shell:

    % setenv MYCUBE /work2/user/temp/cubs/biglongcubefilename.cub
    % taetm
    TAE>labels FROM=$MYCUBE
The setenv command can use another environment variable in the definition. For example, if MOONDATA were defined to be /work3/user/moon, then the following setenv command could be used (C shell):

    % setenv MYCUBE $MOONDATA/biglongcubname.cub
The ISIS I/O routines will use a two-step process when trying to determine the actual disk filename to use. First, an intermediate string will be formed by assuming that the supplied string is an environment variable and by translating it into its defined value. If the user-supplied string is not a defined environment variable, then the intermediate string will be just the original user-supplied string. In the second step, the I/O routines will check the intermediate string to see if it has an extension, i.e., a ".<something>" on the end. If there is no extension, then the default extension (".cub", ".isl", or ".tbl") will be added. This will allow the user to omit the file extension on the setenv command, e.g., the following will have equivalent results (C shell):

    % setenv MYCUBE /work3/user/filename.cub
    % setenv MYCUBE /work3/user/filename
Note that the directory can be omitted from the definition of the environment variable, e.g.:

    % setenv MYCUBE filename.cub
When this environment variable is used, the current default directory will be used as the location of the file.

Also note that a directory can be combined with the reference to an environment variable. e.g. (C shell):

    % setenv MYCUBE filename.cub
    % taetm
    TAE>labels from=/work3/user/$MYCUBE
When ISIS records the TAE parameters in the HISTORY entry and the Session Log file, the exact string specified by the user will be recorded. In addition, most programs will also record the translated string that gives the actual disk file name that is to be used. This will be recorded as a comment line delimited by "/*" and "*/". (The translated string that is recorded will not include an assumed default file extension.)

4.9 - MEMORY BUFFER ENVIRONMENT VARIABLE

ISIS offers an environment variable, called ISISBUFMEM, which allows users to set the amount of physical memory an ISIS application can allocate for its work buffers.

Some ISIS applications will read the ISISBUFMEM environment variable in order to determine the memory utilization. This variable should initially be determined by the system manager. (The amount of I/O intermediate work buffers assigned by the system manager is a function of the I/O memory available on a computer and the number of users using the system). However, if this is not enough memory for an application, the user can alter the allocation amount by setting the ISISBUFMEM environment variable as follows (C Shell):

       % setenv  ISISBUFMEM xxxxxxxx 
Where xxxxxxxx = number of bytes to be used for the I/O intermediate work buffers for an application. If the ISISBUFMEM environment variable has not been set up, then a default of 4 mega-bytes will be used.

4.10 - PROGRAM PROGRESS ENVIRONMENT VARIABLES

As a default, when a standard ISIS program is run, ISIS lists to the screen the percentage done using 10 percent increments (e.g., 10 % done, 20 % done, etc). ISIS allows one to define output increments other than 10 or to output the amount of time left to complete the program. To accomplish this, ISIS provides the two environment variables ISISPROGRESS and ISISETA.

4.10.1 - PERCENTAGE PROGRESS

To set a percentage different from the 10 percent default, use the following statement (C shell):

       % setenv ISISPROGRESS inc 
where 'inc' is the increment you want (only values 1 thru 20 are valid).

4.10.2 - TIME REMAINING PROGRESS

To have a program output the time remaining, use the command (C shell):

     % setenv ISISETA time
where 'time' is the amount of time in seconds to wait before printing the time left (only values 15 thru 60 are valid). If the program run in less than the number of seconds set then no time status will be output.

5.0 - CUBE FILES

The following describes the overall format and characteristics of the standard three-dimensional cube files that are processed by the standard ISIS programs. This includes a description of the pixel data types that are supported in cube files. The format and access to data in cube files are described in more detail in the ISIS System Design document (ISD).

5.1 - BASIC CUBE FILE STRUCTURE

ISIS cube files are stored in a format that is similar to "standard" PDS labeled files. There are several reasons why it is not feasible for ISIS file labels to be exactly compatible with the PDS label standard. When the ISIS label structure was defined and implemented, we attempted to be as compatible as possible with PDS standards. However, some differences were necessary because PDS is an archive/data exchange format whereas the ISIS labels must allow dynamic on-line updating of information. Also, when the ISIS labels were designed, some necessary features had not yet been defined in the PDS "standard." (For example, three-dimensional data files were not defined.) These were subsequently defined by PDS in ways that were different from what had been implemented in ISIS. Another problem is that the PDS "standard" has changed since the ISIS labels were designed. Many ISIS files already exist, and it would not be feasible to convert them to conform to the current PDS standard.

A PDS or ISIS labeled file consists of two basic parts. The first part is the label, which is located at the beginning of the file. The remainder of the file is the data area, which contains one or more data objects. The label describes the structure and content of the file. Information in the label is stored in a "keyword=value" text format. Appendix A contains an example label for an ISIS cube file.

The QUBE files that are supported by the ISIS I/O routines are used for storing multi-dimensional data arrays. The maximum number of supported dimensions is six. A CUBE file is a special case of a qube file in which the number of dimensions in the data array is three. (See the ISD for more information on the generalized qube files.) Cube files are normally used for storing data produced by imaging spectrometers. Thus, two of the dimensions correspond to spatial coordinates and the other dimension corresponds to the spectral (wavelength) coordinate. The contents of the data values in a standard cube file can of course consist of things other than observations measured by an imaging spectrometer.

Each cube file contains at least two data objects, a HISTORY object and a QUBE object. A cube file may optionally contain "extra" data objects in addition to the HISTORY object and QUBE object. The standard ISIS cube-processing programs will propagate these "extra" data objects unchanged from an input file to an output file.

5.2 - CUBE FILE HISTORY OBJECT

The HISTORY object contains text information that records the sequence of processing operations that created the file. This consists of a sequence of history entries, each of which describes an application program that was run and the TAE parameters that were used. The format of history entries is the same as entries in the ISIS session log file. Appendix B contains an example cube file history. The listing in Appendix B was produced by running the "labels" program, which extracts the label and/or history from a cube file and displays it on the terminal and puts it into the session log file.

All ISIS programs automatically maintain the history information. When an application program reads data from an input file and writes processed data to an output file, it will copy the input file history information to the output file and then append a new history entry for the current program.

5.3 - LABEL AND HISTORY SPACE ALLOCATION

When a cube file is created, the label area and history object are initialized with a defined amount of free space. This allows space for label and/or history additions as programs operate on the file. When an input file is processed and results written to a new output file, most programs insure 15 unused 512-byte records in the label and 25 unused 512-byte records in the history. An example of a program that allocates empty space to the label and history of the newly created cube is "dsk2dsk".

If there are several program runs that operate on a cube file without creating a new file, the label and/or history area can become full, causing a program failure. There are several ways to avoid this problem:

  1. To clear the history object, you can run "labels" with the parameter INITHIST equal to YES.
  2. Run "dsk2dsk" to make a copy of the cube before the label/history is filled. This will allocate new free space in the new copy of the cube.
  3. Some programs (e.g. "mosaic") allow you to disable all HISTORY recording.

5.4 - CUBE OBJECT DESCRIPTION KEYWORDS

The label of a cube file contains various keywords that describe the cube data object. These include keywords that specify the number of axes (AXES, which is always 3 for cube files), the name of each axis (AXIS_NAME), the physical storage order (i.e., order of the axis names), the length of each axis (CORE_ITEMS), the number of suffix planes (if any) on each axis (SUFFIX_ITEMS), various keywords that describe the pixel data type for the file (CORE_ITEM_TYPE, CORE_ITEM_BYTES, etc.), and the name and units for the values stored in the main cube data array (CORE_NAME and CORE_UNIT).

Some of the label keywords are contained within GROUPs. The most important of these is the BAND_BIN group, which describes the spectral (wavelength band) axis of the cube. These include a list of wavelengths (BAND_BIN_CENTER) and a list of the original instrument band numbers that correspond to the file band numbers (BAND_BIN_ORIGINAL_BAND). This keyword group might optionally include a list of instrument grating positions and detector numbers (BAND_BIN_GRATING_POSITION and BAND_BIN_DETECTOR).

5.5 - CUBE DATA OBJECT FORMAT

The following diagram illustrates the general structure of a cube data object. Note that this is a conceptual or "logical" view of the cube. Most application programs provide this logical view of a cube to the user. For example, the user usually can refer to logical band numbers rather than referring to indices along the first, second, or third axis of the cube. (This logical view is separate from the issue of the physical storage of the cube data object in a file, which is discussed later in this document.)

            ______________________             .
           /                 /   /|            .          ^            
          /   backplane     /   / |  } suffix  .         /
         /_ _ _ _ _ _ _ _ _/_ _/  |  } region  .        /            
        /                 /   /|  |            .       / d
       /                 /   / |c |            .      / n
      /                 /   /  |o |            .     / a
     /       core      /   /   |r |            .    / B
    /                 /   /    |n |            .   /
   /_________________/___/   e |e |            .  /----------------> 
   |                 |   |  d e|r |            .  |     Sample      
   |                 |   | i n | /|  l         .  |
s  |                 |   |s a  |/ / a          .  | L
p  |                 |   | l   / / r           .  | i
a  |       core      |   |p   /|/ t            .  | n
t  |                 |   |   / / c             .  | e
i  |                 |   |  / / e              .  |
a  |                 |   | / / p               .  |
l  |_________________|___|/ / s                .  v
   |   bottomplane   |   | /       } suffix    .
   |_________________|___|/        } region    .
         spatial      ^^^                      .
                   suffix region               .

5.5.1 - CORE REGION

The core region of the cube data object contains the main data array that is being stored in the cube file. For a standard ISIS cube file, this is normally the imaging spectrometer data cube, which consists of a number of spatial-spatial image planes corresponding to different wavelength bands.

The AXES keyword specifies the number of dimensions (axes) in the core. For standard cube files, this value is always three. The CORE_ITEMS keyword has an array of integer values that specify the length of each axis of the core. These values are expressed as the number of pixels. The AXIS_NAME keyword has an array of literal values that specify the names of the axes. (A literal keyword value is similar to a string value except that it is either enclosed by single quotes or not enclosed by quotes at all. String keyword values are always enclosed by double quotes.) The order of the values in CORE_ITEMS and AXIS_NAME corresponds to the physical storage order of the data. (See below for details on physical storage orders.) The core data values are further described by the required application-owned keywords CORE_NAME and CORE_UNIT. (These are called "application-owned" because they are maintained by the applications code of the programs rather than by the ISIS system routines. The applications programs can set these keywords to any desired values.) The label also includes a number of keywords that describe the format of individual core pixel values. These are described in detail later in this document.

5.5.2 - SUFFIX REGIONS

Each of the three axes in a cube data object may optionally include suffix data that extend the length of the axis. Conceptually, this can be viewed as forming one or more suffix planes that are attached to the core cube as shown in the diagram above. Suffix planes that extend the (logical) band dimension are called BACKPLANES. Suffix planes that extend the (logical) sample dimension are called SIDEPLANES. Suffix planes that extend the (logical) line dimension are called BOTTOMPLANES. Note that these terms refer to the logical axes and are not necessarily related to the physical storage of the cube data object. The suffix planes are used for storing auxiliary data that are associated with the core data. For example, a backplane might be used for storing the latitude values for each spatial-spatial pixel. Another backplane might be used for storing the wavelength of the deepest absorption feature that was found in the spectrum at each spatial-spatial pixel. One or more sideplanes might be used for storing engineering data that are associated with each spatial line.

The SUFFIX_ITEMS keyword has an array of integer values that specify the number of suffix planes that exist along each of the three axes. The order of the values in SUFFIX_ITEMS corresponds to the physical storage order of the data file, i.e., the values in the array correspond to the values in the AXIS_NAME keyword. (See below for details on physical storage orders.)

In a manner similar to the core description keywords, the name of each suffix plane and the units of the data values are described in suffix description keywords of the form axis_SUFFIX_NAME and axis_SUFFIX_UNIT, where "axis" is the name of the axis that is extended by the suffix data, e.g., BAND_SUFFIX_NAME and BAND_SUFFIX_UNIT for suffixes that are backplanes. The "axis" value must correspond to one of the axis names listed in the AXIS_NAME keyword. The suffix description keywords all have array values, where the number of values corresponds to the number of suffix planes on the axis. This allows each suffix plane to have a description that is independent of all the other suffix planes. Note that the name of each suffix plane is required to be unique within each suffix region in a cube file. Additional suffix description keywords are described later in this document.

5.5.3 - CORNER REGIONS

If a cube file includes suffixes on more than one axis, then the region that is the intersection between two (or all three) of the suffix regions is called a CORNER region. Space for corner region data are allocated in the cube files, but these data are currently not used by any application programs.

5.5.4 - PIXEL STORAGE SIZES

In a standard three-dimensional cube file, core pixels can be stored in one of three different formats, which occupy one, two, or four bytes. All the core pixels within a single file must be of the same storage size. All suffix pixels always occupy four bytes of storage in the file. However, several different formats are supported for the contents of a four-byte suffix pixel. Handling of different pixel data types is described in detail below.

5.5.5 - PIXEL COORDINATES

The pixel indexing that is used by the I/O routines allows use of the "logical" coordinate system shown in the diagram above. SAMPLE=1 is the left edge of the spatial-spatial core image. LINE=1 is the top edge of the spatial-spatial core image. BAND=1 corresponds to the spatial-spatial image at the "front" of the diagram. Coordinates within the suffix regions are independent of the core coordinates and are not an extension of the core coordinates. Thus, the first suffix on an axis has a coordinate of 1. However, suffixes are usually identified by name rather than by index number.

5.5.6 - STORAGE ORDERS

The disk file in which a cube data object is stored is physically accessed as if it were a one-dimensional data structure. Storing the cube pictured above thus requires that the "logical" three-dimensional structure be mapped into the one-dimensional physical file structure. This involves moving through the three-dimensional structure in certain patterns to determine the linear sequence of core and suffix pixel values that occur in the file. In ISIS cube files this pattern is defined by specifying which axis index varies fastest in the linear sequence of pixel values in the file, which axis varies second fastest, and which axis varies slowest.

In a standard three-dimensional cube file, the names of the three axes are always SAMPLE, LINE, and BAND. The AXIS_NAME keyword has an array of values that list the names of the axes in the cube. The order of the names specifies the cube storage order in the file. The first axis is the fastest varying, and the third axis is the slowest varying. The standard ISIS cube-processing programs support the following three storage orders:

(SAMPLE, LINE, BAND) - Band Sequential (BSQ)
(SAMPLE, BAND, LINE) - Band Interleaved by Line (BIL)
(BAND, SAMPLE, LINE) - Band Interleaved by Pixel (BIP)

The CORE_ITEMS and SUFFIX_ITEMS keywords have array values. The order of these values corresponds to the order of the axes listed in the AXIS_NAME value.

In the physical file storage, suffix pixel data (if present) is "scrambled" in with the associated core pixel data. For example, in a BSQ storage order file, the physical cube storage in the file begins with the pixels in the first (top) line of the spatial-spatial image plane at the first wavelength band. This is followed by the sideplane pixel values that extend this line of core pixels. Next are the core pixels for the second line, followed by the sideplane pixels for the second line. After the last line of this first core image plane (and its associated sideplane pixels) come the bottomplane pixels associated with the first band. This is then repeated for the second through last bands. Finally, all the backplane data are stored after all the core data and associated sideplane and bottomplane pixels. Thus, in a BSQ file, all of the backplane pixels are stored together in a single contiguous block at the end of the cube data object. This means that backplanes can be added to a BSQ file without creating a new file because it only requires adding new data at the end of the file. (This is true only if the qube object is the last object in the file, which is usually the case since any "extra" data objects usually are located between the history object and the qube object.) However, adding backplanes to a BIL or BIP file requires creating a new file.

In general, almost all application programs are capable of processing all of the storage orders. The storage orders that can be processed and the efficiency of processing usually depend upon whether the program does its processing in terms of spatial planes or in terms of spectra. For example, the "ratio" program divides each spatial plane by a given spatial plane. This program is most efficient when processing a BSQ file, but BIL and BIP files can also be processed. A warning will be given if an application will be substantially slower because of storage order, but will continue with processing.

Note that the "transpose" program will convert a standard cube file from one physical storage order to any other physical storage order.

5.6 - PIXEL DATA TYPES

The following sections describe the standard supported data types for core pixels and suffix pixels.

5.6.1 - TYPE CODES AND LABEL KEYWORDS

The data type for core pixels is described by two label keywords. CORE_ITEM_BYTES has an integer value that specifies the number of storage bytes occupied by a pixel value. CORE_ITEM_TYPE has a string value that specifies the internal binary representation of the stored value. The CORE_ITEM_TYPE values stored in the label of a cube file include a system-specific prefix, e.g. "SUN_INTEGER" or "SUN_REAL".

It is important to note that the standard ISIS cube-processing programs assume that values stored in a file are stored in formats that are native to the host computer on which the software is currently running. Thus, for example, there is no capability for "on the fly" conversion of "foreign" floating point formats to the native floating point format. This conversion would need to be performed by the "convert" program, which converts a file from a "foreign" format to the standard ISIS cube file format for the current host computer.

A single integer type code value is used to identify the various supported data types, i.e., combinations of CORE_ITEM_TYPE and CORE_ITEM_BYTES keyword values. This is described in more detail below.

5.6.2 - CORE PIXEL DATA TYPES

Conceptually, a core pixel is always considered to have a "real" number value ("real" as opposed to "integer"). It is very easy to misunderstand the following discussion if this critical concept is not recognized. Thus, we will emphasize it:

   ---------------------------------------------------------------
   |   CONCEPTUALLY, A CORE PIXEL IS ALWAYS CONSIDERED TO HAVE   |
   |   A "REAL" NUMBER VALUE.                                    |
   ---------------------------------------------------------------
In standard ISIS cube files, these core pixel values can be represented in three different ways, corresponding to different numbers of bytes used to store the values in the file. When 4-byte format is used, the stored number is in the format of a native (for the current host computer) floating-point number that directly represents the "real" value of the pixel. The other two formats store a pixel value using 1-byte or 2-byte storage items in the file. For these, the stored value is considered to be an integer value and type conversion parameters (given by the CORE_BASE and CORE_MULTIPLIER label keywords) are used to convert the stored integer values into the "real" pixel values that are being represented. The "real" value being represented is determined as follows:

        "real_value" = CORE_BASE +
                     (CORE_MULTIPLIER * REAL(stored_value))
From the point of view of the user, core pixels always appear to have these "real" values, i.e., the user can never directly see the 1-byte or 2-byte file values before the application of the type conversion parameters. In most cases, an application program will automatically convert 1-byte or 2-byte input pixel values to floating point and then perform its computations using these floating point values. It will then automatically convert the computed values to 1-byte or 2-byte format for writing to an output file. Thus, most application programs are capable of processing files that use any of the three different formats.

However, there are several aspects of the different storage formats that are important to the user. First, different data types result in different disk file sizes. Second, different data types allow different precision (number of significant digits) in the stored pixel values. 4-byte format uses the most disk space and allows the greatest precision. 1-byte format uses the least disk space and allows the least precision. The third important aspect of the different data formats is the range of values that can be represented. With 4-byte format, the representable range is the range of floating point values and is thus nearly unlimited for all practical purposes. For 2-byte or 1-byte format, the representable range varies depending upon the current values of the type conversion parameters. This is discussed in more detail below.

The following is a summary of the three standard pixel representation formats for core pixels:

       Type     CORE_ITEM_BYTES   CORE_ITEM_TYPE    Type Conversion
       Code                                           Parameters
        1           1           UNSIGNED_INTEGER        Yes
        2           2           INTEGER                 Yes
        3           4           REAL                    No
For 4-byte format the stored values are floating point values that directly represent the pixel values. In this case, the CORE_BASE and CORE_MULTIPLIER values are not used. However, they are still required to be present in the label and should have values of 0.0 and 1.0.

5.6.3 - SUFFIX PIXEL DATA TYPES

In a manner similar to core pixel values, several different data formats can be used for storing suffix pixels. However, unlike core pixel formats (which can usually be chosen by the user), suffix pixel formats are usually automatically chosen by the application programs according to their needs. Also, unlike core pixel values, suffix pixel values are always stored in 4-byte items in a standard ISIS cube file. This is indicated by the SUFFIX_BYTES label keyword, which always has a value of four and is always required to be present (even if there are no suffixes stored in the file).

In a manner similar to the core description keywords, the data type for suffix pixels is described in suffix description keywords of the form axis_SUFFIX_ITEM_BYTES and axis_SUFFIX_ITEM_TYPE, where "axis" is the name of the axis that is extended by the suffix data, e.g., BAND_SUFFIX_ITEM_BYTES and BAND_SUFFIX_ITEM_TYPE for the suffixes that are backplanes. The "axis" name corresponds to one of the axis names listed in the AXIS_NAME keyword. Each suffix plane within a single file can have a different data format. Thus, the values of these keywords are arrays. Each element of the array refers to a separate suffix plane. Type conversion parameters can also be used for some of the standard suffix data types. These are contained in keywords of the form axis_SUFFIX_BASE and axis_SUFFIX_MULTIPLIER, e.g., BAND_SUFFIX_BASE and BAND_SUFFIX_MULTIPLIER for the backplane suffixes. Each suffix can have different type conversion parameters, so these keywords have array values.

When a suffix pixel contains a numerical value, it is expected that in most cases it will be stored in one of three standard formats:

       Type      axis_SUFFIX_      axis_SUFFIX_      Type Conversion
       Code       ITEM_BYTES       ITEM_TYPE          Parameters
        4               1          UNSIGNED_INTEGER      Yes
        5               2          INTEGER               Yes
        3               4          REAL                  No

5.6.4 - BIT-MASKS IN SUFFIX PIXELS

Suffix pixels can be used to store data other than numerical values, e.g., bit-masks or engineering data that are packed into non-standard formats. This is indicated by an axis_SUFFIX_ITEM_TYPE value of "BIT_STRING" or "BYTE_STRING". The ISIS system design does not put any constraints on the interpretation of such suffix pixel values. The application programs can use these data values in any desired manner.

5.6.5 - SPECIAL PIXEL VALUES

For each standard core pixel type, a set of stored values is defined for representing the "special" pixel values. These include the "Null" pixel value and four different "Saturated" values. Two of the saturation values are generated when valid pixel values are being converted to 1-byte format or 2-byte format and the value being converted is outside (high or low) the range of values that can be represented in the output representation with the given type conversion parameters. The other two saturation values are used for pixel values that were saturated (high or low) by the instrument that generated the data. These will normally be identified by the logging program or the translator program that converts a "foreign" file into standard ISIS cube file format.

The ISIS application programs will normally preserve the identity of special pixel values when converting pixels from one representation to another. For example, when converting from 2-byte format to 4-byte format, a 2-byte "Null" pixel will be converted to a 4-byte "Null" pixel.

For the 1-byte (8-bit) pixel data type, there are only two special pixel values: "Null" and "High Instrument Saturation." Thus, "Null", "Low Instrument Saturation" and "Low Representation Saturation" in 2-byte or 4-byte format are all converted the the 1-byte "Null" value. "High Instrument Saturation" and "High Representation Ssaturation' in 2-byte or 4-byte format are both converted to the 1-byte "High Instrument Saturation" value.

An exception to this is the "convert" program which allows translating each type of special pixel value into another type of special value or into a specified valid value.

The "specpix" program allows converting user-specified valid pixel values into special pixel values.

5.6.6 - SPECIFYING OUTPUT CORE PIXEL DATA TYPES

In most cases, application programs perform their computations using 4-byte floating point format. If the core pixel values in an input file are not stored in 4-byte format, then they are first internally converted to 4-byte format using the type conversion parameters (CORE_BASE and CORE_MULTIPLIER label keyword values). Most application programs include the OTYPE TAE parameter, which allows the user to specify the pixel data type (i.e., type code) for the output file. The permitted OTYPE values include 1, 2, and 3, which select 1-byte, 2-byte, and 4-byte format. In addition, OTYPE = -- (the TAE null parameter value) may be specified. This indicates that the data type for the output file will be the same as the data type of the input file.

When the output pixel data type is 1-byte or 2-byte (OTYPE=1 or OTYPE=2), then the application program converts the internal 4-byte floating point values to the selected output type using the output file's type conversion parameters. In most cases, the user has control over the type conversion parameters that are used. However, rather than directly specifying these parameters, the ORANGE TAE parameter allows the user to specify the range of pixel values that can be represented in the output file. The application program will then compute the type conversion parameters (CORE_BASE and CORE_MULTIPLIER) that will allow representing the desired range of values. In most cases, the TAE default value of ORANGE equals (0.0, 0.0) or -- (the TAE null parameter value), which indicates that the desired range for the output file is the same as the range of values that can be represented in the input file. (If the input file is 4-byte floating point format and the output file is 1-byte or 2-byte format, then the user must specify an explicit range for the output file.)

For output pixel type 3 (32-bit floating point), the ORANGE TAE parameter and the corresponding CORE_BASE and CORE_MULTIPLIER label keywords are not used because each pixel already directly represents the "true" floating point value. In this case, the ORANGE value is ignored and the CORE_BASE and CORE_MULTIPLIER keywords are always set to 0.0 and 1.0 in the output file.

During the conversion of internal 4-byte floating point values to output 2-byte format, some values may be outside the range of values that can be represented in the output format using the current type conversion parameters. These will be converted to either the High or Low "Representation Saturation" special values. For output 1-byte format, values outside the range that can be represented are converted to "Null" and "High Instrument Saturation". These are thus valid output data values that are "lost" because they are outside the specified ORANGE. The application program will report a count of such "lost" output data values. If desired, the user could then re-run the application program with a larger ORANGE in order to reduce or eliminate these lost values. Note that most application programs will propagate a saturated input pixel value into a corresponding saturated output pixel value. These pixels are NOT included in the count of saturated pixels that are created during the output type conversion process.

Thus, if a "poor" ORANGE is selected, the output data may be saturated or needlessly compressed into very few output "bins." For example, when one file is divided by another (using the "ratio" program) the output range may be unpredictable. The user may have to guess what a reasonable ORANGE will be and look at a histogram of the output to decide a better range. The resulting data may be saturated or compressed into only one or two DN values. If the data are saturated, a lower ORANGE(1) and/or a higher ORANGE(2) must be used. If the data are compressed, the ORANGE values should be set closer to the compressed range. Using OTYPE=3 will eliminate the saturation and bin compression problems.

The CORE_BASE (ADD) and CORE_MULTIPLIER (MULT) are determined from the following approximate equations:

                      ORANGE(2) - ORANGE(1)
        MULT = -----------------------------------
                (VALID_MAX+0.5) - (VALID_MIN-0.5)

        ADD = ORANGE(1) - MULT*(VALID_MIN-0.5)
The inverse equations are:

        ORANGE(1) = ADD + MULT*(VALID_MIN-0.5)

        ORANGE(2) = MULT*((VALID_MAX+0.5)-(VALID_MIN-0.5))+ORANGE(1)
Where:
VALID_MIN and VALID_MAX are set for the appropriate pixel type.
For 8 bit: VALID_MIN=1., VALID_MAX=254.
For 16 bit: VALID_MIN=-32752., VALID_MAX=32767.
ORANGE(1) is the minimum expected value in the image data range
ORANGE(2) is the maximum expected value in the image data range

Note that the VALID_MIN and VALID_MAX values are adjusted by the 0.5 factors so that the specified ORANGE is mapped into the bottom of the lowest output bin and the top of the highest output bin. Also, note that the actual computations done by ISIS are somewhat more complex than indicated by the above simplified equations. This is due to several reasons: 1) need to carefully deal with precision and round-off effects, 2) need to maximize efficiency in the pixel conversion process, and 3) need to store the binary CORE_BASE and CORE_MULTIPLIER values in a character representation in the cube label.

For example, for a core base of 0.0 and a core multiplier of 1.0

                    For 8 bit                     
        ORANGE(1) = 0.0+1.0*(1.-.5)    =    .5
        ORANGE(2) = 1.0*(254.-1.+1)+.5 = 254.5

                    For 16 bit
        ORANGE(1) = 0.0+1.0*(-32752.-.5)             = -32752.5
        ORANGE(2) = 1.0*(32767.-(-32752.)+1)-32752.5 =  32767.5
There also exists an ORANGE PDF file that is not used for invoking an actual ISIS program. Instead, this provides on-line help information that is more extensive than the help information in each individual program. At the TAE prompt type:

    TAE>t orange
    ?> help *
    ?> help orange

5.6.7 - DETERMINING RANGE OF VALUES IN A FILE

The pixel data type and the associated type conversion parameters determine the range of pixel values that can be represented. However, the range of values actually contained in a file might be much smaller. The "dsk2dsk program allows the user to determine the range of pixel values that are actually contained in the core of a cube file. This can be done without creating a new output file. "dsk2dsk" also allows creating a new output file that stores the same pixel values using a different data type and/or different set of type conversion parameters. And, the input subcube specifier (see below) allows "dsk2dsk" to create an output file that is a subarea of the input file.

5.7 - DETACHED CUBE FILE LABELS

For most ISIS cube files, the label is imbedded in the same file as the actual core and suffix data. However, the ISIS I/O routines also support detached labels. A detached label is a text file that contains the label that describes cube file data that is stored in a separate file. The detached label includes a pointer to the data file and information that specifies where the core/suffix data are located in the data file. This allows accessing data files that contain header information other than a standard ISIS label at the beginning of the data file. For example, some of the Clementine lunar image data is distributed on CD-ROMs in files that contain PDS labels at the beginning of the data files. An associated detached ISIS label allows accessing these data files with the ISIS software. When a detached label is to be used for accessing a file, the name of the detached label file is supplied to the ISIS programs. The ISIS I/O routines then automatically find the associated data file.

6.0 - INPUT SUBCUBE SPECIFIER

6.1 - PURPOSE OF THE INPUT SUBCUBE SPECIFIER

The purpose of the input subcube specifier is to allow the user to specify what parts of an input cube file are to be passed to the actual applications code of an application program. This consists of specifying a list of index values for each region (core and suffix) for each dimension in the cube. The input subcube specifier allows the user to specify these index value lists by specifying individual index values and/or ranges of values. Ranges can optionally be modified by specifying either an increment value or values within the range that are to be excluded.

For most application programs, the user specifies the input subcube specifier with the SFROM TAE parameter, which has a string value. In many cases, the user can specify the subcube specifier string without enclosing it in double quotes. However, if it contains commas or parenthesis, the TAE syntax requires that it be enclosed in double quotes.

The default value of the SFROM parameter is usually the NULL TAE parameter value, which selects the entire input cube file (both core and suffix) and is thus the most commonly used value.

The application program may or may not actually process or use all the data selected by the input subcube specifier. The part of the selected data that is used will depend upon the definition and intended purpose of the application program. For example, one application program might be defined to process all selected core data and never process selected suffix data. Another program might make use of a suffix plane. In this case, the input subcube specifier would select the set of suffix planes that are passed to the application code, but there would be a separate TAE parameter that selects which suffix plane is to be actually used by the program.

Some programs may be capable of processing a selected subarea of the input core data. Most of these programs use the input subcube specifier to select the input data that are passed to the program and then use other TAE parameters to specify the part that will actually be processed. The exact method of specifying this process selection will vary according to the needs of the individual programs.

For most programs that process an input file and put the results into an output file, the size of the output file will correspond to the size of the part of the input file selected by the input subcube specifier. (In some cases, the output file might be larger than the input subcube specifier due to the addition of new suffix planes. Also, a program that performs a geometric transformation might produce an output file that has a spatial size that is different from the input file.) Selected input data that are not processed will normally be propagated unchanged to the output file. For example, if the definition of a program consists of processing only core data, then any selected input suffix data will be propagated unchanged to the output file.

The SFROM TAE parameter specifies the subcube using a single string for all three dimensions of the cube. The order of the three dimensions is normally "samples:lines:bands". If a dimension is left blank, all the data for that dimension is selected. The default value of NULL for SFROM selects the entire cube.

There also exists an SFROM PDF file that is not used for invoking an actual ISIS program. Instead, this provides on-line help information that is more extensive than the help information in each individual program. At the TAE prompt type:

    TAE>t sfrom
    ?> help *
    ?> help sfrom
For a more detailed description of SFROM and more examples look in Appendix D-Subcube Specifier. Note that this includes additional capabilities that are not illustrated by the following examples.

6.2 - INPUT SUBCUBE SPECIFIER EXAMPLES

BASIC FORMAT:
           sfrom = "ss-es(sinc):sl-el(linc):sb-eb(binc)"

  where ss    = starting sample
        es    = ending sample
        sinc  = sample increment
        sl    = starting line
        el    = ending line
        linc  = line increment
        sb    = starting band
        eb    = ending band
        binc  = band increment

EXAMPLE:

     INPUT CUBE FILE = 150 samples x 150 lines x 255 bands
  (Assuming this cube is band sequential, it also has 12 backplanes)
      (Examples of suffix selection are the very last examples)
 
1)  Select from input file a subcube that is 100x100x10, starting
    with sample 15, line 20 and band 60
                sfrom="15-114:20-119:60-69"

2)  Select the same 100 samples and 100 lines as above,
    but all 255 bands
                sfrom="15-114:20-119:"  (samples:lines:bands left blank)

3)  Select all 150 samples and 150 lines of input cube,
    but just bands 60-69
                sfrom="::60-69"  (samples and lines left blank)

4)  Select same area as example 1, except every-other sample, line,
    and band. (The result will actually be 50x50x5)
                sfrom="15-114(2):20-119(2):60-69(2)"

5)  Select every-other sample, every third line and every 10th band
                sfrom="(2):(3):(10)"

SPECIAL CHARACTERS:

6) * = maximum possible value of a dimension (sample,line or band)

   Select an entire cube except starting at sample 3, line 4 and
   band 5
                sfrom="3-*:4-*:5-*"


7) ~ = Exclude values or ranges enclosed by ()

    Exclude lines 3 thru 10 and bands 240 thru 255, keep all samples
               sfrom=":~(3-10):~(240-255)"

8) # = Given a starting point, a value following the # sign is the
       number of values selected

    Select 100 bands starting with band 10
          sfrom="::10#100"
  
    Select first 10 samples, and 10 lines starting at 25 
          sfrom="1#10:25#10:"

SUFFIX SELECTION:

9) S or s = Suffix selection
   
     Select all lines and all samples, the first 10 bands,
     (every other band), and the first 3 backplanes
            sfrom="::1-10(2),S(1-3)"

     Select all samples, all lines, all bands, but exclude all
     backplanes
            sfrom="::S(~(1-*))"

     Select all samples, all lines, all bands, specific backplanes
     3,5,7,8,10,12 (excluding 9 and 11)
            sfrom="::s(3,5,7-12~(9,11))"

10) If a subcube specification is very complex or is to be used
    a repeated number of times, then the specification can be 
    written to a text file, such as subspec.txt.
            sfrom="<subspec.txt>"

7.0 - INSTRUMENT SPECTRAL LIBRARY FILES

Instrument spectral library (ISL) files are used for storing collections of spectra and associated header information that identifies each spectrum. All the spectra in a given ISL file have the same set of wavelengths, which corresponds to one of the observing modes of a given instrument. Different ISL files can contain different sets of wavelengths. ISL files can be used for storing data obtained from laboratory spectral measurements. ISL files can also be used for storing spectra (or average spectra) that are extracted from cubes of data produced by flight instruments.

An ISL file is a special type of cube file that is stored in BIP (band interleaved by pixel) storage order. The actual spectral data are stored in the core of the cube, and the spectral header data are packed into a set of backplanes on the cube. This storage organization allows rapid searching through the list of spectra in a file since each spectrum and its associated header data are stored as a contiguous piece of the physical disk file.

Appendix F contains an example of an ISL label. Appendix G contains a description of the spectral header that is used for entries in an ISL file.

The ISIS Spectral Libraries document describes available binary spectral library data that can be converted to ISL format with the "specpr2isl" and "spam2isl" programs. The document on Exchanging Data with Other Software Systems describes how to import ASCII spectral library data using the "asc2tbl", "hdrkeys", and "tbl2isl" programs. The "islist" program lists spectral header data from an ISL file.

The Region of Interest function in the "cv" display program writes average spectra for a set of arbitrary spatial regions to an ISL file. These data can then be used by the "unmix" program for classification based on spectral unmixing.

8.0 - TABLE FILES

One of the major types of data files handled by ISIS is the Table file. A typical example of a Table file is an average spectrum file containing data for a given spatial area. Such a data file can be produced by the "cubespec" program or the average spectrum functions in the "cv" display program. Each row of the table corresponds to one wavelength band of the spectrum and contains several different data values, e.g., band number, wavelength, average spectrum value, standard deviation, etc. Different table files can have different numbers of rows or columns in order to serve various instrumental and analysis objectives.

8.1 - BASIC TABLE FILE STRUCTURE

ISIS Table files are stored in a format that is similar to standard PDS labeled files. (As with cube files, there are various reasons why the ISIS Table files are not identical to PDS table files.) In a manner similar to cube files, the beginning of the file contains a descriptive label area. Appendix C contains an example label for an ISIS Table file. A Table file is required to contain a History data object, which is the same as the cube file History object. The actual table data are stored in a binary format in a Table data object within the Table file.

In a cube file, the name of the data object that contains the cube data is always "QUBE". However, in a Table file, the name of the data object that contains the table data is arbitrary and can vary from one Table file to another. One reason for this is to allow a single Table file to contain more than one table data object. (Duplicate object names are not permitted within one file.) In some cases, the name of the Table data object is determined by the application program. In other cases, the name of the Table data object can be specified by the user.

Note that a cube file can contain one or more Table objects in addition to the QUBE and HISTORY objects.

8.2 - TABLE DATA OBJECT FORMAT AND LABEL KEYWORDS

A Table data object contains rows of data stored in a binary format. The number of rows contained in a Table object is recorded in the ROWS keyword in the Table data object description in the label.

The number of columns of data contained in each row of the table is recorded in the COLUMNS label keyword. Each different column is identified by a unique name, which is recorded in the array of values in the COLUMN_NAME keyword. Note that column names are always recorded in the label using upper-case characters. User specified column names are not case-sensitive, i.e. any lower-case characters are converted to upper-case before being used.

Each column can contain either a scalar value or an array of values. (Most Table objects contain only scalar values.) The number of elements in each column is recorded in the COLUMN_ITEMS keyword. The data type of each element of each column is recorded in the COLUMN_ITEM_TYPE and COLUMN_ITEM_BYTES keywords. Most tables use only INTEGER*4 and REAL*4 data types, but a variety of other data types are supported, e.g., INTEGER*2 and CHARACTER*n (character strings of specified length n).

The table object description can optionally include the AUX_ROW_DEFINITION group of keywords. This group contains keyword names of the form name_NOTE, where "name" is the column name that is recorded in the COLUMN_NAME keyword. The values of these keywords are text strings that describe the meanings of the columns.

8.3 - TRANSLATING TABLE DATA BETWEEN BINARY FORMAT AND ASCII FORMAT

The "tbl2asc" program translates table data from a standard ISIS binary table file into an ASCII text format so that it can be printed or examined with a text editor. The default operation of this program will translate all columns of table data using the default formats that are recorded in the COLUMN_ITEM_FORMAT label keyword. The user can also specify other desired formats for the binary to text conversion and/or select a subset of the columns to be converted. Multiple text lines will be used for each row of table data if the ASCII format of the converted columns is too long to fit within the user-specified line length.

The "asc2tbl" program translates table data from an ASCII text format into a standard ISIS binary table file. This can be used for converting the output of the "tbl2asc" program back into the standard ISIS binary table format. This allows the use of a text editor for editing values in a table file. The "asc2tbl" program can also be used for translating other types of text table data, e.g., "foreign" text table data that contains no header information.

9 - UNDERSTANDING THE FILE_STATE KEYWORD

The I/O for ISIS cube files and table files is buffered, i.e., part of the data for a file is held in memory and is not actually written to the file until the file is closed. This improves processing efficiency. However, when a new file is opened for creation or an existing file is opened for update (Read/Write) access, the file will not be properly closed if a system crash occurs or if the program is aborted (either due to a program malfunction or due to user action). This results in a possibility that the file contains corrupted data. When this happens, the FILE_STATE label keyword is set to "DIRTY" and most ISIS applications normally refuse to process this potentially corrupted data.

ISIS includes a keyword called FILE_STATE in every ISIS cube (qube), table, and Instrument Spectral Library (ISL) data file. This keyword will be set to either CLEAN or DIRTY. Each time the cube is opened this keyword will be checked. If the FILE_STATE is equal to CLEAN, then the program will continue on normally. However, if the FILE_STATE is DIRTY, then the application will halt with the appropriate error message.

When a FILE_STATE becomes DIRTY, it indicates that something has gone wrong in a previously run application. ISIS will always set the FILE_STATE to DIRTY when the file is being opened for writing. If the application crashes and does not close the cube properly the FILE_STATE will remain DIRTY.

However, this does not always mean the file is corrupt. To help restore a file from DIRTY to CLEAN, ISIS has an application called "cleanlab". "cleanlab" will modify the FILE_STATE keyword in the label to a CLEAN state. This program should be used with caution as the contents of the file may not be valid when an ISIS file is left in a DIRTY state.

9.1 - USING ISIS_IGNORE_INTEGRITY

Along with the "cleanlab" application, ISIS has incorporated an environment variable called ISIS_IGNORE_INTEGRITY. This is provided to users so that, in short, the value of the FILE_STATE keyword in the label is ignored when opening an existing file. This means that the state (integrity) of each ISIS cube (qube), table, and Instrument Spectral Library (ISL) data file is not verified to be CLEAN. If FILE_STATE is DIRTY, the file will be treated as a CLEAN file, whether or not the contents of the file are corrupt. In fact, the label and/or history may not be parsable. You may or may not know this when you choose to utilize this option.

To use this option, simply set the UNIX environment variable ISIS_IGNORE_INTEGRITY to TRUE. This can be done using (C Shell):

	% setenv ISIS_IGNORE_INTEGRITY TRUE
Note that "TRUE" can be upper-case or lower-case characters. This must be done prior to entering TAE and cannot be modified within the TAE environment. For this case, all programs that open the ISIS file will:

  1. Determine the value of the FILE_STATE keyword in the label

  2. Generate a "warning" message if the value of FILE_STATE is *not* CLEAN

  3. If the FILE_STATE is not CLEAN, it will return a success code to the calling routine as if nothing is wrong as long as all other normal operations are successful

Other than ignoring a DIRTY file indication, nothing else changes from the normal file operations. If ISIS_IGNORE_INTEGRITY does not exist or it is set to some value other than "TRUE", then ISIS file operations will proceed in the normal fashion. This is the default mode of all file opening procedures.

Note that this feature is only supported on ISIS cube files, instrument spectral library files, and table files. It has no effect on other file operations. You should be warned that with this featured "turned on", a corrupted file will be processed as if it were valid.

10.0 - PROCESSING MODES

10.1 - SINGLE INPUT FILE

Some ISIS application programs have a single input cube file but no output cube file. In this case the input file is specified by the FROM TAE parameter, and the TO parameter is not used. An example of this is the "labels" program, which produces a listing of the label and/or history of an input cube file.

10.2 - SINGLE OUTPUT FILE

Some ISIS application programs create a new output cube file without reading an input cube file. Such programs use the TO parameter but do not use the FROM or SFROM parameters.

10.3 - SINGLE INPUT FILE PLUS SINGLE OUTPUT FILE

Many of the cube processing application programs read cube data from an input file and produce a processed version of these data. There are three basic processing modes in which such programs might operate:

  1. Create new output file - processed data are put into a file being created

  2. Update input file - processed data are put back into the input file

  3. Mosaic - processed data are inserted into a sub-area of an existing file

All three of these processing modes can be found among the various ISIS applications programs. However, not all programs include all the modes.

Mosaic operations are implemented only by a few special mosaic programs.

A few programs are only capable of operating in the update mode. These programs include the FROM TAE parameter, which specifies the name of the input file that is to be updated. (These programs do not use the TO parameter since the output file is the same as the input file.)

Some of the programs are only capable of operating in the mode in which a new output file is created. These programs include the FROM and TO TAE parameters to specify the input file and new output file being created.

Many of the programs are capable of either updating the input file or creating a new output file. These programs include both the FROM and TO parameters. If the TO parameter is specified as the blank string (all blanks), then the input FROM file will be updated. If a non-blank TO is specified, then it will be the name of the new file that is created. Note that if TO specifies the same filename and extension as FROM, then the input file will NOT be updated. (See the 10.5 - File Output Environment section below.) Also, note that in most cases an input file cannot be updated if the input subcube specifier is used to select a sub-area of the input file.

10.4 - MULTIPLE INPUT/OUTPUT FILES

Some application programs may have multiple input and/or multiple output cube files. These make use of parameters such as FROM and FROM2 to specify the input files and parameters such as TO and TO2 to specify the output files. In most cases, these programs are only capable of reading from the input files and writing to the new output files that are being created.

10.5 - FILE OUTPUT ENVIRONMENT

By default, ISIS does not permit overwriting existing cube files, instrument spectral library files, and table files. The ISIS_FILE_OVERWRITE environment variable allows overwriting (replacing) existing files. To select this option, set ISIS_FILE_OVERWRITE to TRUE using (C Shell):

      % setenv ISIS_FILE_OVERWRITE TRUE
Note that "TRUE" can be upper-case or lower-case characters. If ISIS_FILE_OVERWRITE does not exist, or has a value other than "TRUE", then file overwrite is disabled.

When an existing file is overwritten, this will result in the immediate (i.e., at time of file creation) loss of existing data within that file. This implies that one cannot simultaneously write a new "version" of a file and read from a previous "version" of the file. All new files are immediately initialized and truncated to the requested size, which by definition, results in the loss of any previously existing data in the file.

Note that this feature is not supported for file creation methods other than ISIS file I/O. In particular note that this feature does not apply to the "transformation data files" (TFILE) used by some programs. (Attempting to overwrite an existing TFILE is always a fatal error.) Furthermore, realize that utilization of this feature can result in irrecoverable data loss, thus caution is urged in the use of this feature.

11.0 - DISPLAY PROGRAMS

As stated in the first section, many of the ISIS programs do not involve interactive display. However, ISIS includes several interactive display programs. Most of these require the Interactive Data Language (IDL) from Research Systems, Inc. The "qview" display program is initiated with TAE and does not require IDL.

By using IDL for ISIS display programs, users have access to easy point and click interfaces. ISIS display programs enable users to interactively display images, extract and plot spectra, query pixels, set match points and many other functions.

For more information on the display programs, see the Display Program Summary and the Notes on Use of the IDL Display Programs.

----
Documentation | Tutorials | Technical Documents | Isis Support Center (URL: http://isis.astrogeology.usgs.gov/IsisSupport/) | Home

Last updated: Oct 24 2003