pathtool Documentation

pathtool - Search and replace paths in ISIS keyword labels
pathtool is an application that will search for and replace file paths
in ISIS cube label keywords.  This application is mainly designed to
handle ISIS SPICE kernel file paths that have been traditionally  stored
as absolute paths, although they are initially referenced using
environment variables.

The need for this application arises mainly because the use of ISIS
files on other systems would fail if ISIS data areas are installed in
different directories.  This application will use the values of ISIS
shell environment variables that specify locations of ancillary data,
including SPICE kernel files, to locate files and replace existing paths
with new locations where they exist on the local system.  (See PATHS
help for details.)

It is important to note that this application does not alter or replace
file names.  pathtool will only alter the directory portion of file
paths in specified keywords.  If the names of, for example, SPICE
kernel files need to be altered, the proper procedure for this is to
rerun levinit.

Users may provide a single ISIS cube file in the FROM parameter and/or
provide a list of files in the FROMLIST parameter.  All files are
processed even if errors occur.

Some errors that may occur is that the file is not found or a keyword
specified may not have strictly string values.  pathtool will only
work on keywords that have strings as values since that is the standard
for file names in ISIS labels.  Errors will still be reported but will
not cause the application to terminate.  Note that keywords that do not
exist in the labels is not deemed an error and are silently ignored.

pathtool will report the number of paths that are found and not found.
These numbers are reported for each file processed and totals for all
files at the end of the run.  Note that these numbers are based upon
results after processing all paths with the input conditions.  If all
is well, then there will be zero for the total missing.

There is two key sources of information critical to the execution of
this application:  paths and keywords.

PATHS
Directory paths are provide in the PATHS and MYPATHS parameters.  PATHS
provides a default list of directories to standard ISIS ancillary data
(target definitions, etc) common to most missions.  MYPATHS provides a
convenient mechanism to add paths to the search list. Combining the
value of the PATHS and MYPATHS parameters creates an aggregate search
list of paths.

Note that path searches are carried out sequentially.  Thus, the first
path found to contain the ancillary data file is the one the existing
path is replaced with.  The MYLOC parameter is provided so the user may
choose to have their list of paths occur first (PREPEND) in the list of
paths or last (APPEND) after the contents of PATHS.  Users can
completely eliminate searches/replacements of paths by nulling the PATHS
keyword. Nulling both PATHS and MYPATHS parametrs essentially runs a
check of all existing keys in the label.

The default provided for PATHS specifies directories that are common
to all missions.  MYPATHS is intended to add the standard mission
specific paths for particular missions.  It can also be used to add
your own directories to search when appropriate.  (See the MYPATHS
parameter help for details.)

Note that no checks are made as to the binary compatability of either
the ISIS file being processed or the files that are found in on the
system used run the application.

A word of caution:  this application should be run on the system
intended to process the ISIS cube file.  Some installations have
unique directory paths to files based upon the binary architecture
although the distributed version of ISIS installations default paths
are the same for all architectures.

See the EXAMPLES section below on usage.  See also detailed help for
the PATHS parameter.

KEYWORDS
Keywords are specified in the KEYS and MYKEYS parameters.  KEYS
contain standard keywords in ISIS labels that are known to require
path scrutiny when moving between systems with different ISIS
installation locations.  MYKEYS is a convenient mechanism for the user
to add keywords to the standard keyword list.

Keywords are specified using a "keyspec".  A keyspec is a specification
that provides simple references to keywords in a ISIS label.  The
general form is /OBJECT/GROUP/KEYWORD.  Alternatives of this form are
/OBJECT/KEYWORD, /KEYWORD and just KEYWORD.  An example would be
/QUBE/ISIS_GEOMETRY/BASE_KERNELS.  This is a reference to the
BASE_KERNELS keyword in the ISIS_GEOMETRY group which is located within
the QUBE object in the ISIS label.  See the help for the KEYS parameter
for a detailed description.


CONTROLLING PROGRAM BEHAVIOR
The REPORT parameter is provided to allow the user to specify what
information is reported. The default (ALL) will generate verbose
reporting of file processing.  This reports missing paths, paths
that are found and keywords that are not found in the label.  Output
can be restricted to paths only (PATHS) and missing paths only
(MISSING).

Keywords are reported as "keyspecs"...the same format as provided in
KEYS and MYKEYS.  They are marked as "KeySpec".  If a keyword does
not exist in the label, they are marked as "(KEYDNE)".

The values of the keywords are listed after each keyspec.  Paths that
do not exist are marked as "Old::Missing" or "New::Missing".  Paths
that do exist are marked as "Old::Found" or "New::Found".  Virtually
all permutations of these two path conditions are possible (when not
preserving existing paths (Old::Found) you could see a situation where
the path search yields no file found (New::Missing)).

The PRESERVE parameter allows the user to check each path for existance
prior to running through the paths contained in the aggregate list
composed of PATHS and MYPATHS.  If PRESERVE=YES and the file exists in
each path retrieved from all keyspecs in the label, then the search is
not performed on that path.  If PRESERVE=NO, the search is ran on every
value regardless if it already exists or not.

UPDATE specifies if the result of the path search is written to the
labels of the input file(s).  All files are initially opened for read
only access when retrieving keywords.  The path values are internalized
and path processing is performed.  If UPDATE=YES, the results of the
search are written to the labels of the file if values have changed.
The default is to just run the processing and report the results.
Users must set UPDATE=YES to actually write the search results to input
files.

EXAMPLES
The following examples are intended to illustrate the usefullness of
this application.  Note all of the examples given are run from the
Unix command shell and not inside TAE.  The only difference would
be the use of single quotes around some of the parameters.  These
quotes should not be used in TAE.  Also, shell line continuation
character is the backslash (\).  In TAE it is the plus sign (+).

Unless otherwise specified, all examples will prepend paths specified
in MYPATHS (MYLOC=PREPEND), show all information (REPORT=ALL),
preserve existing paths that exist (PRESERVE=YES) and will not update
the labels with the results of the run (UPDATE=NO).  At the time
of this writing, these were the defaults.

1)  Check for all missing data files in a label but do not modify the
label.  This command is most useful determining if a file contains
valid paths.  If any paths are reported missing chances are subsequent
processing will fail.

pathtool FROM=isis.cub PATHS=-- REPORT=missing

2)  Process a Viking ISIS cube file and write the results back to the
input file:

pathtool FROM=viking.cub MYPATHS='$ISISVIKDATA' UPDATE=yes

3)  Process a list Mars Exploration Rover (MER) files in the
/work1/user and update each file:

ls -1 /work1/user/*.cub > merfiles.lis
pathtool FROMLIST=merfiles.lis MYPATHS='$ISISMERDATA' UPDATE=yes

4)  Process a list of THEMIS and MGS MOC images in the same run
of the application and show all paths.  Do not preserve existing
paths but replace all existing paths with new paths if they exist.
Write the results back to the input files.

ls -1 /work1/user/*themis*.cub > files.lis
ls -1 /work1/user/*moc*.cub >> files.lis
pathtool FROMLIST=files.lis MYPATHS='($ISISM01DATA,$ISISMGSDATA)' \
         PRESERVE=no REPORT=paths UPDATE=yes
	

Programmer: Kris Becker, USGS, Flagstaff, Az

Name of single ISIS file
to process

--

Input list of ISIS files
to process

--

Default of paths to use as
replacements for existing
paths

"$ISISDATA","$ISISDATA/targets"

Additional paths that user
can add to search list

--

Preference for adding MYPATHS
to search list:
APPEND  - paths will be appended
          to PATHS
PREPEND - Paths will be
          prepended to PATHS	

"PREPEND"

List of keywords to search
and process for path
replacement

"/QUBE/ISIS_INSTRUMENT/INSTPARS",		         "/QUBE/ISIS_TARGET/TARGDEF",                          "/QUBE/ISIS_INGESTION/PDS2ISIS_TRANSLATION_TABLE",                          "/QUBE/ISIS_GEOMETRY/BASE_KERNELS",                         "/QUBE/ISIS_GEOMETRY/SPACECRAFT_KERNELS",                         "/QUBE/ISIS_GEOMETRY/CAMERA_KERNELS",                         "/QUBE/ISIS_GEOMETRY/KERNLST",			 "/QUBE/IMAGE_MAP_PROJECTION/DEM_FILE"

Additional keyspecs to process

--

Specify what to report to the
user after processing
ALL
PATHS
MISSING

"ALL"

Preserve existing paths that
exist without searching for
a new one in paths provided
in PATH

"YES"

Option to write (YES) the
path changes to the file or
not (NO)

"NO"

ADDITIONAL NOTES:

Parm	Description
FROM	Name of a single ISIS cube to process. Use this option for single files. FROM files are always prepended to the list so both FROM and FROMLIST are processed.
FROMLIST	Name of a file containing a list of ISIS files. To create a list of all ISIS files in the current directory, you could use the following command: find $PWD -name '*.cub' -print > files.lis
PATHS	The default list of standard ISIS paths that are common to all ISIS cube files. This list will typically include the files in the $ISISDATA directories. These are generalized data files used in the processing of ISIS data. Paths to ISIS files, typically NAIF SPICE kernel files, are stored in label keywords as absolute path references. For (NASA) missions, ISIS configuration files found in ISIS data directories (referenced through the $ISISDATA environment variable) use environment variables to refer to these files. Once ingested into ISIS applications the environment variables must be translated to absolute paths so they can be opened properly. During subsequent processing, these files may need to be referred to for data that is required for some computations, typically involving geometric coordinates. ISIS stores paths to these files in absolute form as opposed to their original environment variables. This is precisely the need for this application...to search alternative ISIS installations that have these files stored in different directories and resolve these new paths. Once they are found, this application updates (i.e., writes) the labels with the new absolute file path references. The defaults are typically $ISISDATA and $ISISDATA/targets. Note that paths can be specified using environment variables or absolute paths. Environment variables are translated to absolute paths and used to search for files. This implies that it will use currently defined environment variables and assumes those provided are paths that exist on the system currently running this application. On Unix file systems, paths are delineated by the forward slash (/). Characters following the last forward slash is the actual name of the file. All characters between slashes are directory names. In most cases, directory and file names are case sensitive. ISIS uses environment variables to generically reference directories that contain data required to process nearly any ISIS cube data file. The ISIS initialization startup file will typically set all ISIS environment variables based upon the installation process. To set an environment variable, use the "setenv" command: setenv ISISDATA /usgs/isisd/data Values provided in the PATHS and MYPATHS parameters can include an environment variable or specify the absolute path, the complete path that results from the translation of the enviromnent variable. A typical ISIS label keyword that refers to a file will appear as: TARGDEF = "/usgs/isisd/data/targets/mars.def.4" Here, the path specification of this keyword value would be "$ISISDATA/targets/mars.def.4". pathtool processes these values by stripping off all directory names, appending the remaining file name to all the paths specified in the aggregate list (MYPATHS & PATHS) and checking to see if the file exists in the resulting path. All paths provided in the aggregate list are searched in this manner until the first full path is found (i.e., the files exists in the absolute path). Once this occurs, the search is terminated and the resulting path is what is eventually written to the label if requested (see the UPDATE parameter). Using the above TARGDEF path, the "/usgs/isisd/data/targets" portion of the value is removed leaving the filename "mars.def.4". If pathtool is run using the default values as PATHS=($ISISDATA, $ISISDATA/targets), and the environment variable ISISDATA is now set to "/usgs/isis2/lsb/data", then the result of the search would be "/usgs/isis2/lsb/data/targets/mars.def.4".
MYPATHS	This parameter provides users with a convenient mechanism to add paths of their choice to the aggregrate search list. MYPATHS will typically be one or more mission specific path but may be any directory. Below is a list of environment variables and a brief description of their association to missions: $ISISCLEMDATA - Clementine Orbiter (Moon) $ISISCSSDATA - Cassini (Saturn) data including VIMS, ISS, DISR, and UVIS instruments $ISISGALDATA - Galileo (Jupiter) including the SSI instrument $ISISIMPDATA - Mars Pathfinder including the Imager for Mars Pathfinder (IMP) $ISISLODATA - Lunar Orbiter (Moon) including IV and V $ISISMAR9DATA - Mariner 9 (Mars) $ISISMAR10DATA - Mariner 10 (Mercury) $ISISMERDATA - Mars Exploration Rovers including both MERA and MERB rovers $ISISMEXDATA - Mars Express $ISISMGSDATA - Mars Global Surveyor including the MOC and MOLA instruments $ISISM01DATA - Mars Odessey data including the THEMIS instrument $ISISNEARDATA - Near Earth Asteroid Reconnaisance mission $ISISVIKDATA - Viking including Viking Orbiter 1 and 2 $ISISVOYDATA - Voyager 1 and 2 To process a specific mission, one or more of these environment variables can be specified in the MYPATHS parameter. pathtool FROM=moc.cub MYPATHS='($ISISMGSDATA)' UPDATE=yes
MYLOC	This parameter specifies that the paths provided in the MYPATHS parameter are prepended (PREPEND) or appended (APPEND) to the PATHS parameter defaults. This defines the order of paths where files are searched for.
KEYS	To illustrate the use of the keyspec, below is a portion of a typical ISIS label: CCSD3ZF0000100000001NJPL3IF0PDS200000001 = SFDU_LABEL /* File Structure */ LABEL_RECORDS = 30 ... ^QUBE = 66 OBJECT = QUBE /* Qube object description */ AXES = 3 AXIS_NAME = (SAMPLE,LINE,BAND) ... GROUP = ISIS_GEOMETRY BASE_KERNELS = ("/usgs/isisd/data/mk98264a.tls", "/usgs/isisd/galdata/mk00062b.tsc", "/usgs/isisd/data/pck00005.tpc") SPACECRAFT_KERNELS =("/usgs/isisd/galdata/s990114a.bsp", "/usgs/isisd/galdata/s000131a.bsp") END_GROUP = ISIS_GEOMETRY ... END In this example, /QUBE/ISIS_GEOMETRY/BASE_KERNELS refers to the BASE_KERNELS keyword in the ISIS_GEOMETRY group which is contained in the QUBE object. The keyspec to the LABEL_RECORDS keyword is /LABEL_RECORDS or just LABEL_RECORDS. And /QUBE/AXIS_NAME is the keyspec to the AXIS_NAME keyword in the QUBE object. This keyword has no group name in the keyspec.
MYKEYS	MYKEYS is a provided as a convenient method to add keywords to be retrieved from each file and its values to be searched as paths. Note that these keywords must have string values in order for them to be processed at all. The form of values provided in this parameter are keyspecs, same as is for the KEYS parameter. They follow the form /OBJECT/GROUP/KEYWORD or /OBJECT/KEYWORD. See the KEYS parameter for details of this format and how they are processed. These keywords are processed in addition to the ones specified in the KEYS parameter. Users can add keywords to be processed that are not included in the KEYS parameter using this parameter.
REPORT	REPORT controls output content when reporting the results of the search. REPORT=ALL will all values as retrieved from the keywords in the label, the results of directory searchs for each value even if no changes were required and any keywords that do not exist in the label. Note that some old and new paths will be the same value. REPORT=PATHS will report missing and found paths, both original values as retrieved from labels and those after the directory search is performed. It will not report keywords that do not exist in the label. REPORT=MISSING reports only files that are still missing after the path search has been performed. If all files exist, you should only see keyspecs reported. This mode is helpful to quickly determine which files do not exist on the system running the application.
UPDATE	UPDATE is used to write the results of the resulting search to the ISIS file(s) where the original path values are retrieved from. If UPDATE=YES, then all new paths that are found in the search are written back to the keywords they were retrieved from.