Home

Quick Links

Software Manual
GitHub
API Reference

Documentation Versions

Latest Release
Dev
8.3.0
8.2.0
8.1.0
8.0.0
7.2.0
7.1.0
7.0.0
6.0.0
3.9.0
3.5.0
USGS

ISIS Application Documentation

isisdataeval

Printer Friendly View | TOC | Home

Evaluate ISISDATA structure and validity

Overview Parameters Example 1 Example 2 Example 3

Description

isisdataeval reads the contents of an ISISDATA area and verifies its contents. This is done by traversing the DATADIR directory and finding all kernel.????.db and kernel.????.conf. The contents of these files contain configurations of kernel file patterns that are used by spiceinit to attach/associate all required NAIF SPICE kernels to individual image cubes after ingestion into ISIS. Every File keyword found in a Selection group in databases is expanded using the same process applied in spiceinit. spiceinit translates (environment) variables using special ISIS translation values (e.g., mission names) and numerical versioning to resolve file naming patterns into absolute path references. A valid ISISDATA setup will result in valid formulations of absolute file name paths in File keyword values to exising files.

The second aspect of this application is use this as a tool to help assess problems encountered with user installations of ISISDATA. There are numerous occurances of users having problems with local ISISDATA installations. Many of these problems are related to the integrity of the local installation. isisdataeval computes a file hash value for each file in the DATADIR directory and all its subdirectories. This means it will also inspect calibration files that are in the ISISDATA installation. It will also calculate a volume hash. This hash is a running hash of all the files combined into a single hash value. If the TotalVolumeHash value does not match values computed in others, particularly the USGS ISISDATA source, then its likely the installation failed and/or files are corrupted/missing.

Goals and Objectives

Motivation for this tool is to validate the ISISDATA installation on a local processing system. Recent changes to how the USGS provides ISISDATA to ISIS users separates the kernel configuration (USGS/AWS) from the kernel download resources (typically NAIF). This can be a useful tool to quickly determine if spiceinit failures may be due to missing files or improper local ISIS installation. It can also be used to validate a local ISISDATA install where kernel file download filters have been applied to ISISDATA installation processing. For example, AWS S3 storage does not support symbolic links to files, so a great deal of effort has been made to identify and report these types of files (they lead to copies/redundancy of the linked file). When given the $ISISDATA directory (the default), this tool will inspect all files and directories contained in the installation for a valid ISISDATA configuration. As it pertains to isisdataeval, a valid ISISDATA installation is confirmed when all the contents of the kernel database (DB) and configuration files are translated into files that exist in the system. isisdataeval does not, however, confirm all necessary or required files have been downloaded to the local system, but it can be verified by comparing results of the application on each install of ISISDATA.

ISISDATA Kernel Architecture

There are two types of kernel DB files that are evaluated - the kernel.????.db database lookup file and the kernel.????.conf configuration file.The kernel DB files contain a sequence of Selection groups that have one or more File PVL keyword/values entries. Each of the File keywords may have one or two values. These values, when combined properly, must resolve to an existing file within the ISISDATA directory structure to be valid. For the value in the keyword form File = "file_spec", "file_spec" may start with a "$" indicating a special value that ISIS translates to an absolute path, or an absolute path to a file. "file_spec" may also contain a contiguous series of "????" that corresponds directly to the position of characters in the file name that are numerical values indicating a version of the file referenced. ISIS will search for files that satisfy the pattern and then choose the highest version of that file pattern.

In the two value case, the general form is File = ("mission", "file_spec") where "mission" refers to a keyword in the DataDictionary group of an IsisPreferences file. The value of this keyword is a path that may contain an environment variable, such as "$ISISDATA", and additional path specifications. The second value of this keyword contains the remaining portion of a valid path that may or may not contain a file pattern such as "kernels/pck/moon_pa_de421_1900-????.bpc". A valid two value File keyword may look like File = ("osirisrex", "kernels/pck/moon_pa_de421_1900-????.bpc"). If the environment variable $ISISDATA is set to ISISDATA=/opt/isis/data, the fully resolved file name will have the form /opt/isis/data/osirisrex/kernels/pck/moon_pa_de421_1900-2050.bpc. Here is an example of a Mariner10 CK kernel.?????.db file: 

# The times in this file come from the naif toolkit application "spacit", with
#  a 1 minute padding on the start and end times.

Object = SpacecraftPointing
  RunTime = 2010-03-03T16:25:09

  Group = Dependencies
    SpacecraftClockKernel = $mariner10/kernels/sclk/mariner10.0001.tsc
    LeapsecondKernel      = $base/kernels/lsk/naif0009.tls
  End_Group

  Group = Selection
    Time = ("1974 MAR 28 19:17:26.574 TDB", "1975 MAR 17 09:57:46.641 TDB")
    Match = ("Instrument","InstrumentId","M10_VIDICON_A")
    File = $mariner10/kernels/ck/MERCURY_MARINER_10_A.bc
    Type = Smithed
  End_Group

  Group = Selection
    Time = ("1974 MAR 28 19:17:26.574 TDB", "1975 MAR 17 09:57:46.641 TDB")
    Match = ("Instrument","InstrumentId","M10_VIDICON_B")
    File = $mariner10/kernels/ck/MERCURY_MARINER_10_B.bc
    Type = Smithed
  End_Group

  Group = Selection
    Time = ("1973 NOV 03 21:04:54.460 TDB", "1975 MAR 17 10:13:56.517 TDB")
    Match = ("Instrument","InstrumentId","M10_VIDICON_A")
    File = $mariner10/kernels/ck/MARINER_10_A_gem.bc
    Type = Reconstructed
  End_Group

  Group = Selection
    Time = ("1973 NOV 03 21:05:36.460 TDB", "1975 MAR 17 10:14:38.517 TDB")
    Match = ("Instrument","InstrumentId","M10_VIDICON_B")
    File = $mariner10/kernels/ck/MARINER_10_B_gem.bc
    Type = Reconstructed
  End_Group

End_Object
End

Kernel configuration files are of the form "kernel.????.conf" where "????" corresponds to an ordered version number sequence that specifies the versions of a kernel configuration file. These types of kernel configuration files typically contain patterns of alternative kernel DB file names thus providing support for more complex NAIF SPICE kernel configurations. They often contain more than one kernel DB File pattern specification in Selection groups. When utilized in the camera kernels (CK) mission directory, it can be used to specify two or more independent kernel DB files that are required to compute spacecraft/instrument attitude/pointing epoch states. below is an example of a multi-kernel CK specification for the MESSENGER MDIS instruments. The File keywords are the two-valued form where "messenger" must map to an entry in the DataDictionary group of a loaded IsisPreferences file. 

Object = Instrument

  Group = Selection
    Match = ("Instrument","InstrumentId","MDIS-WAC")
    File = ("messenger", "kernels/ck/mdis_kernels.????.db")
    File = ("messenger", "kernels/ck/messenger_mdis_att_kernels.????.db")
    File = ("messenger", "kernels/ck/messenger_kernels.????.db")
  End_Group

  Group = Selection
    Match = ("Instrument","InstrumentId","MDIS-NAC")
    File = ("messenger", "kernels/ck/mdis_kernels.????.db")
    File = ("messenger", "kernels/ck/messenger_mdis_att_kernels.????.db")
    File = ("messenger", "kernels/ck/messenger_kernels.????.db")
  End_Group

End_Object
End

Note the Dependencies group, or any other group/keyword section, are ignored and not evaluated/validated by this application.

Validation

The procedure to validate the ISISDATA area will start with the DATADIR directory provided by the user. The default refers to the top level ISIS ancillary data istallation directory specifed by the environment variable $ISISDATA. However, this can be any existing directory in the $ISISDATA hierarchy, such as DATADIR=$ISISDATA/osirisrex. isisdataeval will evaluate every file in that directory and all subdirectories for validity.

The total number files are counted and sizes of every file is accumulated as the top directory is traversed. The algorithm to validate ISISDATA searches for two basic file patterns in each directory. First, all kernel configurations are searched for using the file pattern form kernel.????.conf. For all configurations found, the highest verson of the config file is selected and its contents read. The highest version of each file pattern found in all File keywords in Selection groups are determined. These files are treated as kernel DB files and evaluated as described below. Note these file names are strongly recommended not to be of the generic form kernel.????.db to minimize confusion and erroneous behavior.

After the configuration files are processed, a second search is made for kernel DB files using the form kernels.????.db. The highest version of these files are selected and its contents are read. For every File specification in a Selection group, the highest version is determined and then checked for existance. If no files are found, which typically occurs when looking for the highest version, it is reported as missing. The missing file, the file that contains the reference to it, and the status of the file are written to the file name specifed in the TOISSUES parameter.

There are occurances of kernel DB files that contain no specification or content at all. These are oddball files in the system and are also reported as invalid. All files contained in the TOISSUES output file would lead to errors in spiceinit for images obtained from that particular mission.

Categories

Related Objects and Documents

Applications

History

Kris J. Becker2023-01-09 Original version
U.S. Department of the Interior | U.S. Geological Survey
ISIS | Privacy & Disclaimers | Astrogeology Research Program
To contact us, please post comments and questions on the USGS Astrogeology Discussion Board
To report a bug, or suggest a feature go to: ISIS Github
File Modified: 02/21/2025 19:28:11
X

Files: PREFERENCES

Description

Defines data directory substitutions that is assumed to be part of the ISISDATA directory hierachy. Specifying an ISIS Preferences file using "-pref=PREFERENCES" is equivalent to using this parameter except that using this parameter will log the Preferences file used. If both are options are given, the contents of this PREFERENCES file overrides them all loaded before it. isisdataeval uses the contents of the DataDictionary group for ISIS mission and environment variable file path translations. This DataDictionary group is reported to the user and logged to the print.prt file.

Type string
Internal Default None
Close Window
X

Files: DATADIR

Description

Top data directory that is assumed to be part of the ISISDATA directory hierachy. The default value of this parameter will evaluate all of ISISDATA that is installed in the specified directory.

Type string
Default $ISISDATA
Close Window
X

Files: ISISDATA

Description

This may be required to efficiently switch to different installations of ISISDATA. It is needed because the translations within the kernel DB and config files will eventually translate $ISISDATA to an absolute directory name. It can be altered easily by manually adding the translation of ISISDATA into the DataDictionary after reading all the IsisPreferences files (several versions may be read for each ISIS application ran). This essentially replaces the current value of $ISISDATA without the need to expressly set an environment variable. This is accomplished by adding the value given to the DataDictionary which will be used to fully expand ISIS file names.

Type string
Internal Default $ISISDATA
Close Window
X

Files: TOISSUES

Description

Provide the name of a file to write problematic (e.g., missing, symlinks, etc...) kernels encountered during validation. There will be six columns separated by a comma. The columes are "status, filespec, sourcespec, source, target, category". where the status indicates the issue/problem with the file, filespec is the original form of the name, sourcespec is the name of the versioned patterned source db/conf, target is the fully expanded filespec, and category is the type or category of the source reference. An brief example of the contents: 

status, filespec, sourcespec, source, target, category
missing,$apollo17/kernels/ik/apollo17_panoramic.????.ti,/opt/isisdatafull/apollo17/kernels/ik/kernels.????.db,/opt/isisdatafull/apollo17/kernels/ik/kernels.0002.db,/opt/isisdatafull/apollo17/kernels/ik/kernels.0002.db,null
missing,$base/kernels/spk/nep096.bsp,/opt/isisdatafull/base/kernels/spk/kernels.????.db,/opt/isisdatafull/base/kernels/spk/kernels.0006.db,/opt/isisdatafull/base/kernels/spk/kernels.0006.db,null
missing,$base/kernels/spk/nep096.bsp,/opt/isisdatafull/base/kernels/spk/kernels.????.db,/opt/isisdatafull/base/kernels/spk/kernels.0006.db,/opt/isisdatafull/base/kernels/spk/kernels.0006.db,null
empty,/opt/isisdatafull/cassini/kernels/ik/kernels.????.db,/opt/isisdatafull/cassini/kernels/ik/kernels.0001.db,/opt/isisdatafull/cassini/kernels/ik/kernels.0001.db,/opt/isisdatafull/cassini/kernels/ik/kernels.0001.db,Instrument

Type filename
Internal Default None
Close Window
X

Files: TOINVENTORY

Description

Provide the name of a file to results of the inventory volumn in DATADIR. validation. There will be six columns separated by a comma. The columns are "status, filespec, sourcespec, source, target, category". where the status indicates the issue/problem with the file, filespec is the original form of the name, sourcespec is the name of the versioned patterned source db/conf, target is the fully expanded filespec, and category is the type or category of the source reference. An brief example of the contents: 

filespec,filepath,exists,file,symlink,target,created,createdet,modified,modifiedet,size,md5hash
/opt/isisdatafull/apollo15/calibration/ApolloPanFiducialMark.cub,/opt/isisdatafull/apollo15/calibration/ApolloPanFiducialMark.cub,true,/opt/isisdatafull/apollo15/calibration/ApolloPanFiducialMark.cub,false,/opt/isisdatafull/apollo15/calibration/ApolloPanFiducialMark.cub,2022-12-02T14:10:29,723262298.1831114,2022-08-12T22:10:19,713614288.1829929,330891,9336f9b68d0686004b0f4bdc0633ccf6
/opt/isisdatafull/apollo15/calibration/METRIC_flatfield.cub,/opt/isisdatafull/apollo15/calibration/METRIC_flatfield.cub,true,/opt/isisdatafull/apollo15/calibration/METRIC_flatfield.cub,false,/opt/isisdatafull/apollo15/calibration/METRIC_flatfield.cub,2022-12-02T14:11:58,723262387.1831114,2022-08-12T22:10:19,713614288.1829929,2099909504,c28e9708cd4b99db50e43b91faa28cc1
/opt/isisdatafull/apollo15/kernels/ck/AS15_M_REV04_SMITHED_V01.bc,/opt/isisdatafull/apollo15/kernels/ck/AS15_M_REV04_SMITHED_V01.bc,true,/opt/isisdatafull/apollo15/kernels/ck/AS15_M_REV04_SMITHED_V01.bc,false,/opt/isisdatafull/apollo15/kernels/ck/AS15_M_REV04_SMITHED_V01.bc,2022-12-02T14:10:31,723262300.1831114,2022-08-12T22:10:19,713614288.1829929,49152,44743a36fba4b394752fbf5749c626ec

Type filename
Internal Default None
Close Window
X

Files: TOERRORS

Description

Provide the name of a file to write errors (e.g., missing, symlinks, etc...) files during the inventory. Errors in the inventory are defined as any symlink (file), symlinks that reference non-existant files or files external to the DATADIR directory. And any file that cannot be opened/read are also reported. The format of this file will be as follows: 

status, filespec, sourcespec, source, target, category
error,/opt/isisdatafull/mex/kernels/dsk/MEX_SC_SA,/opt/isisdatafull/mex/kernels/dsk/MEX_SC_SA+Y_V00.OBJ,/opt/isisdatafull/mex/kernels/dsk/MEX_SC_SA,/opt/isisdatafull/mex/kernels/dsk/MEX_SC_SA+Y_V00.OBJ,openfailed
error,/opt/isisdatafull/rosetta/kernels/dsk/ROS_SC_SA,/opt/isisdatafull/rosetta/kernels/dsk/ROS_SC_SA+Y_V00.BDS,/opt/isisdatafull/rosetta/kernels/dsk/ROS_SC_SA,/opt/isisdatafull/rosetta/kernels/dsk/ROS_SC_SA+Y_V00.BDS,openfailed

Type filename
Internal Default None
Close Window
X

Options: VERIFY

Description

This parameter is provided to explicity turn on the process of verification of the ISISDATA FILE inventory. This is not needed if HASH!=None and/or a file is provided in the TOINVENTORY parameter.

Basic vertification of ISISDATA inventory will run through all the files in ISISDATA and ensure that the expanded version of the filename exists. There is at least one case this option will reveal is if a `+` character exists in a file name. It is flagged as an error here.

Type boolean
Default false
Close Window
X

Options: HASH

Description

Compute file hash values for all files found in the ISISDATA directory. Note if this parameter is given, it will invoke a complete inventory of the DATADIR directory and compute the HASH for the file using the specified algorithm. Not only does it compute the hash for the file, it also computes a a complete volume hash of all files in the inventory. This option can be most useful for verifying ISISDATA installations are valid.

BE FOREWARNED!! Appling this option will significantly slow down the the execution of this application. It essentially reads all the files in the DATADIR path and all subdirectories (i.e., the volume) and reports a single hash value for the complete DATADIR contents. If DATADIR = ISISDATA, the hash value for the complete ISISDATA volume is computed as a single value and reports it to the console and in the ISIS log file.

The short description of hash algorithms below are provided from this link. As such, each file that has a hash value can easily compared for the same contents using programs provided by most computer operating systems. Linux provides the md5sum application. On the Mac platform, it is just md5.

The TOINVENTORY file will contain a md5hash column similar to this example: 

filespec,size,md5hash
/opt/isisdatafiltered/apollo15/calibration/ApolloPanFiducialMark.cub,330891,9336f9b68d0686004b0f4bdc0633ccf6
To compare this file directly, you can use the command: 
md5  /opt/isisdatafiltered/apollo15/calibration/ApolloPanFiducialMark.cub

Type string
Default NOHASH
Option List:
Option Brief Description
NOHASH Hashes will not be computed This indicates that no hash values will be computed for the the ISISDATA inventory (default).
MD5 Compute a crypto hash using the MD5 algorithm The MD5 hash function produces a 128-bit hash value. It was designed for use in cryptography, but vulnerabilities were discovered over the course of time, so it is no longer recommended for that purpose. However, it is still used for database partitioning and computing checksums to validate files transfers.
SHA1 Compute a crypto hash using the SHA-1 algorithm

SHA stands for Secure Hash Algorithm. The first version of the algorithm was SHA-1, and was later followed by SHA-2 (see below).

Whereas MD5 produces a 128-bit hash, SHA1 generates 160-bit hash (20 bytes). In hexadecimal format, it is an integer 40 digits long. Like MD5, it was designed for cryptology applications, but was soon found to have vulnerabilities also. As of today, it is no longer considered to be any less resistant to attack than MD5.

SHA256 Compute a crypto hash using the SHA-256 algorithm

The second version of SHA, called SHA-2, has many variants. Probably the one most commonly used is SHA-256, which the National Institute of Standards and Technology (NIST) recommends using instead of MD5 or SHA-1.

The SHA-256 algorithm returns hash value of 256-bits, or 64 hexadecimal digits. While not quite perfect, current research indicates it is considerably more secure than either MD5 or SHA-1.

Performance-wise, a SHA-256 hash is about 20-30% slower to calculate than either MD5 or SHA-1 hashes.

Close Window
X

Options: HASHBUFFER

Description

This allows users to have more control of the size of buffer to read inventory file data to compute hashes. Specify this value in megabytes ( 1024 * 1024 * HASHBUFFER). The buffer is not allocated if HASH=NOHASH.

Type integer
Default 256
Close Window