ISIS Documentation

isisdataeval

Evaluate ISISDATA structure and validity

isisdataeval reads the contents of an ISISDATA area and verifies its contents. This is done by traversing the DATADIR directory and finding all kernel.????.db and kernel.????.conf. The contents of these files contain configurations of kernel file patterns that are used by spiceinit to attach/associate all required NAIF SPICE kernels to individual image cubes after ingestion into ISIS. Every File keyword found in a Selection group in databases is expanded using the same process applied in spiceinit. spiceinit translates (environment) variables using special ISIS translation values (e.g., mission names) and numerical versioning to resolve file naming patterns into absolute path references. A valid ISISDATA setup will result in valid formulations of absolute file name paths in File keyword values to exising files.

The second aspect of this application is use this as a tool to help assess problems encountered with user installations of ISISDATA. There are numerous occurances of users having problems with local ISISDATA installations. Many of these problems are related to the integrity of the local installation. isisdataeval computes a file hash value for each file in the DATADIR directory and all its subdirectories. This means it will also inspect calibration files that are in the ISISDATA installation. It will also calculate a volume hash. This hash is a running hash of all the files combined into a single hash value. If the TotalVolumeHash value does not match values computed in others, particularly the USGS ISISDATA source, then its likely the installation failed and/or files are corrupted/missing.

Goals and Objectives

Motivation for this tool is to validate the ISISDATA installation on a local processing system. Recent changes to how the USGS provides ISISDATA to ISIS users separates the kernel configuration (USGS/AWS) from the kernel download resources (typically NAIF). This can be a useful tool to quickly determine if spiceinit failures may be due to missing files or improper local ISIS installation. It can also be used to validate a local ISISDATA install where kernel file download filters have been applied to ISISDATA installation processing. For example, AWS S3 storage does not support symbolic links to files, so a great deal of effort has been made to identify and report these types of files (they lead to copies/redundancy of the linked file). When given the $ISISDATA directory (the default), this tool will inspect all files and directories contained in the installation for a valid ISISDATA configuration. As it pertains to isisdataeval, a valid ISISDATA installation is confirmed when all the contents of the kernel database (DB) and configuration files are translated into files that exist in the system. isisdataeval does not, however, confirm all necessary or required files have been downloaded to the local system, but it can be verified by comparing results of the application on each install of ISISDATA.

ISISDATA Kernel Architecture

There are two types of kernel DB files that are evaluated - the kernel.????.db database lookup file and the kernel.????.conf configuration file.The kernel DB files contain a sequence of Selection groups that have one or more File PVL keyword/values entries. Each of the File keywords may have one or two values. These values, when combined properly, must resolve to an existing file within the ISISDATA directory structure to be valid. For the value in the keyword form File = "file_spec", "file_spec" may start with a "$" indicating a special value that ISIS translates to an absolute path, or an absolute path to a file. "file_spec" may also contain a contiguous series of "????" that corresponds directly to the position of characters in the file name that are numerical values indicating a version of the file referenced. ISIS will search for files that satisfy the pattern and then choose the highest version of that file pattern.

In the two value case, the general form is File = ("mission", "file_spec") where "mission" refers to a keyword in the DataDictionary group of an IsisPreferences file. The value of this keyword is a path that may contain an environment variable, such as "$ISISDATA", and additional path specifications. The second value of this keyword contains the remaining portion of a valid path that may or may not contain a file pattern such as "kernels/pck/moon_pa_de421_1900-????.bpc". A valid two value File keyword may look like File = ("osirisrex", "kernels/pck/moon_pa_de421_1900-????.bpc"). If the environment variable $ISISDATA is set to ISISDATA=/opt/isis/data, the fully resolved file name will have the form /opt/isis/data/osirisrex/kernels/pck/moon_pa_de421_1900-2050.bpc. Here is an example of a Mariner10 CK kernel.?????.db file: 

# The times in this file come from the naif toolkit application "spacit", with
#  a 1 minute padding on the start and end times.

Object = SpacecraftPointing
  RunTime = 2010-03-03T16:25:09

  Group = Dependencies
    SpacecraftClockKernel = $mariner10/kernels/sclk/mariner10.0001.tsc
    LeapsecondKernel      = $base/kernels/lsk/naif0009.tls
  End_Group

  Group = Selection
    Time = ("1974 MAR 28 19:17:26.574 TDB", "1975 MAR 17 09:57:46.641 TDB")
    Match = ("Instrument","InstrumentId","M10_VIDICON_A")
    File = $mariner10/kernels/ck/MERCURY_MARINER_10_A.bc
    Type = Smithed
  End_Group

  Group = Selection
    Time = ("1974 MAR 28 19:17:26.574 TDB", "1975 MAR 17 09:57:46.641 TDB")
    Match = ("Instrument","InstrumentId","M10_VIDICON_B")
    File = $mariner10/kernels/ck/MERCURY_MARINER_10_B.bc
    Type = Smithed
  End_Group

  Group = Selection
    Time = ("1973 NOV 03 21:04:54.460 TDB", "1975 MAR 17 10:13:56.517 TDB")
    Match = ("Instrument","InstrumentId","M10_VIDICON_A")
    File = $mariner10/kernels/ck/MARINER_10_A_gem.bc
    Type = Reconstructed
  End_Group

  Group = Selection
    Time = ("1973 NOV 03 21:05:36.460 TDB", "1975 MAR 17 10:14:38.517 TDB")
    Match = ("Instrument","InstrumentId","M10_VIDICON_B")
    File = $mariner10/kernels/ck/MARINER_10_B_gem.bc
    Type = Reconstructed
  End_Group

End_Object
End

Kernel configuration files are of the form "kernel.????.conf" where "????" corresponds to an ordered version number sequence that specifies the versions of a kernel configuration file. These types of kernel configuration files typically contain patterns of alternative kernel DB file names thus providing support for more complex NAIF SPICE kernel configurations. They often contain more than one kernel DB File pattern specification in Selection groups. When utilized in the camera kernels (CK) mission directory, it can be used to specify two or more independent kernel DB files that are required to compute spacecraft/instrument attitude/pointing epoch states. below is an example of a multi-kernel CK specification for the MESSENGER MDIS instruments. The File keywords are the two-valued form where "messenger" must map to an entry in the DataDictionary group of a loaded IsisPreferences file. 

Object = Instrument

  Group = Selection
    Match = ("Instrument","InstrumentId","MDIS-WAC")
    File = ("messenger", "kernels/ck/mdis_kernels.????.db")
    File = ("messenger", "kernels/ck/messenger_mdis_att_kernels.????.db")
    File = ("messenger", "kernels/ck/messenger_kernels.????.db")
  End_Group

  Group = Selection
    Match = ("Instrument","InstrumentId","MDIS-NAC")
    File = ("messenger", "kernels/ck/mdis_kernels.????.db")
    File = ("messenger", "kernels/ck/messenger_mdis_att_kernels.????.db")
    File = ("messenger", "kernels/ck/messenger_kernels.????.db")
  End_Group

End_Object
End

Note the Dependencies group, or any other group/keyword section, are ignored and not evaluated/validated by this application.

Validation

The procedure to validate the ISISDATA area will start with the DATADIR directory provided by the user. The default refers to the top level ISIS ancillary data istallation directory specifed by the environment variable $ISISDATA. However, this can be any existing directory in the $ISISDATA hierarchy, such as DATADIR=$ISISDATA/osirisrex. isisdataeval will evaluate every file in that directory and all subdirectories for validity.

The total number files are counted and sizes of every file is accumulated as the top directory is traversed. The algorithm to validate ISISDATA searches for two basic file patterns in each directory. First, all kernel configurations are searched for using the file pattern form kernel.????.conf. For all configurations found, the highest verson of the config file is selected and its contents read. The highest version of each file pattern found in all File keywords in Selection groups are determined. These files are treated as kernel DB files and evaluated as described below. Note these file names are strongly recommended not to be of the generic form kernel.????.db to minimize confusion and erroneous behavior.

After the configuration files are processed, a second search is made for kernel DB files using the form kernels.????.db. The highest version of these files are selected and its contents are read. For every File specification in a Selection group, the highest version is determined and then checked for existance. If no files are found, which typically occurs when looking for the highest version, it is reported as missing. The missing file, the file that contains the reference to it, and the status of the file are written to the file name specifed in the TOISSUES parameter.

There are occurances of kernel DB files that contain no specification or content at all. These are oddball files in the system and are also reported as invalid. All files contained in the TOISSUES output file would lead to errors in spiceinit for images obtained from that particular mission.

Categories

Related Objects and Documents

Applications

History

Kris J. Becker2023-01-09 Original version