The isisminer application assists in the identification, manipulation, and output of data from a variety of data sources. One main source of input is caminfo. Processing in the isisminer program is controlled by the contents of a PVL-based, object-structured, configuration file provided in the CONFIG parameter. There are two main PVL objects used in the configuration of an isisminer program run: the IsisMiner object and the Strategy object.

Strategy Algorithms

The IsisMiner object contains a series of Strategy objects. Strategy objects are associated with a defined algorithm that performs some operation on data called a Resource. Strategy objects contain parameters that are needed by the algorithm to process Resource data. The order of Strategy objects is important as the algorithms are executed in the order they occur in the IsisMiner object. Each Strategy object must be associated with a known Strategy Type. Other objects may exist anywhere in the CONFIG file but are ignored when parsing Strategy objects unless they are part of the algorithm. One common use for other objects in the CONFIG file is to include database profiles that are used to provide access information for the DatabaseReader strategy (see below).

Data Resources

Strategy algorithms operate on a Resource. A Resource is a data container that stores a set of PVL keyword/values that can have data storage extensions (called Assets and GisGeometries). These Assets can contain anything created by Strategy objects for further analysis or processing and could be unique to a set of Strategies. Resources have three major components: keyword/values, GIS geometries and Assets. Most of the data manipulations occur on keyword/value elements. A Resource may contain a GIS geometry that is used in algorithms that work on geometric regions using GIS operators (GIS Intersect, GIS Overlap, GIS Union, StereoPair ). Currently, GIS geometries must be in well-known-text (WKT) or well-known-binary (WKB) as conversions are restricted to text representations of geometries (format requirements are the same as functions like ST_GeomFromWKT and ST_GeomFromWKB, respectively). Resources may also contain Assets. Assets are typically another list of Resources created from Strategies that are attached to a (parent) Resource. If Assets are represented as Resource lists, they can be processed by almost all Strategy algorithms using the AssetSidebar strategy (see below).

Resources can come from many sources. One of the most common input sources are databases. SQL queries can be specified directly in the DatabaseReader strategy. Each row returned when the query is executed is converted into a Resource. Other data sources are comma separated values (CSV) files (using the CsvReader strategy) where each row is a Resource and each column is a keyword/value. The keyword name in the case of a CSV file is the name of the column as specified in the header. Users can also read in ISIS control networks and process each Control Measure as a Resource. And, of course, ISIS cube files and other PVL file formats are also input sources (see PvlReader strategy). Strategies are designed to operate on a single Resource. This allows for a very narrow scope of design considerations to make coding Strategies quite easy.

Control Options using Keyword Argument Substitution

Some very clever and sophisticated Strategy configurations can be created but to make them generally useful, the user may need to change some parameters within Strategies, such as output directories or file names that can be modified at runtime without having to edit the CONFIG files directly. To allow some control options, there are two optional application parameters provided: GLOBALS and PARAMETERS.

The GLOBALS parameter can specify the name of a file that contains PVL keywords and values. The complete contents of all the keywords found in the file, regardless of Object/Group structure are loaded immediately when isisminer first runs. The keywords contained in this file can be used as substitutions in all or part of many keyword value specifications used within most Strategies (described below).

The PARAMETERS input parameter is provided so that users can create simple keyword/value pairs on the command line that can be used in substitutions of part of all keyword values used in Strategies in the same manner GLOBALS keywords are substituted. For any keyword that is contained in both the GLOBALS and PARAMETERS keyword set, keywords in PARAMETERS always take precedence. This allows users to override keywords in GLOBALS as needed during keyword lookup and replacement operations to fully determine values of keywords in real time when Strategies are executed. Since equal signs are not allowed in ISIS application parameters, keyword names and values are separated by a colon. Multiple keywords can be specified and are separated by the @ symbol. For example, PARAMETERS="inputdir:$PWD@outputdir:$PWD/Results" will create two keywords, inputdir and outputdir, with inputdir set to the full path of the current directory running the job and outputdir set to a subdirectory in the current directory called Results.

Essentially, the GLOBALS and PARAMETERS elements creates a hierarchical lookup for keyword substitutions where PARAMETERS are searched first, then GLOBALS, for argument substitution of elements within a keyword value. At times, when individual Resources drive output operations (e.g., AssetSidebars), the individual Resource keywords become available for substitution as well and are the first set of keywords searched before PARAMETERS and GLOBALS. Argument substitutions are specified using the %n symbol in keyword values. Here, the n is a number from 1 to the number of values corresponding to an associated keyword with the same name but with Args appended to it. The nth value of this keyword is itself a keyword where its value is substituted in the corresponding %n occurrence in the original keyword. For example, in the CSVReader strategy, the name of a file to read the contents from is specified in the CsvFile keyword. To allow arbitrary specification of the input file without editing the CONFIG file to change it for each use of the CONFIG file, the PARAMETERS option can be used to specify the path to a named file or the entire file name by creating a keyword called CsvFileArgs. This is the general form for creating a keyword value substitution that uses two associated keywords with the first being any keyword and the second keyword having Args appended to the first keyword which contains other keywords that will be used to substitute arguments with.

See the complete stereo example provided for isisminer that demonstrates many of the concepts discussed here and in the Strategy descriptions below. It will also provide output and describe the results of a MESSENGER MDIS stereo processing run.

Descriptions of Strategies

The following section describes all the Strategy algorithms currently available in IsisMiner. Strategies iterate through all active Resources. One of the features in isisminer that provides a lot of flexibility in the system is that Resources are never deleted unless users provide explicit instructions to do so (see ResourceManager). Each Resource has an inherent state that indicates if it is active, meaning it is available for processing, or inactive. Strategies like the Limit or Filter will evaluate values of keywords for boolean equations or strings, respectively, where true conditions keep Resource states as active, and false conditions set their states to inactive. Inactive Resources are not processed in subsequent Strategies unless explicitly reactivated (again using the ResourceManager). Strategies typically only use active Resources when normally processed. Exceptional is when a Strategy provides options to select an additional processing set of Resources (see GisOverlap, StereoPair and Sidebar). Those Resources accept a complete isisminer configuration where the object specification can be embedded within the Strategy object or a file name can be provided that contains a complete isisminer configuration (much like provided in CONFIG). This design allows users to progressively select Resources for processing and reactivate the entire list at any point in the execution of Strategies. The Strategies below provide the user with many options to help identify only data with specified/desired characteristics for evaluation and subsequent processing.

Special Strategy Keywords

There are some keywords available that can be used in every Strategy beyond their unique keywords described below. Many of these keywords provide additional processing control or options that produce information in the form of text to the standard output stream. The following tables provide descriptions of the keywords that can be used in all Strategies. Some will apply only under certain conditions which are described in the context of the descriptions. All of these keywords are optional except where indicated.

Some miscellaneous keywords are available that provide general information and/or assist users in debugging Strategies that appear to result in unexpected behavior. Here are other keywords that users can set and are available in all Strategies.

The following table describes keywords that pertain to manipulation of GIS geometries. They are available to all Strategies that use GIS geometries but only apply where indicated.

Types of Strategies

• AssetSidebar Strategy Description
• Calculator Strategy Description
• Control Network Reader Strategy Description
• CSV Reader Strategy Description
• CSV Writer Strategy Description
• Database Reader Strategy Description
• Filter Strategy Description
• GIS Intersect Strategy Description
• GIS Overlap Strategy Description
• GIS Union Strategy Description
• IsNumeric Strategy Description
• Limit Strategy Description
• Numerical Sort Strategy Description
• PDS Table Creator Description
• PDS Table Format Strategy Description
• PDS Table Reader Strategy Description
• PVL Reader Strategy Description
• Resource Manager Strategy Description
• Run Command Strategy Description
• Sidebar Strategy Description
• StereoPair Strategy Description

AssetSidebar Strategy Description

This Strategy can be used to add Assets to any active Resource. By providing the Operation keyword with a value of Create, the user can then process an Asset to be added to the active Resources (with the PvlReader or CsvReader strategies, for example). Note that when using create as a value for the Operation keyword, the specified Strategy in the AssetSidebar strategy will not be applied to any existing Assets contained in the active Resources. In fact, any Asset that exists in a Resource of the same name will be deleted and replaced with the results of the AssetSidebar processing. Note that some Strategies create Assets directly in the course of executing its algorithm and do not require an AssetSidebar to create them.

When creating Assets, users have several options to create the Resource list for the Asset. Using the CreateSource keyword with a value of Copy, a copy of all Resources in the input list is generated prior to executing the Strategies to process the Assets. The active/deactive status of each new Resource in this list is managed independently from the list outside of the AssetSidebar processing it. However, keywords, other Assets, and the GIS Geometry remain identical to those of the parent Resource. To create a completely independent copy of the Resources, set CreateSource to Clone. This option clones all the status and data of each Resource and creates a completely new Resource. To start from scratch and create an Asset by other means, use None. Defaults to Copy the Resource list.

This Strategy may also be used to apply a specified Strategy to any active Resource's Assets. For Operation set to Process, the specified Strategy in the AssetSidebar will be applied to the active Resources' Assets and replace the processed Asset in active Resources as new Assets. This mode is particularly useful for processing Assets created directly by other Strategies.

The following is an example of an AssetSidebar strategy

Object = Strategy
  Name        = TestCreateAsset
  Type        = AssetSidebar
  Asset       = AssetA
  Operation   = Create
  CreateSource = None
  Description = "Adds AssetA (Assets read from test.csv) to 
  the active Resources" 
  Object = IsisMiner 
    Object = Strategy
      Name = ReadAssetTest
      Type = CsvReader
      CsvFile = "test.csv"
      HasHeader = True
      SkipLines = 0
      IgnoreComments = False
      Delimiter = ", "
    EndObject
  EndObject
EndObject
      

Return to List of Strategy Types

Calculator Strategy Description

There are two ways equations can be specified. The basic method requires two keywords to compute, execute, and store the results in each Resource. The Equation keyword stores the complete equation to execute for each Resource and uses the Result keyword value to name the keyword to store the result of the equation.

An alternative form uses a PVL Group named Equations which contains a sequence of keywords whose values are equations. A keyword of the same name is created in each Resource with the value of each successfully computed equation. Values of previous equations can be used in subsequent equations. No other keyword may exist in the Equations group as every keyword is expected to be a valid equation.

Currently this Strategy supports scalars in the form of integers and doubles. In addition, it supports the circle constant pi and Euler's constant e. Functions, such as sin and cos, are followed by parentheses and a list of arguments. Operators such as * and + are placed in between their arguments. If a token is not recognized as a scalar, operator, or function, then it is assumed to be a keyword, and will be replaced by the corresponding value in the Resource.

All trigonometric functions expect angles in radians, not degrees so use the rads() function!

The following table shows all currently supported operators and functions sorted by precedence (0 is the highest precedence). Functions have a precedence of 1 and operators have a precedence of 2 or higher, according to order of operations. All examples are valid equations using 'x' and 'y' as variables:

The following is an example of a Calculator strategy:

Object = Strategy
  Name = Trigonometry
  Type = Calculator
  Description = "Calculate trigonometric functions of angle."
  Group = Initializers
    Sine    = 0
    Cosine  = 0
    Tangent = 0
  EndGroup
  Group = Equations
    Sine    = "sin(angle)"
    Cosine  = "cos(angle)"
    Tangent = "tan(angle)"
  EndGroup
EndObject
      

Return to List of Strategy Types

Control Network Reader Strategy Description

If the Identity keyword is not provided or its value is an empty string, each Resource is named with a unique number (starting at 0).

The following is an example of a CnetReader strategy:

Object = Strategy
  Name          = ControlNetworkReader
  Type          = CnetReader
  CnetFile      = "%1/Alph_VIS.net"
  CnetFileArgs  = "inputdir"
  Identity      = "%1_%2"
  IdentityArgs  = (PointId, SerialNumber)
  Description   = "Read from a control network file"
EndObject
      

Return to List of Strategy Types

CSV Reader Strategy Description

The CSV file must have the following properties. The rows must be delimited by a new line. The columns must be delimited by a single character. The number of columns in each row must be the same for all rows. If IgnoreComments is set to True, then Comments must be on their own line and be preceded by a #.

The following is an example of a CsvReader strategy:

Object = Strategy
  Name = ReadFilterData
  Type = CsvReader
  CsvFile        = "%1/csvreader_data.csv"
  CsvFileArgs    = "inputdir"
  HasHeader      = False
  SkipLines      = 0
  IgnoreComments = False
  Delimiter      = ","
  Identity = "%1_%2"
  IdentityArgs = ( SourceProductId,  StereoSource )
EndObject
      

Return to List of Strategy Types

CSVWriter Strategy Description

Resource geometries can be written to the CSV file as keywords using the GisGeometryKey parameter in the format specified by the GisType parameter. The two supported formats are well-known binary (WKB) and well-known text (WKT).

Note: Writing a geometry to a CSV file in WKT format while using the default comma delimiter may result in the CSV file being unreadable by CsvReader. Change the delimiter to a different character if using WKT format.

Rows in the CSV file will be delimited by a new line. Columns are delimited by the character specified by the Delimiter parameter, which defaults to a comma.

The following is an example of a CsvWriter strategy:

Object = Strategy
  Type     = CsvWriter
  Name     = mdismla
  CSVFile  = "mdis_mla_ridelong.lis"
  Mode     = Create
  Header   = True
  Keywords = (YearDoy, SourceProductId, StartTime, EtStartTime, 
              ExposureDuration, CenterLongitude, CenterLatitude,
              PixelResolution, MeanGroundResolution,
              IncidenceAngle, EmissionAngle, PhaseAngle, 
              SubSolarGroundAzimuth, SubSpacecraftGroundAzimuth, 
              ParallaxX, ParallaxY, ShadowX, ShadowY)
  Delimiter = ","
  DefaultValue = "NULL"
EndObject
      

Return to List of Strategy Types

DatabaseReader Strategy Description

The access details of the database can be stored in a configuration file in a profile of the Database object. The configuration file containing the Database object is specified by the DbConfig parameter. The database profile is specified by the DbProfile parameter. A special parameter called DbFile is provided for SQLite files. This shortened specification for SQLite files eliminates the need for creating an elaborate Database configuration for file-based databases.

Generally, any valid SQL query can be used that returns rows from the results of the query issued to any database. Any database that is supported by the Qt system configuration shipped with each ISIS release can accept SQL queries. This typically includes PostgreSQL, MySQL and SQLite. When specifying the query in the Query, some special quoting may be needed in order to comply with SQL syntax. Before a query is sent to the database, all occurrences of " and ' in the query string are replaced with double quotes [ " ] and single quotes [ ' ], respectively.

The following is an example of a valid configuration file named mdis.conf with a DatabaseReader strategy within an IsisMiner object and a Database object:

Object = IsisMiner
  Object = Strategy
    Name = MDISSelection
    Type = DatabaseReader
   
    DbConfig = mdis.conf
    DbProfile = MDIS
   
    Query = "select  i.SourceProductId, i.YearDoy, 
              g.StartTime, i.EtStartTime, i.ExposureDuration,
              g.CenterLongitude, g.CenterLatitude, i.ObservationType,
              g.PixelResolution, g.MeanGroundResolution,
              g.IncidenceAngle, g.EmissionAngle, g.PhaseAngle, 
              g.SubSolarGroundAzimuth, g.SubSpacecraftGroundAzimuth, 
              g.ParallaxX, g.ParallaxY, g.ShadowX, g.ShadowY,
              p.GisFootprint
                from Polygon p
                INNER JOIN Image i
                ON p.SourceProductId=i.SourceProductId 
                INNER JOIN Geometry g
                ON p.SourceProductId=g.SourceProductId 
                INNER JOIN Statistics s
                ON p.SourceProductId=s.SourceProductId
                where (i.Center=747.7 or i.Center=748.7) 
                 order by g.PixelResolution"
    Target = Resource
    Identity = "%1"
    IdentityArgs = "SourceProductId"
    GisGeometryRef = GisFootprint
    GisType        = WKB
    RemoveGisKeywordAfterImport = True
  EndObject
EndObject

Object = Database
  DefaultProfile = "MDIS"
  Group = Profile
      Name = "MDIS"
      Dbname = "Messenger_DOY_2011095_TO_2013011_gis.db"
      Type = SQLite
      Description = "MDIS Sqlite database"
  EndGroup
EndObject
      

Return to List of Strategy Types

Filter Strategy Description

All comparisons are case-insensitive. The keyword used to filter Resources is specified in the Keyword key in the Filter Strategy object definition found in the Strategy object. Only the first value in the named Resource Keyword will be checked unless the [optional] CheckAll keyword is set to True.

Values contained in the Include keyword in the Strategy object will be compared with values in the Resource key specified in the Keyword key. If a value exists in both keywords, then the Resource is retained (as active). Values contained in the Exclude keyword in the Strategy object that exist in the Resource key will deactivate a Resource.

Both Include and Exclude keywords may exist in the Filter Strategy object. In this case, values are first checked in the Include list then the Exclude list. Whichever occurs first, that action is taken for the Resource. If none of the keyword values exist in either Include or Exclude keywords, then disposition of the Resource depends on the lists. If the Include list is not empty and the Exclude list is empty, the Resource is deactivated. Otherwise, the Resource is retained as active.

If needed, users can provide a regular expression string that will use Qt's QRegExp class implementation to apply to keyword values. If given, this test is applied after the Include and before the Exclude test to determine if the value satisfies the regular expression. If after all the tests are applied and this string was supplied, the Resource is discarded since it did not succeed in its test.

If neither Include nor Exclude keywords exist, then no action is taken for the Resources.

The following is an example of a Filter strategy:

Object = Strategy
  Name = TossIgnores
  Type = Filter
  Description = "Deactivate all ControlMeasures that are ignored" 
  Keyword = Ignore 
  Exclude = True 
EndObject
      

Return to List of Strategy Types

GISIntersect Strategy Description

The following is an example of a GisIntersect strategy:

Object = Strategy
  Name = DeactiveNonintersecting
  Type = GisIntersect
  Description = "Define bounding box (lower left, upper right (lon lat)) 
                       around Degas"
  GisType = "WKT"
  GisGeometry =  "LINESTRING(225.00 30.00, 240.00 45.00)"
  BoundingBox = True
  ComputeRatio = True
EndObject
      

Return to List of Strategy Types

GisOverlap Strategy Description

Parameters to this algorithm can define the ratio (or percentage) of common overlap for the two GIS geometries. As implied, all Resources must have GIS geometries in order to participate in the overlap determination.

Overlap candidates are determined from other Resources for every incoming active Resource and a list of Resources that are determined from another isisminer Strategy configuration. The configuration can either be specified directly in the GisOverlap Strategy object (an embedded IsisMiner object called a CandidateMiner) or come from an external file called CandidateConfigFile. Candidate overlap Resources can be either part of the incoming Resource list (active and inactive) or from a completely different source.

Once overlaps are determined for a Resource, the candidate and its source are merged/combined into a single Resource and added to an Asset of the primary Resource. Keywords from both Resources are combined into a single Resource by renaming each keyword from both Resources and propagating them to the merged Asset Resource. An A is appended to the source Resource keywords and a B is appended to the candidate Resource keywords before they are merged to distinguish keywords. The suffix of keyword sources can be specified by the user using the MergeSuffixA and MergeSuffixB keywords, respectively. The identity of the merged Resource is a combination of the two Resource identities separated by an underscore character.

Once the list of overlapping Resources is determined, the user can opt to immediately process the overlaps. If a second isisminer configuration is provided called OverlapMiner or OverlapConfigFile, the list of overlapping Resources that are contained combined Resources and geometries is run though the Strategy list in the OverlapMiner configuration. This option is useful to eliminate the need for intermediate storage which could cause memory issues.

The following is an example using the GisOverlap strategy. In this example we are reading image data from an SQLite database containing WAC and NAC MESSENGER data created from caminfo runs. It determines all the images overlapping a square geometry around Debussy crater.

Object = IsisMiner

  Object = Strategy
  Name = MDISSelection
    Type = DatabaseReader

    DbConfig = wac_nac_overlaps.conf
    DbProfile = MDIS
    Mode = Select

    #  Select strings that need double quotes must use " since they cannot
    #  be stored in keyword values.  These strings will be prescanned before
    #  issuing to the DB and they will be properly replaced.
    Query = "select  i.SourceProductId, i.YearDoy, i.File,
              g.StartTime, i.EtStartTime, i.ExposureDuration, i.ObservationId,
              g.CenterLongitude, g.CenterLatitude, i.ObservationType,
              g.PixelResolution, g.MeanGroundResolution,
              g.IncidenceAngle, g.EmissionAngle, g.PhaseAngle, 
              g.SubSolarGroundAzimuth, g.SubSpacecraftGroundAzimuth, 
              ParallaxX, ParallaxY, ShadowX, ShadowY,
              p.GisFootprint
                from Polygon p
                INNER JOIN Image i
                ON p.SourceProductId=i.SourceProductId 
                INNER JOIN Geometry g
                ON p.SourceProductId=g.SourceProductId 
                INNER JOIN Statistics s
                ON p.SourceProductId=s.SourceProductId"

  # Resource targets are for reading multiple rows from the DB.  The fields
  # from each row are created as Resources.
    Target = Resource
    Identity = "%1"
    IdentityArgs = "SourceProductId"

  #  Can specify a GIS keyword that will be converted on input
    GisGeometryRef = GisFootprint
    GisType        = WKB
    RemoveGisKeywordAfterImport = True
  EndObject

 Object = Strategy
    Type = GisIntersect
    Name = SelectDebussy
    GisType = WKT 
    #  Define bounding box (lower left, upper right (lon lat)) around Debussy
    #  crater
    GisGeometry = "LINESTRING(11.56 -34.91, 13.83 -33.03)"
    BoundingBox = True
    ComputeRatio = True
    RatioRef = DebussyOverlapRatio
    ShowProgress = True
  EndObject

# Since there is no CandidateMiner specified so the current active list is 
#  processed, recursively. Meaning each Resource is checked for overlap
#  of all other active Strategies
  Object = Strategy
    Type = GisOverlap
    Name = DebussyOverlaps
    Description = "Check for image overlaps near Debussy"
    ShowProgress = True
    OverlapMinimum = 0.5
    OverlapMerge = Intersection
  EndObject

  Object = Strategy
    Type = AssetSidebar
    Name = WriteOverlapFiles
    Description = "Write CSV files of overlap data"
    Asset = GisOverlap

    Object = IsisMiner
      Object = Strategy
        Type = CsvWriter
        Name = WriteOverlaps
        Description = "Write individual files with overlap data"

        SkipEmptyLists = True
        CsvFile = "overlaps/%1.csv"
        CsvFileArgs = "Identity"
        Keywords = ("SourceProductIdA", "SourceProductIdB",
                    "OverlapRatioA", "OverlapRatioB",
                    "YearDoyA", "YearDoyB", 
                    "CenterLongitudeA", "CenterLatitudeA",
                    "CenterLongitudeB", "CenterLatitudeB",
                    "GisOverlapCentroidX", "GisOverlapCentroidY",
                    "PixelResolutionA", "PixelResolutionB", 
                    "EmissionAngleA", "EmissionAngleB", "IncidenceAngleA",
                    "IncidenceAngleB", "PhaseAngleA", "PhaseAngleB")
      EndObject
    EndObject

  EndObject

EndObject


Object = Database
  DefaultProfile = "MDIS"
  Group = Profile
    Name = "MDIS"
    Dbname = "Messenger_Monochrome_DOY_2011095_TO_2014304_gis.db"
    Type = SQLite
    Description = "MDIS Sqlite database"
  EndGroup
EndObject

End
      

Return to List of Strategy Types

GISUnion Strategy Description

If the union geometry is present and good, an entry will be added to the Resource with the parameter RatioRef as the key and the overlap ratio as the value. The double parameters OverlapMinimum and OverlapMaximum indicate the acceptable range for the overlap ratio. Geometries with overlap ratios which do not fit the constraints will be discarded.

The following is an example of a GisUnion strategy:

Object = Strategy
  Name = UnionHighOverlap
  Type = GisUnion
  Description = "Union remaining geometries which have a high overlap ratio."
  OverlapMinimum = 0.65
  OverlapMaximum = 1.0
  RatioRef = "HighOverlapRatio"
EndObject
      

Return to List of Strategy Types

IsNumeric Strategy Description

A keyword value is considered numeric if it can be converted to a floating point number. If the keyword does not exist in the Resource, then its value is not considered numeric.

The following is an example of an IsNumeric strategy:

Object = Strategy
  Name = RemoveNotApplicable
  Type = IsNumeric
  Description = "Deactivate all responses which did not respond to rating."
  Keyword = Rating
EndObject
      

Return to List of Strategy Types

Limit Strategy Description

If keyword values are within the limits, the Resource disposition is set to activated. If the keyword is not within the limits provided, the Resource disposition is set to deactivated.

This Strategy uses the Calculator strategy. Any equation constructed that is valid for the Calculator strategy is valid for Limit. This includes the use of Equations group to use more than one equation. The difference here is the value of the last equation is used to evaluate a true (!0) or false (0) limit for the Resource. Also note, that a side effect of using the Equations group is that any new keyword created by an equation remains in the Resource.

Following are few examples of Limit strategies:

Object = Strategy 
  Type = Limit
  Name = EmissionAngle
  Equation = "((emissionangle > 30) && (emissionangle < 80))"
EndObject
      

Object = Strategy
  # This algorithm will select only images where the center latitude
  # and longitudes are within the defined region provided in the GLOBALS
  # parameter file.
  Name = SelectRegion
  Type = Limit
  Description = "Find all images with center image geometry in the
                 defined region"

  # This set of equations will result in 1 if test are good, meaning the
  # clat/clon is in the defined region. If any test fails, InRegion = 0
  # (false) and the Resource is disabled. Note this also defines the GIS
  #  bounding box.
  Group = Equations
    InRegion = 1
    InRegion = "InRegion * (CenterLongitude >  RegionLowerLeftLongitude)"
    InRegion = "InRegion * (CenterLatitude  >  RegionLowerLeftLatitude)"
    InRegion = "InRegion * (CenterLongitude <= RegionUpperRightLongitude)"
    InRegion = "InRegion * (CenterLatitude  <= RegionUpperRightLatitude)"
  EndGroup
EndObject
      

Return to List of Strategy Types

NumericalSort Strategy Description

SortKey must have a numerical value for each Resource that will be sorted. See the IsNumeric strategy.

Order may be specified as either Ascending or Descending. If not provided, the Resources will be sorted in Ascending order.

The following is an example of a NumericalSort strategy:

Object = Strategy
  Name = RankSort
  Type = NumericalSort
  Description = "Sorts the Resources in ascending order by Rank" 
  SortKey = Rank
  Order = Ascending
EndObject
      

Return to List of Strategy Types

PDS Table Creator Strategy Description

Records are delimited by a new line. Fields are delimited as specified by the Delimiter parameter, which defaults to a comma.

The following is an example of a PdsTableCreator strategy:

Object = Strategy
  Name = TestAppendTableWriter
  Type = PdsTableCreator
  PdsFormatFile = "%1/TestGeneralFormat.txt"
  PdsFormatFileArgs = "inputdir"
  PdsTableFile = "%1/TestAppend.TAB"
  PdsTableFileArgs = "outputdir"
  Mode = Append
  Delimiter = ","
  DefaultValue = "NULL"
EndObject
      

Return to List of Strategy Types

PDS Table Format Strategy Description

Records are delimited by a new line. Fields are delimited as specified by the Delimiter parameter, which defaults to a comma.

Example: POINT_ID_BYTES = 32

The following is an example of a PdsTableFormat strategy:

Object = Strategy 
  Name = PdsTableFormat
  Type = PdsTableFormat
  PdsFormatFile = "POINTCLOUDTAB.FTM"
  PdsFormatFileArgs = "outputdir"
  Delimiter = ","
  Column = ("POINT_ID", "STATUS") 
  DataType = ("CHARACTER", "CHARACTER")
  Unit = ("NONE", "NONE") 
  DataDescription = ("Unique point identifier.", "Status of point") 
  POINT_ID_BYTES = 32 
  STATUS_BYTES= 32
EndObject 
      

Return to List of Strategy Types

PDS Table Reader Strategy Description

If the Identity keyword is not provided or its value is an empty string, each Resource created will be named the row number. Rows are read (starting at 0) from the PDS table data in PdsTableFIle.

The following is an example of an PdsTableReader strategy:

Object = Strategy
  Name             = PdsTableReaderTestDefault
  Type             = PdsTableReader
  PdslabelFile     = "%1/index.lbl"
  PdslabelFileArgs = "inputdir"
  TableName        = "INDEX_TABLE"
  Identity         = "%1"
  IdentityArgs     = File_Name
  UseFormattedName = False
  Description      = "Test the default functionality of PdsTableReader"
EndObject

      

Return to List of Strategy Types

PVL Reader Strategy Description

If the Identity keyword is not provided or its value is an empty string, each Resource created will be named as PvlN. N is the position number (starting at 0) of the PVL file contained in FromList. The basename, Pvl, can be changed by setting PvlBaseName.

For example, if FromList contains PVL Files ~/somepvl.pvl and ~/someotherpvl.pvl, their default Identity values would be Pvl0 and Pvl1, respectively.

The Includes keyword operates on PVL Objects and Groups (containers). When provided by itself, the Includes keyword will exclusively include the keywords belonging to the containers specified. Since Groups must be contained in Objects, you cannot include a Group without also including its containing Object. Similarly, you must include the Root Object since all PVL files have a Root Object container. For example:

Object = Outer
    Id = 001
    Group = ExtraInfo
        Name = outside
        Time = today
    EndGroup
    Group = OtherInfo
        Info = "No information present"
    EndGroup
    Group = Important
        Level = High
    EndGroup
    Object = Inner
        Data = "Some test data"
    EndObject
EndObject
      

Includes = (Root, Outer, ExtraInfo)

The Excludes keyword operates on PVL Objects and Groups (containers). When provided by itself, the Excludes keyword will exclude the keywords belonging to the containers specified, while all other containers and keywords will be included.

When both Includes and Excludes are provided, an Object and all of its keywords and Groups will be included except for any excluded Groups (Inclusions are applied to Objects while exclusions are applied to Groups).

Referring to the Includes example, to include the Outer Object except for its OtherInfo Group, you would specify the following:

Includes = (Root, Outer)
Excludes = OtherInfo
      

The KeyListFile keyword operates on PVL Keywords and expects a file containing PVL Keywords. If provided, only the keywords in the file will be included in the created Resource. Note that these keywords will only be included if their containers are included or not excluded.

The following is an example of an PvlReader strategy:

Object = Strategy
  Name          = PvlReader
  Type          = PvlReader
  FromList      = "%1/testpvl.pvl"
  FromListArgs  = "inputdir"
  Identity      = "%1"
  IdentityArgs  = Id
  KeyListFile   = "%1/keystoinclude.lis"
  KeyListFileArgs = "inputdir"
  Description   = "A PvlReader Strategy that includes the keywords contained in keystoinclude.lis
                             when creating the Resource from testpvl.pvl"
EndObject
      

Return to List of Strategy Types

Resource Manager Strategy Description

For DeleteAsset and HasAsset, the Asset is specified in the Operations Keyword as
Operations = HasAsset::AssetName or Operations = DeleteAsset::AssetName.

The following is an example of a ResourceManager strategy:

Object = Strategy
  Name = SwitchActivated
  Type = ResourceManager
  Description = "Swap activated (Included) and deactivated (Excluded) entries in a Resource."
  Operations = ToggleDiscard
EndObject
      

A second example with multiple management operations

# Clear the entire Resource list (useful in Sidebar Strategies)
Object = Strategy
  Name = ClearResources
  Type = ResourceManager
  Description = "Delete all Resources"
  Operations = ( ResetDiscard, ToggleDiscard, DeleteDiscard )
EndObject
        

Return to List of Strategy Types

Run Command Strategy Description

Commands can be formulated using the keyword argument substitution options available for using global parameter pools for replacing arguments in commands with Resource keyword values. One keyword, CommandArgs, is available to list all the Resource and global pool variables that are to be used in replacement. All commands can use a %1 to %n where n is the total number of keywords list in the CommandArgs keyword. This one list of keywords is used to substitute for all commands.

The following is an example of a RunCommand strategy. The " text is required and should be included in the configuration of your command. The " and ' are substituted prior to issuing the command to the shell for execution. This is required because of PVL syntax rules.

Object = Strategy
  Name         = RunManyCommands
  Type         = RunCommand
  Description  = "Tests running many command"

  CommandArgs = (outputdir, SourceProductId, StartTime)
  # Ensure the output file exists
  Group = PreCommands
    Command0 = "touch %1/runmanycommands.txt"
  EndGroup

  # Append the Resource name and start time
  Group = Commands
    Command1 = "echo "Resource name %2 and date %3." >> %1/runmanycommands.txt"
  EndGroup
EndObject

      

Return to List of Strategy Types

Sidebar Strategy Description

Sidebar strategies are useful for running Strategies that have the potential to throw out Resources or alter a current state that may be undesirable. For these cases, you can use a Sidebar that saves all the Resources, activated and deactivated, their current states and preserves any operations that may alter or discard Resources. It can also be used to process an entirely new set of Resources while retaining the current state of existing Resources.

Sidebar strategies can be inserted anywhere in the sequence of Strategies but are best placed strategically at critical points in the processing sequence where preservation of the current active Resource state is needed.

The following is an example of a Sidebar strategy:

Object = Strategy
  Name = SetUpResources
  Type = SideBar
  SavePoint = True
  SaveDelete = False
  StrategyConfigFile = "%1/setup.conf"
  StrategyConfigFileArgs = "inputdir"
EndObject
      

Return to List of Strategy Types

StereoPair Strategy Description

Parameters for this algorithm define two levels of stereo constraints that are described in [Becker 2015]. The first level of constraints are typically for constraining photometric (incidence, emission, phase) and geometric angles. The constraints for these limits are located in the ImageStrength group. Each keyword there should exist in each Resource or have a keyword map specified in the KeywordMap group. The second level constraints contained in the StereoStrength group are stereo limits and are applied to values computed from the stereo algorithm with contributions from both pair Resources. The format for keywords in both of these groups are four element vectors but only 3 values are required. For example, to limit the emission angle between 0 and 50 degrees, the form of the keyword in the ImageStrength group is Emission = (0.0, 50, 15.0 [, 1.0]). The first value in this keyword is the minimum value, the second is the maximum value and the third value is the value for emission that provides the best stereo conditions. The last optional value is a multiplier of the computed strength, or rank, for each keyword limit that can be used to weight the computed strength of the limit beyond the 0 - 1 range.

The rank of each keyword is computed using the following scheme based upon this form of limit keywords in both ImageStrength and StereoStrength: LimitKey = (t0, t1, t2, t3) where all t0 through t3 values are numerical numbers and t0 is the minimum, t1 is maximum, t2 is the best value in the range and t3 is a weighting factor. The formula to compute the rank of each keyword is as follows:

range = max(abs(t2-t0), abs(t1-t2))
rank = 1 - abs(t2-value)/range
rank *= t3          
      

#  Now specify how the stereo quality is determined.
Equation = "imagestrengthrank * 1.0 + stereostrengthrank * 1.0"
Result = StereoPairRank
      

If data is provided by caminfo, then the values required by this Strategy are provided at the center image pixel coordinate. This may not provide accurate results, particularly for the StereoStrength group of keywords, because the values are likely not of the latitude/longitude within the overlapping region of the images. However, the computation of the StereoAngle could be more accurate if you were to take advantage of GisOverlap OverlapMerge options. The centroid (longitude/latitude) values for all of these options is computed and stored in new keywords GisOverlapCentroidX (longitude) and GisOverlapCentroidY (latitude) making this coordinate the same point in both stereo image pairs. By setting a parameter in the KeywordMap group as CenterLongitude = GisOverlapCentroidX and CenterLatitude = GisOverlapCentroidY the StereoAngle is computed from a common surface point. If desired, the user can specify whether to use the stereo angle in the computation of the VerticalPrecision stereo keyword rather than the ParallaxHeightRatio. Take note the equation to compute StereoPairRank can be changed as desired or weights can be provided in the fourth limit keyword value to emphasize or de-emphasize any limit.

Example: Incidence = (  30.0,  70.0,  50.0 )

Example: Emission = (   0.0,  55.0,  15.0 )

Example: Phase = ( 15.0, 120.00, 30.0 )

Example: Resolution  = ( 275.0, 500.0, 300.0 )

Example: ParallaxHeightRatio = ( 0.1, 1.0, 0.4 )

Example: ShadowTipDistance = ( 0.0, 1.0, 0.5 )

Example: ResolutionRatio  = (  1.0,  2.5,  1.0 )

Example: DeltaSolarAzimuth    = (  0.0, 90.0,  45.0 )

Example: DeltaSpacecraftAzimuth  = (  0.0, 65.0,  25.0 )

Example:  StereoAngle  = ( 10.0,  65.0,  25.0 )

#  Now specify how the stereo quality is determined. Based upon
#  ImageStrength and StereoStrength, the maximum (best) rank is 8.0.
Equation = "ImageStrengthRank * 1.0 + StereoStrengthRank * 1.0"
Result = StereoPairRank
          

The following is an example using the StereoPair strategy. In this example we are reading image data from an SQLite database containing WAC and NAC MESSENGER data created from caminfo runs. It determines stereo from overlapping images in the vicinity of Degas crater on Mercury.

Object = IsisMiner

  Name = DegasStereoPair
  RequiredParameters = ( inputdir, outputdir )

  Object = Strategy
    Name        = ReadDegasPerimeter
    Type        = CsvReader
    Description = "Read Degas data"

    CsvFile        = "%1/degaus_stereo_perimeter.csv"
    CsvFileArgs    = "inputdir"

     #Ignore the first several lines including the header.
    HasHeader      = True

    Identity = "%1"
    IdentityArgs = "SourceProductId"

  #  Can specify a GIS keyword that will be converted on input
    GisGeometryKey = GisFootprint
    GisType        = WKB
    RemoveGisKeywordAfterImport = False
  EndObject

  Object = Strategy
    Type = StereoPair
    Name = DegasStereoPairs
    Description = "Find all stereo pairs in the Degas crater region"
    OverlapMerge = None
    ShowProgress = True
    #Debug = True

    PixelPrecisionMatch  = 0.2
    Asset = StereoPairs
`
    #  Now specify how the stereo quality is determined.
    Equation = "imagestrengthrank * 1.0 + stereostrengthrank * 1.0"
    Result = StereoPairRank

    #  First level contraints/strengths
    Group = ImageStrength
      Incidence       = (  30.0,  65.0,  50.0 )
      Emission        = (   0.0,  55.0,  15.0 )
      Phase           = (  15.0, 120.00, 30.0 )
      Resolution      = ( 25.0, 5000.0, 200.0 )
    EndGroup

    #  Second level constraints/strengths
    Group = StereoStrength
      ResolutionRatio      = (  1.0,  2.5,  1.0 )
      ParallaxHeightRatio  = (  0.1,  1,    0.4 )
      ShadowTipDistance    = (  0.0,  2.58, 0.0 )
      OverlapRatio         = (  0.1,  1.0,  0.5 )
    EndGroup

    #  This shows the keyword mappings to values required for stereo
    #  matching and constraint values
    Group = KeywordMap
      Resolution = PixelResolution
      Emission   = EmissionAngle
      Incidence  = IncidenceAngle
      Phase      = PhaseAngle
    EndGroup

    #  Select candidates from input list.  Matches over itself
    Object = CandidateMiner
      Object = Strategy
        Name = PairCandidates
        Type = ResourceManager
        Description = "Choose all Resources to determine stereo"
        Operations = ResetDiscard
      EndObject
    EndObject
  EndObject

  Object = Strategy
    Type = AssetSidebar
    Name = WriteStereoFiles
    Description = "Process all stereo pair Assets"
    Asset = StereoPairs
      
    Object = IsisMiner

      Object = Strategy
        Name = SortStereoPairs
        Type = NumericalSort
        Description = "Sort stereo pairs by StereoPairRank"
        SortKey = StereoPairRank
        Order = Descending
      EndObject

      Object = Strategy
        Type = CsvWriter
        Name = WritePairs
        Description = "Write individual files with stereo pair data"

        SkipEmptyLists = True
        CsvFile = "%1/%2.csv"
        CsvFileArgs = ( "outputdir", "Identity" )
        Keywords = ("SourceProductIdA", "SourceProductIdB", "StereoPairRank",
                    "ImageStrengthRank", "StereoStrengthRank",
                    "ParallaxHeightRatio", "ShadowTipDistance", "ResolutionRatio", 
                    "DeltaSolarAzimuth", "DeltaSpacecraftAzimuth", 
                    "VerticalPrecision", "StereoAngle",
                    "ImageStrengthRankA", "ImageStrengthRankB",
                    "IncidenceRankA", "IncidenceRankB",
                    "EmissionRankA", "EmissionRankB",
                    "PhaseRankA", "PhaseRankB",
                    "OverlapRatioA", "OverlapRatioB",
                    "YearDoyA", "YearDoyB", 
                    "CenterLongitudeA", "CenterLatitudeA",
                    "CenterLongitudeB", "CenterLatitudeB",
                    "PixelResolutionA", "PixelResolutionB", 
                    "EmissionAngleA", "EmissionAngleB",
                    "IncidenceAngleA", "IncidenceAngleB",
                    "PhaseAngleA", "PhaseAngleB")
      EndObject
    EndObject
  EndObject
EndObject
End

      

Return to List of Strategy Types

Description

This file contains a series of Strategy objects enclosed in an object named IsisMiner. The main documentation contains many different examples of the format of this file.

Description

All keywords contained in this file are loaded in the global variable pool. Unique keywords that do not exist in Resources can be used as keyword value substitutions in your Strategies.

Description

This parameter allows the user to add global parameters to each Strategy so that they can be used in translation of keyword ARG values. This parameter has the form:

PARAMETERS = "key1:value1[@key2:value2...]"

This example has been used to process MESSENGER MDIS global image dataset. The purpose of this isisminer run is to

1. identify all images whose center pixel longitude/latitude coordinate are contained within a defined 10x10 degree map section,
2. from that list, find all images that overlap each satisfying a specified overlap ratio, and
3. determine all images that satisfy stereo conditions for each image in the map section.

File naming conventions used for these section definition files are EquiYYYHXXXED10 where

• YYY is the center latitude of the section,
• H is the hemisphere (i.e. H = N for northern or H = S for southern),
• XXX is the center longitude of the section,
• E indicates positive east longitude, and
• D10 indicates the spacing is 10 degrees in both latitude and longitude.

The section in this example includes Degas crater located at 37.08 degrees latitude and 232.75 positive east longitude. This falls in the section defined in the file Equi035N235ED10.def. Here is the contents of this file:

# Define sub region in 10 degree increments below the NP
# Define lower left and upper right coordinates
Region = "Equi035N235ED10"

RegionGisGeometry = "LINESTRING(230 30, 240 40)"
RegionBoundingBox = True

RegionCenterLongitude     = 235
RegionCenterLatitude      = 35

RegionLowerLeftLongitude  = 230
RegionLowerLeftLatitude   = 30

RegionUpperLeftLongitude  = 230
RegionUpperLeftLatitude   = 40

RegionUpperRightLongitude = 240
RegionUpperRightLatitude  = 40

RegionLowerRightLongitude = 240
RegionLowerRightLatitude  = 30

End
          

To process the global image dataset requires treating data at the poles slightly different when identifying stereo. The incidence angles are consistently high, up to 90 degrees. This requires a slight change to the stereo configuration so two files are required to process the data - one for equatorial data between 65N and 65S degrees and one for polar from 65N/S to 90 N/S degrees. The isisminer configuration file must consider this when processing stereo and it is driven by the map section input file as to which one is selected. The easiest way to do this is to provide the name of the stereo configuration file as a PARAMETER to isisminer. The main isisminer CONFIG file is named mdis_overlaps_and_stereo.conf and its contents with comments is shown in the mdis_overlaps_and_stereo.conf file below.

This configuration file uses the GISOverlap strategy to identify all images that overlap one another and creates a CSV file of those images and a lot of relevant data for computing stereo. The stereo image pairings/sets are determined using a special processing feature of GisOverlaps. It has a provision for immediately processing the result of the overlap determination for each image. Just so happens that the output CSV file contains all the information needed to compute stereo. As each image overlap set is determined and written to the output CSV file, GisOverlap runs the stereo configuration file on that CSV file using the StereoPair strategy. Its configuration is contained in the file mdis_global_stereo_equatorial.conf and is provided in the command line to isisminer in PARAMETERS using the StereoConfig parameter. The contents of mdis_global_stereo_equatorial.conf is shown below.

Finally, there is a complete configuration that has command line control depending upon which latitude region is being processed and input/output directories. To process the entire global dataset of MDIS images collected by MESSENGER, 267,769 of them, will require vast computing Resources. It is highly recommended that a processing cluster be used if available. Below is the command line used to process this map section.

The results summarized here show the output of the command and a summary of the results below.

Creating Strategies...
Finished creating 24 strategies...

Running DatabaseReader::MDISSelectRegionOct2014 (TimeIn:  02:46:51.626)
Description: Strategy::MDISSelectRegionOct2014 is running a DatabaseReader algorithm.
248586 of 0 processed in DatabaseReader::MDISSelectRegionOct2014 (TimeOut: 03:18:36.966)
ElapsedTime(s): 1905

Running DatabaseReader::MDISSelectRegionMay2015 (TimeIn:  03:18:36.966)
Description: Strategy::MDISSelectRegionMay2015 is running a DatabaseReader algorithm.
19183 of 0 processed in DatabaseReader::MDISSelectRegionMay2015 (TimeOut: 03:20:59.204)
ElapsedTime(s): 142

Running IsNumeric::FindBadOnes (TimeIn:  03:20:59.204)
Description: Identify images with invalid center pixel geometry
267474 of 267769 processed in IsNumeric::FindBadOnes (TimeOut: 03:21:00.209)
ElapsedTime(s): 0

Running ResourceManager::DeleteBadOnes (TimeIn:  03:21:00.209)
Description: Delete Resources with bad values in required keywords
295 of 0 processed in ResourceManager::DeleteBadOnes (TimeOut: 03:21:00.247)
ElapsedTime(s): 0

Running Limit::SelectRegion (TimeIn:  03:21:00.247)
Description: Find all MDIS images with center image geometry in the defined region
557 of 267474 processed in Limit::SelectRegion (TimeOut: 03:21:20.489)
ElapsedTime(s): 20

Running RunCommand::MakeOutputDirectory (TimeIn:  03:21:20.489)
Description: Ensure output directories exists
Running Mkdir1 PRE command: mkdir -p /scratch/kbecker/MESSENGER/Region10D/Global/Images
Running Mkdir2 PRE command: mkdir -p /scratch/kbecker/MESSENGER/Region10D/Global/Overlaps/Equi035N235ED10
2 of 0 processed in RunCommand::MakeOutputDirectory (TimeOut: 03:21:20.976)
ElapsedTime(s): 0

Running CsvWriter::WriteRegionImages (TimeIn:  03:21:20.976)
Description: Write images that are in the defined region to a file
557 of 557 processed in CsvWriter::WriteRegionImages (TimeOut: 03:21:21.398)
ElapsedTime(s): 0

Running GisOverlap::ImageOverlap (TimeIn:  03:21:21.398)
Description: Find all overlaps images in specfied region
557 of 557 processed in GisOverlap::ImageOverlap (TimeOut: 04:42:28.181)
ElapsedTime(s): 4866

Session complete in 01:55:36.555 of elapsed time