Home

User Documentation

Getting Started
Learn More
Explore in Detail
Get Inspired

Contributor Documentation

Getting Started
Learn More
Explore in Detail
Get Inspired

Quick Links

Software Manual
AstroDiscuss
GitHub
API Reference

Documentation Versions


ISIS 2

Documentation
Tutorials
Technical Documents
USGS

ISIS Application Documentation


isisminer

Printer Friendly View | TOC | Home

Run a series of algorithms (Strategies) that perform various operations on input sources (Resources)

Overview Parameters Example 1

Description

The isisminer application assists in the identification, manipulation, and output of data from a variety of data sources. One main source of input is caminfo. Processing in the isisminer program is controlled by the contents of a PVL-based, object-structured, configuration file provided in the CONFIG parameter. There are two main PVL objects used in the configuration of an isisminer program run: the IsisMiner object and the Strategy object.

Strategy Algorithms

The IsisMiner object contains a series of Strategy objects. Strategy objects are associated with a defined algorithm that performs some operation on data called a Resource. Strategy objects contain parameters that are needed by the algorithm to process Resource data. The order of Strategy objects is important as the algorithms are executed in the order they occur in the IsisMiner object. Each Strategy object must be associated with a known Strategy Type. Other objects may exist anywhere in the CONFIG file but are ignored when parsing Strategy objects unless they are part of the algorithm. One common use for other objects in the CONFIG file is to include database profiles that are used to provide access information for the DatabaseReader strategy (see below).

Data Resources

Strategy algorithms operate on a Resource. A Resource is a data container that stores a set of PVL keyword/values that can have data storage extensions (called Assets and GisGeometries). These Assets can contain anything created by Strategy objects for further analysis or processing and could be unique to a set of Strategies. Resources have three major components: keyword/values, GIS geometries and Assets. Most of the data manipulations occur on keyword/value elements. A Resource may contain a GIS geometry that is used in algorithms that work on geometric regions using GIS operators (GIS Intersect, GIS Overlap, GIS Union, StereoPair ). Currently, GIS geometries must be in well-known-text (WKT) or well-known-binary (WKB) as conversions are restricted to text representations of geometries (format requirements are the same as functions like ST_GeomFromWKT and ST_GeomFromWKB, respectively). Resources may also contain Assets. Assets are typically another list of Resources created from Strategies that are attached to a (parent) Resource. If Assets are represented as Resource lists, they can be processed by almost all Strategy algorithms using the AssetSidebar strategy (see below).

Resources can come from many sources. One of the most common input sources are databases. SQL queries can be specified directly in the DatabaseReader strategy. Each row returned when the query is executed is converted into a Resource. Other data sources are comma separated values (CSV) files (using the CsvReader strategy) where each row is a Resource and each column is a keyword/value. The keyword name in the case of a CSV file is the name of the column as specified in the header. Users can also read in ISIS control networks and process each Control Measure as a Resource. And, of course, ISIS cube files and other PVL file formats are also input sources (see PvlReader strategy). Strategies are designed to operate on a single Resource. This allows for a very narrow scope of design considerations to make coding Strategies quite easy.

Control Options using Keyword Argument Substitution

Some very clever and sophisticated Strategy configurations can be created but to make them generally useful, the user may need to change some parameters within Strategies, such as output directories or file names that can be modified at runtime without having to edit the CONFIG files directly. To allow some control options, there are two optional application parameters provided: GLOBALS and PARAMETERS.

The GLOBALS parameter can specify the name of a file that contains PVL keywords and values. The complete contents of all the keywords found in the file, regardless of Object/Group structure are loaded immediately when isisminer first runs. The keywords contained in this file can be used as substitutions in all or part of many keyword value specifications used within most Strategies (described below).

The PARAMETERS input parameter is provided so that users can create simple keyword/value pairs on the command line that can be used in substitutions of part of all keyword values used in Strategies in the same manner GLOBALS keywords are substituted. For any keyword that is contained in both the GLOBALS and PARAMETERS keyword set, keywords in PARAMETERS always take precedence. This allows users to override keywords in GLOBALS as needed during keyword lookup and replacement operations to fully determine values of keywords in real time when Strategies are executed. Since equal signs are not allowed in ISIS application parameters, keyword names and values are separated by a colon. Multiple keywords can be specified and are separated by the @ symbol. For example, PARAMETERS="inputdir:$PWD@outputdir:$PWD/Results" will create two keywords, inputdir and outputdir, with inputdir set to the full path of the current directory running the job and outputdir set to a subdirectory in the current directory called Results.

Essentially, the GLOBALS and PARAMETERS elements creates a hierarchical lookup for keyword substitutions where PARAMETERS are searched first, then GLOBALS, for argument substitution of elements within a keyword value. At times, when individual Resources drive output operations (e.g., AssetSidebars), the individual Resource keywords become available for substitution as well and are the first set of keywords searched before PARAMETERS and GLOBALS. Argument substitutions are specified using the %n symbol in keyword values. Here, the n is a number from 1 to the number of values corresponding to an associated keyword with the same name but with Args appended to it. The nth value of this keyword is itself a keyword where its value is substituted in the corresponding %n occurrence in the original keyword. For example, in the CSVReader strategy, the name of a file to read the contents from is specified in the CsvFile keyword. To allow arbitrary specification of the input file without editing the CONFIG file to change it for each use of the CONFIG file, the PARAMETERS option can be used to specify the path to a named file or the entire file name by creating a keyword called CsvFileArgs. This is the general form for creating a keyword value substitution that uses two associated keywords with the first being any keyword and the second keyword having Args appended to the first keyword which contains other keywords that will be used to substitute arguments with.

See the complete stereo example provided for isisminer that demonstrates many of the concepts discussed here and in the Strategy descriptions below. It will also provide output and describe the results of a MESSENGER MDIS stereo processing run.

Descriptions of Strategies

The following section describes all the Strategy algorithms currently available in IsisMiner. Strategies iterate through all active Resources. One of the features in isisminer that provides a lot of flexibility in the system is that Resources are never deleted unless users provide explicit instructions to do so (see ResourceManager). Each Resource has an inherent state that indicates if it is active, meaning it is available for processing, or inactive. Strategies like the Limit or Filter will evaluate values of keywords for boolean equations or strings, respectively, where true conditions keep Resource states as active, and false conditions set their states to inactive. Inactive Resources are not processed in subsequent Strategies unless explicitly reactivated (again using the ResourceManager). Strategies typically only use active Resources when normally processed. Exceptional is when a Strategy provides options to select an additional processing set of Resources (see GisOverlap, StereoPair and Sidebar). Those Resources accept a complete isisminer configuration where the object specification can be embedded within the Strategy object or a file name can be provided that contains a complete isisminer configuration (much like provided in CONFIG). This design allows users to progressively select Resources for processing and reactivate the entire list at any point in the execution of Strategies. The Strategies below provide the user with many options to help identify only data with specified/desired characteristics for evaluation and subsequent processing.

Special Strategy Keywords

There are some keywords available that can be used in every Strategy beyond their unique keywords described below. Many of these keywords provide additional processing control or options that produce information in the form of text to the standard output stream. The following tables provide descriptions of the keywords that can be used in all Strategies. Some will apply only under certain conditions which are described in the context of the descriptions. All of these keywords are optional except where indicated.

Some miscellaneous keywords are available that provide general information and/or assist users in debugging Strategies that appear to result in unexpected behavior. Here are other keywords that users can set and are available in all Strategies.

Miscellaneous Keywords

Keyword Disposition Description
Program Constant Name of the program (isisminer).
Version Constant A text string that provides the version of the application.
Revision Constant The revision number of the program as provided in the source code repository.
Runtime Constant A string that provides the time that the application was started. It is of the form YYYY-MM-DDThh:mm:ss.
CONFIG Constant Provides the value (name of the file) given by the user in the CONFIG parameter.
PARAMETERS Constant Provides the value (keyword/value) given by the user in the PARAMETERS parameter.
GLOBALS Constant Provides the value (name of the file) given by the user in the GLOBALS parameter.
ApplyToDiscarded Optional When True, all Resources regardless of active or discard state, are processed in the current Strategy. This is handy when you want all Resources to be processed and would otherwise require use of a Sidebar strategy to set states for all Resources.
ShowProgress Optional When True, a progress bar as seen in ISIS applications is created and updated as each Resource is processed in a Strategy. This keyword can be set in any Strategy definition. This is useful to ensure processing is occurring in long Strategy algorithms.
Debug Optional If set to True in any Strategy definition, information in the form of print statements are sent to standard output stream that report certain information about processing that is occurring in a Strategy. This can result in a large amount of information that developers use to determine Strategy behavior. The reported form and content will vary widely and is solely at the discretion of the developer.

The following table describes keywords that pertain to manipulation of GIS geometries. They are available to all Strategies that use GIS geometries but only apply where indicated.

GisGeometry Related Keywords

Keyword Disposition Description
GisGeometry Optional Specifies the source of the geometry. The value of this keyword varies according to the type of GIS geometry being processed. The geometry type is specified in the GisType keyword.
GisGeometryArgs Optional Allows user to provide keywords that are used as substitution values in the GisGeometry keyword. The value in GisGeometry may contain strings of the form %n where n corresponds to the array index, starting from 1, of the values in the GisGeometryArgs keyword. Each value in the GisGeometryArgs array of values is expected to name a valid keyword that can be resolved using the hierarchical parameter search.
GisGeometryKey Optional This is a shorthand method to specify a keyword in the Resource which fully contains the GisGeometry. It will override any value specified in GisGeometry and must name a valid Resource keyword containing a GIS geometry. GisGeometryArgs can be used to substitute arguments that may be contained in the value retrieved in the keyword specified by GisGeometryKey.
GisType Required Specifies the type of GIS geometry to be processed (converted from text form). Valid types are WKT, WKB, CUBE or ISISCUBE.
RemoveGisKeywordAfterImport Optional If set to True, then the keyword referenced in GisGeometryKey is deleted from the Resource after its value is retrieved for processing. This is an attempt to reduce memory if this value is no longer needed as these strings can be quite large.
GisSimplifyTolerance Optional This keyword, if present, specifies the tolerance in units of the GIS geometry (typically in units of degrees) to use when applying a Douglas-Peucker algorithm to simplify an imported geometry after conversion from text. The tolerance in degrees can be converted to pixels by using a simple formula of circumference at the equator. For example, for Mercury the radius is 2440 km. At the equator the circumference is 2 * pi * 2440 = 15,331 km. Each degree of contains 15,331/360 = 42 km. If GisSimplifyTolerance = 0.005, the tolerance is ~0.212 km (212 meters) at the equator. This tolerance was chosen because our target shape model/map is ~200 meters/pixel. In a set of three consecutive points along the boundary, the minimum distance from the middle point of the set to the line segment that connects the first and third points is computed. If this distance is less than the tolerance, then these boundary points are eliminated.
GisGeometryPointsKey Optional If provided when GisSimplifyTolerance is specified, this is the base name of a set of keywords that describe the simplified geometry. Three keywords are created from the GIS geometry simplification process. If the value of GisGeometryPointsKey = GisSimplePoints, then the keywords GisSimplePoints contains the total number of points after simplification. GisSimplePointsOriginal contains the total number of points prior to simplification and GisSimplePointsTolerance is the value of GisSimplifyTolerance. These keywords are written to each Resource from which GIS geometries are converted from text.
GisMethod Optional When applying geometry operators in Strategies, the GEOS system provides a very efficient, geometry-based, RTree search algorithm to make multiple search queries very fast. If the search is limited to one geometry operating on all active Resource geometries, then creating the RTree is unnecessary and the operation can be applied to every geometry. This will be faster than creating the RTree from all Resource geometries and then applying the RTree search for a single geometry. Set the value of this keyword to direct if you want to use direct GIS operators. Or use rtree for use of the RTree search. Defaults to direct. This value does not apply to GisOverlap or StereoPair.

Types of Strategies

AssetSidebar Strategy Description

The AssetSidebar strategy provides the ability to process Resource Assets with most other isisminer Strategies.

This Strategy can be used to add Assets to any active Resource. By providing the Operation keyword with a value of Create, the user can then process an Asset to be added to the active Resources (with the PvlReader or CsvReader strategies, for example). Note that when using create as a value for the Operation keyword, the specified Strategy in the AssetSidebar strategy will not be applied to any existing Assets contained in the active Resources. In fact, any Asset that exists in a Resource of the same name will be deleted and replaced with the results of the AssetSidebar processing. Note that some Strategies create Assets directly in the course of executing its algorithm and do not require an AssetSidebar to create them.

When creating Assets, users have several options to create the Resource list for the Asset. Using the CreateSource keyword with a value of Copy, a copy of all Resources in the input list is generated prior to executing the Strategies to process the Assets. The active/deactive status of each new Resource in this list is managed independently from the list outside of the AssetSidebar processing it. However, keywords, other Assets, and the GIS Geometry remain identical to those of the parent Resource. To create a completely independent copy of the Resources, set CreateSource to Clone. This option clones all the status and data of each Resource and creates a completely new Resource. To start from scratch and create an Asset by other means, use None. Defaults to Copy the Resource list.

This Strategy may also be used to apply a specified Strategy to any active Resource's Assets. For Operation set to Process, the specified Strategy in the AssetSidebar will be applied to the active Resources' Assets and replace the processed Asset in active Resources as new Assets. This mode is particularly useful for processing Assets created directly by other Strategies.

AssetSidebar Strategy Object Keywords

Keyword Disposition Description
Name Required User sets the name of the AssetSidebar strategy.
Type Required Identifies the AssetSidebar strategy - must be AssetSidebar or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the AssetSidebar strategy.
Asset Required Specify the name of the Asset to create or append to the active Resource.
Operation Optional Specifies how Assets will be processed (see the AssetSidebar description). Valid values are process and create. Use process if you want to process an existing list and preserve the resulting contents. In this mode, you can delete them all after processing the Resource lists that exist for each Resource (a list of Resources attached to a Resource).
CreateSource Optional

Asset Creation Options

When creating new Assets, users have options that include creating Assets from all existing Resources, activated and deactivated. To create Assets whose contents and status are preserved, this Strategy is designed to create separate management of Asset Resources that are not affected from other processes (Strategies). Incoming lists can be copied, cloned or an empty list can be chosen to start from scratch. It should be noted that the ResourceManager strategy can be used to manipulate the resulting list contents.
Parameter Description
Copy This option will create a shallow copy of all existing Resources as candidates for the newly created Asset. This option creates a copy of the Resource data but all its contents are still shared with the parent Resource it was created from. However, its active status can be managed independent of its parent. When a copy is created, its current active status is propagated to the copy.
Clone This option will make a copy of every Resource in the current list just like the Copy option but performs a deep copy of all data except other Assets. Existing Assets are all deleted when cloned. Resources that remain in this new Asset now become unique and live as long as the Asset remains in the Resource it was created for.
None Starts from scratch and generates an initial empty list of Resources for the Asset prior to executing Strategies. It is assumed other Strategy options available in isisminer will be used to create the Resources in the new Asset.
IsisMiner Optional PVL object containing an IsisMiner definition to run on the Assets. Either this or StrategyConfigFile must be specified.
StrategyConfigFile Optional The CONFIG file to run on the Assets. Either this or an IsisMiner object must be specified.
StrategyConfigFileArgs Optional Replace placeholders in StrategyConfigFile of the form %1, %2, etc. up to %99 with the arguments specified. Each placeholder is replaced by its corresponding argument in order.

The following is an example of an AssetSidebar strategy

Object = Strategy
  Name        = TestCreateAsset
  Type        = AssetSidebar
  Asset       = AssetA
  Operation   = Create
  CreateSource = None
  Description = "Adds AssetA (Assets read from test.csv) to 
  the active Resources" 
  Object = IsisMiner 
    Object = Strategy
      Name = ReadAssetTest
      Type = CsvReader
      CsvFile = "test.csv"
      HasHeader = True
      SkipLines = 0
      IgnoreComments = False
      Delimiter = ", "
    EndObject
  EndObject
EndObject
      

Return to List of Strategy Types

Calculator Strategy Description

The Calculator provides the ability to do calculations on Resource keywords and to store the results in a keyword that is immediately added to the Resource. If any equation fails for a Resource, then that Resource is discarded and no further equations are executed for that Resource.

There are two ways equations can be specified. The basic method requires two keywords to compute, execute, and store the results in each Resource. The Equation keyword stores the complete equation to execute for each Resource and uses the Result keyword value to name the keyword to store the result of the equation.

An alternative form uses a PVL Group named Equations which contains a sequence of keywords whose values are equations. A keyword of the same name is created in each Resource with the value of each successfully computed equation. Values of previous equations can be used in subsequent equations. No other keyword may exist in the Equations group as every keyword is expected to be a valid equation.

Currently this Strategy supports scalars in the form of integers and doubles. In addition, it supports the circle constant pi and Euler's constant e. Functions, such as sin and cos, are followed by parentheses and a list of arguments. Operators such as * and + are placed in between their arguments. If a token is not recognized as a scalar, operator, or function, then it is assumed to be a keyword, and will be replaced by the corresponding value in the Resource.

All trigonometric functions expect angles in radians, not degrees so use the rads() function!

The following table shows all currently supported operators and functions sorted by precedence (0 is the highest precedence). Functions have a precedence of 1 and operators have a precedence of 2 or higher, according to order of operations. All examples are valid equations using 'x' and 'y' as variables:

Precedence
Level
Operator Description Example
0 { [ ( ) ] } Parentheses, brackets, or braces y*(x+[30-{x/pi}])
1 -- or neg Negative sign --x + y or neg(x) + y
1 abs Absolute value abs(y - x)
1 min Minimum x + min(5, y) + min(y, z)
1 max Maximum 20 * max(x,y)
1 sin Sine x * sin(123/321)
1 cos Cosine cos(.02*50)
1 tan Tangent tan(x/y)
1 csc Cosecant 12.3 + csc(x)
1 sec Secant sin(pi/60) + (sec(y))^2
1 cot Cotangent x + cot(x) - 42
1 asin Arcsine 0.006 ^ asin(x*5)
1 acos Arccosine acos(1/[2*pi])
1 atan Arctangent atan(x/e)
1 atan2 Arctangent2 atan2(--10)
1 sinh Hyperbolic sine 55 + sinh(y)
1 cosh Hyperbolic cosine cosh(x^pi)
1 tanh Hyperbolic tangent tanh(x)
1 log or ln Natural log ln(abs(1/[y-x]))
1 log10 Log base 10 99 + log10(x-160)
1 sqrt Square root sqrt(abs[1000 - y])
1 rads Convert degrees to radians x / cos(rads(x))
1 degs Convert radians to degrees degs(acos(x))
1 and Bitwise and and(x, y)
1 or Bitwise or or(x, y)
1 mod Modulus (Note: the arguments are rounded to the nearest integer) mod(x, y)
1 fmod Float modulus, remainder of division fmod(x, y)
2 ^ Exponentiation x ^ 3
3 * Multiplication 10 * x
3 / Division y / x
4 << Bitwise left shift x << 4
4 >> Bitwise right shift y + (x >> 3)
5 + Addition 123 + 0.004 + x
5 - Subtraction 10 - (--x)
6 & Bitwise and x & y
6 | Bitwise or x | y
6 && Logical and x && y
6 || Logical or x || y
6 % Float modulus, remainder of division x % y
6 > Greater than (Note: result is 0 or 1) x > y
6 < Less than (Note: result is 0 or 1) x * (x < y)
6 <= Less than or equal (Note: result is 0 or 1) x <= 0.505
6 >= Greater than or equal (Note: result is 0 or 1) x * (x >= 101)
6 == Equal to (Note: result is 0 or 1) x == (y/2)
6 != Not equal to (Note: result is 0 or 1) x != y

Calculator Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the Calculator strategy.
Type Required Identifies the Calculator strategy - must be Calculator or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the Calculator strategy.
Initializers Optional PVL Group containing Resource keywords to be initialized with initial values. The PVL names are the keywords and the values are the initializers. In some cases, results need to be initialized if they do not already exist in the Resources.
Equation Optional A single equation to be stored in Result. If this and Equations is specified, this equation will be executed and stored first.
Result Optional The keyword where the result of Equation will be stored.
Equations Optional The PVL Group where all keyword values are an equation. The names of the keywords containing the equations in the Equations group are used to store the results of the calculation in each active Resource as a new keyword. If Equation is specified, these equations will be executed after the equation specified in the Equation keyword value.

The following is an example of a Calculator strategy:

Object = Strategy
  Name = Trigonometry
  Type = Calculator
  Description = "Calculate trigonometric functions of angle."
  Group = Initializers
    Sine    = 0
    Cosine  = 0
    Tangent = 0
  EndGroup
  Group = Equations
    Sine    = "sin(angle)"
    Cosine  = "cos(angle)"
    Tangent = "tan(angle)"
  EndGroup
EndObject
      

Return to List of Strategy Types

Control Network Reader Strategy Description

The CnetReader strategy creates Resources from a provided control network file. Each Resource is created from a ControlMeasure group in the control network. The keywords of the ControlMeasure group become the keywords of the Resource.

If the Identity keyword is not provided or its value is an empty string, each Resource is named with a unique number (starting at 0).

CnetReader Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the CnetReader strategy.
Type Required Identifies the CnetReader strategy - must be CnetReader or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the CnetReader strategy.
CnetFile Required User provided control network file.
CnetFileArgs Optional Used for providing any arguments to be substituted in CnetFile's value.
Identity Optional The name given to each Resource. Intended to be used with the IdentityArgs parameter to give each Resource a unique name. If Identity does not have a value, the Resource names are set as unique serial numbers starting at 0.
IdentityArgs Optional Replace placeholders in Identity of the form %1, %2,..., %99, with the arguments specified. Each placeholder is replaced by its corresponding argument in order. The arguments are assumed to be keywords.

The following is an example of a CnetReader strategy:

Object = Strategy
  Name          = ControlNetworkReader
  Type          = CnetReader
  CnetFile      = "%1/Alph_VIS.net"
  CnetFileArgs  = "inputdir"
  Identity      = "%1_%2"
  IdentityArgs  = (PointId, SerialNumber)
  Description   = "Read from a control network file"
EndObject
      

Return to List of Strategy Types

CSV Reader Strategy Description

The CsvReader strategy creates Resources from the rows of a Comma-Separated Values (CSV) file. Each row becomes a Resource with the keyword names from the CSV file header. The keyword values become the values of the corresponding columns in that row. If the CSV file does not have a header, the default keyword names are Column0, Column1, etc.

The CSV file must have the following properties. The rows must be delimited by a new line. The columns must be delimited by a single character. The number of columns in each row must be the same for all rows. If IgnoreComments is set to True, then Comments must be on their own line and be preceded by a #.

CsvReader Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the CsvReader strategy.
Type Required Identifies the CsvReader strategy - must be CsvReader or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the CsvReader strategy.
CsvFile Required The CSV file that the Resources will be created from. The rows must be delimited by a new line. The columns must be delimited by a single character.
CsvFileArgs Optional Replace arguments in CsvFile of the form %1, %2, etc. up to %99 with the values of keywords specified in this array of keywords. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
HasHeader Optional If True, then the first row of the CSV file will be assumed to be the header and used to create keywords of column header names in each Resource (row). If False, the first row is assumed to be data and the keyword will be named using the combination of the ColumnBaseName appended with the column index number. Defaults to True.
Header Optional The list of keywords that the values in each column will be stored in. The column values will be associated with keywords from left to right unless Index is used.
Index Optional The column indices in the CSV file that the keywords in Header will use for their values. The first entry will be the value for the first keyword in Header, the second entry will be the value for the second keyword, etc.
SkipLines Optional The number of lines to be skipped at the beginning of the CSV file. Defaults to 0.
IgnoreComments Optional If True, lines beginning with # will be ignored. If False, all text is processed, regardless of the presence of a # character. Defaults to False.
Delimiter Optional The delimiter used to separate columns of the CSV file. It must be a single character. Defaults to a comma, [ , ].
RowBaseName Optional The default base keyword name to use as the Identity of each Resource. The default Resource names are the base name appended with the row index number. For example Row0, Row1, etc. Defaults to Row.
Identity Optional The name given to each Resource. Intended to be used with the IdentityArgs parameter to give each Resource a unique name. If Identity is not specified, the Resource names are created according to the RowBaseName description.
IdentityArgs Optional Replace arguments in Identity of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
ColumnBaseName Optional The base name used for the default keyword names identifying each column read from the CSV file. If no header is specified, the keyword names will be the value of this keyword with the column index number appended to it, starting at zero. For example Column0, Column1, Column2, etc. Defaults to Column.
GisGeometry Optional The Geometry to be added to the Resources. Must be in either well-known-text or well-known-binary format.
GisGeometryKey Optional The keyword used to reference the Geometry. The name of the column in each row in the CSV file that contains that contains a GIS geometry in well-known text (WKT) or well-known binary (B) format. When converted from its text representation to binary form, it will be added to the Resource (row) as its defining geometry. This keyword, if provided, takes precedence over GisGeometry and GisGeometryRef. Note that special GIS keywords described above will apply to the conversion of these GIS columns to their binary form.
GisGeometryRef Optional Identical to GisGeometryKey
GisGeometryArgs Optional Replace arguments in the GIS geometry keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
GisType Optional The text format of the Geometry. If a Geometry is specified, this keyword is required. Supports WKT (well-known-text) and WKB (well-known-binary).
RemoveGisKeywordAfterImport Optional If True, the Geometry keyword will be removed from the Resources after the Geometry has been added to the Resources. These GIS geometry definitions can be quite large consisting of many characters. To help manage memory, remove these keywords once they are converted to reduce memory requirements.

The following is an example of a CsvReader strategy:

Object = Strategy
  Name = ReadFilterData
  Type = CsvReader
  CsvFile        = "%1/csvreader_data.csv"
  CsvFileArgs    = "inputdir"
  HasHeader      = False
  SkipLines      = 0
  IgnoreComments = False
  Delimiter      = ","
  Identity = "%1_%2"
  IdentityArgs = ( SourceProductId,  StereoSource )
EndObject
      

Return to List of Strategy Types

CSVWriter Strategy Description

The CsvWriter strategy provides users with the ability to export Resources to a Comma-Separated Values (CSV) file. Each row corresponds to a Resource and each column corresponds to a keyword specified by the Keywords parameter. If the Header parameter is set to True, the keyword names are stored in the first row of the CSV file. Each Resource is written to the CSV file as a row with the corresponding keyword values written to each column. If a Resource does not have a specified keyword, the value of the DefaultValue parameter is written instead.

Resource geometries can be written to the CSV file as keywords using the GisGeometryKey parameter in the format specified by the GisType parameter. The two supported formats are well-known binary (WKB) and well-known text (WKT).

Note: Writing a geometry to a CSV file in WKT format while using the default comma delimiter may result in the CSV file being unreadable by CsvReader. Change the delimiter to a different character if using WKT format.

Rows in the CSV file will be delimited by a new line. Columns are delimited by the character specified by the Delimiter parameter, which defaults to a comma.

CsvWriter Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the CsvWriter strategy.
Type Required Identifies the CsvWriter strategy - must be CsvWriter or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the CsvWriter strategy.
CsvFile Required The output CSV file. Can use arguments specified by CsvFileArgs.
CsvFileArgs Optional Replace arguments in the CsvFile keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
Mode Optional The mode in which to write the CSV file. If Create, a new file is created or an existing file with the same name is overwritten. If Append, the output CSV file will be appended to the file. Defaults to Create.
Keywords Required A list of the keywords in the Resources to be written to the CSV file. The keyword names are written to the header of the CSV file as the name of each column. Each Resource is written as a row with the corresponding keyword values written in each column. Only the keywords specified will be written, all other keywords in the Resources are ignored.
Header Optional Option to write a header to the CSV file. Defaults to True, which writes the values specified by the Keywords parameter to the header of the CSV file. False does not write a header to the CSV file, which is useful for appending to another CSV file with the same header.
Delimiter Optional The character or string used to separate fields in a row. Defaults to a comma.
DefaultValue Optional If a Resource does not have a keyword that is specified by the Keywords parameter, it will write this value to the column in the corresponding row CSV file instead. Defaults to NULL.
SkipEmptyLists Optional If True, it will not add empty Resources to the CSV file. Defaults to False.
GisGeometryKey Optional The keyword to write the geometry to. If the keyword already exists in a Resource, then the keyword value will be overwritten with the text version of the GIS geometry. The geometry will be written in the text format specified by GisType. Takes precedence over GisGeometryRef.
GisGeometryRef Optional Identical to GisGeometryKey.
GisType Optional The text format to write the geometry in. The two supported formats are WKT (well-known-text format) and WKB (well-known-binary format). Defaults to WKB. Note that WKT uses commas, which may render the CSV file unreadable if the delimiter is set to a comma.

The following is an example of a CsvWriter strategy:

Object = Strategy
  Type     = CsvWriter
  Name     = mdismla
  CSVFile  = "mdis_mla_ridelong.lis"
  Mode     = Create
  Header   = True
  Keywords = (YearDoy, SourceProductId, StartTime, EtStartTime, 
              ExposureDuration, CenterLongitude, CenterLatitude,
              PixelResolution, MeanGroundResolution,
              IncidenceAngle, EmissionAngle, PhaseAngle, 
              SubSolarGroundAzimuth, SubSpacecraftGroundAzimuth, 
              ParallaxX, ParallaxY, ShadowX, ShadowY)
  Delimiter = ","
  DefaultValue = "NULL"
EndObject
      

Return to List of Strategy Types

DatabaseReader Strategy Description

The DatabaseReader strategy provides direct SQL queries to databases. Data rows retrieved from a query are stored as Resources. Using the Target parameter, new Resources can either be stored on their own as Resources, or as Assets in existing Resources. Using an Asset target is really a shortcut approach to creating Assets in Resources that normally would require an AssetSidebar strategy.

The access details of the database can be stored in a configuration file in a profile of the Database object. The configuration file containing the Database object is specified by the DbConfig parameter. The database profile is specified by the DbProfile parameter. A special parameter called DbFile is provided for SQLite files. This shortened specification for SQLite files eliminates the need for creating an elaborate Database configuration for file-based databases.

Generally, any valid SQL query can be used that returns rows from the results of the query issued to any database. Any database that is supported by the Qt system configuration shipped with each ISIS release can accept SQL queries. This typically includes PostgreSQL, MySQL and SQLite. When specifying the query in the Query, some special quoting may be needed in order to comply with SQL syntax. Before a query is sent to the database, all occurrences of &quot; and &apos; in the query string are replaced with double quotes [ " ] and single quotes [ ' ], respectively.

DatabaseReader Strategy Object Keywords

Keyword Disposition Description
Name Required The user-defined name for the strategy.
Type Required The type of the Strategy. This parameter must be DatabaseReader or the Strategy will not be identified as a DatabaseReader strategy.
Description Optional The user-provided description for the Strategy.
DbConfig Required The configuration file containing the Database object of the database.
DbConfigArgs Optional Replace arguments in the DbConfig keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
DbProfile Required The Name of the database Profile to use in the Database Object in the configuration file specified by the DbConfig parameter.
DbFile Optional A SQLite database file to use as the database. Takes precedence over a database identified by DbConfig and DbProfile.
DbFileArgs Optional Replace arguments in the DbFile keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
Query Required The SQL query to be sent to the database. Double quotes should be escaped by using &quot;.
QueryArgs Optional Replace arguments in the Query keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
QueryStore Optional The Keyword to store the Query used to create the Resource in the new Resources. If this parameter is not specified, the Keyword will not be stored in the Resources.
Target Required Specify how resulting query rows will be stored. If this parameter is set to Resource, the Query will be executed once and each retrieved row will be stored as a new Resource. If this parameter is set to Asset, the Query will be executed once for each Resource, and all rows retrieved for the Query will be stored in each Resource as an Asset containing the list of new Resources, one Resource per row.
Asset Optional If the Target parameter is set to Asset, this parameter is the name of the Asset added to each Resource. If the Target parameter is set to Asset, this parameter is required.
RowBaseName Optional The base of the default Name of each new Resource. The default Resource Names are created from the base and a number. The default base is Row.
Identity Optional The Name of each new Resource. This parameter is intended to be used with the IdentityArgs parameter to give each Resource a unique name. If the Name is not unique or it is empty, the default Name will be used.
IdentityArgs Optional Replace arguments in the Identity keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
GisGeometry Optional The default Geometry to be added to each new Resource. It must be in either well-known-text or well-known-binary format. If the default Geometry is used and this parameter is not specified, no Geometry will be added to the new Resources.
GisGeometryKey Optional The Keyword in the new Resources containing the Geometry to be added to those Resources. If the Keyword does not exist, the default Geometry is used. This parameter takes precedence over GisGeometryRef.
GisGeometryRef Optional Identical to GisGeometryKey.
GisGeometryArgs Optional Replace arguments in the GisGeometry keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
GisType Optional The text format of the Geometry to be added to each new Resource. If this parameter is set to WKT, the Geometry is interpreted as well-known-text. If WKB, it is interpreted as well-known-binary. This parameter is required if GisGeometry, GisGeometryKey, or GisGeometryRef are specified.
RemoveGisKeywordAfterImport Optional Whether the Keyword containing the Geometry should be removed from the new Resources. If this parameter is set to True, the Geometry Keyword will be removed. If it is set to False, the Keyword will remain in the new Resources. Defaults to False.

The following is an example of a valid configuration file named mdis.conf with a DatabaseReader strategy within an IsisMiner object and a Database object:

Object = IsisMiner
  Object = Strategy
    Name = MDISSelection
    Type = DatabaseReader
   
    DbConfig = mdis.conf
    DbProfile = MDIS
   
    Query = "select  i.SourceProductId, i.YearDoy, 
              g.StartTime, i.EtStartTime, i.ExposureDuration,
              g.CenterLongitude, g.CenterLatitude, i.ObservationType,
              g.PixelResolution, g.MeanGroundResolution,
              g.IncidenceAngle, g.EmissionAngle, g.PhaseAngle, 
              g.SubSolarGroundAzimuth, g.SubSpacecraftGroundAzimuth, 
              g.ParallaxX, g.ParallaxY, g.ShadowX, g.ShadowY,
              p.GisFootprint
                from Polygon p
                INNER JOIN Image i
                ON p.SourceProductId=i.SourceProductId 
                INNER JOIN Geometry g
                ON p.SourceProductId=g.SourceProductId 
                INNER JOIN Statistics s
                ON p.SourceProductId=s.SourceProductId
                where (i.Center=747.7 or i.Center=748.7) 
                 order by g.PixelResolution"
    Target = Resource
    Identity = "%1"
    IdentityArgs = "SourceProductId"
    GisGeometryRef = GisFootprint
    GisType        = WKB
    RemoveGisKeywordAfterImport = True
  EndObject
EndObject

Object = Database
  DefaultProfile = "MDIS"
  Group = Profile
      Name = "MDIS"
      Dbname = "Messenger_DOY_2011095_TO_2013011_gis.db"
      Type = SQLite
      Description = "MDIS Sqlite database"
  EndGroup
EndObject
      

Return to List of Strategy Types

Filter Strategy Description

The Filter strategy provides users with the ability to activate (Include) or deactivate (Exclude) Resources that have or do not have keywords with certain values.

All comparisons are case-insensitive. The keyword used to filter Resources is specified in the Keyword key in the Filter Strategy object definition found in the Strategy object. Only the first value in the named Resource Keyword will be checked unless the [optional] CheckAll keyword is set to True.

Values contained in the Include keyword in the Strategy object will be compared with values in the Resource key specified in the Keyword key. If a value exists in both keywords, then the Resource is retained (as active). Values contained in the Exclude keyword in the Strategy object that exist in the Resource key will deactivate a Resource.

Both Include and Exclude keywords may exist in the Filter Strategy object. In this case, values are first checked in the Include list then the Exclude list. Whichever occurs first, that action is taken for the Resource. If none of the keyword values exist in either Include or Exclude keywords, then disposition of the Resource depends on the lists. If the Include list is not empty and the Exclude list is empty, the Resource is deactivated. Otherwise, the Resource is retained as active.

If needed, users can provide a regular expression string that will use Qt's QRegExp class implementation to apply to keyword values. If given, this test is applied after the Include and before the Exclude test to determine if the value satisfies the regular expression. If after all the tests are applied and this string was supplied, the Resource is discarded since it did not succeed in its test.

If neither Include nor Exclude keywords exist, then no action is taken for the Resources.

Filter Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the Filter strategy.
Type Required Identifies the Filter strategy - must be Filter or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the Filter strategy.
Keyword Required Specifies the name of an existing Resource keyword that will be checked for Include and Exclude values.
CheckAll Optional Boolean flag to check all values in the keyword (True) or just the first value (False). Defaults to True.
Include Optional Specify one or more values to check for in the Keyword for existence. If it exists, the keyword is retained (as activated) unless the Exclude parameter is also present - in that case, the rules of Exclude apply. If it does not exist, it is deactivated regardless of the Exclude parameter.
RegExp Optional A single regular expression string can be given in this parameter to test values in the keyword. The syntax in the regular expression must strictly follow the requirements of the Qt QRegExp class. If the string satisfies the regular expression, it is retained as active or it will be discarded.
WildCardMode Optional When using a RegExp, this parameters accepts a value of True or False to signal use of the QRegExp::WildcardUnix mode for regular expressions. If True, QRegExp::WildcardUnix mode is set. If False, the default QRegExp::RegExp mode is used.
Exclude Optional Specify one or more values to look for in the Keyword. If found, the Keyword is deactivated. Otherwise, if Include is not specified, the Keyword is retained.

The following is an example of a Filter strategy:

Object = Strategy
  Name = TossIgnores
  Type = Filter
  Description = "Deactivate all ControlMeasures that are ignored" 
  Keyword = Ignore 
  Exclude = True 
EndObject
      

Return to List of Strategy Types

GISIntersect Strategy Description

The GisIntersect strategy provides users with the ability to select Resources that intersect with a given geometry. Resource geometries must be created when Resources are created (typically from CsvReader and DatabaseReader). Users can define a geometry using the GisGeometry keyword. GisType specifies the type or format of geometry provided in this keyword. Resources which overlap the given geometry are activated and Resources which do not overlap the given geometry are deactivated. Overlaps are determined using the GIS intersect operator.

GisIntersect Strategy Object Keywords

Keyword Description
Name Required User-defined name of the GisIntersect strategy.
Type Required Identifies the GisIntersect strategy - must be GisIntersect or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the GisIntersect strategy.
GisType Required Specifies the type of the geometry. Valid options are WKT, WKB, pvlWKT, pvlWKB, or IsisCube. WKT and WKB are common GIS file formats that indicate well-known-text and well-known-binary, respectively. If GisType is pvlWKT or pvlWKB, GisGeometry must be the path to a PVL file which contains a valid WKT or WKB geometry.
GisGeometry Required The actual geometry or geometry source. This can either be directly equal to a WKB or WKT value like POLYGON ((0.00 22.50, 90.00 22.50, 90.00 65.00, 0.00 65.00, 0.00 22.50)) or can be set to the path to a PVL file containing a valid geometry. If GisGeometry is set to a file name, the associated file must be a PVL file with a valid WKT or WKB geometry, or be an ISIS cube.
GisGeometryArgs Optional Replace arguments in the GisGeometry keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
BoundingBox Optional Specify whether or not to use a bounding box from the source. Possible values are True or False. Default value is False.
ComputeRatio Optional Specify whether or not to compute the overlap ratio and add it to the Resource. Options are True or False. Default value is False.
RatioRef Optional Specify which Keyword to insert the calculated overlap ratio in, if ComputeRatio = True.
GisGeometryRef Optional Used when using a file name as a GisGeometry source. This indicates the Keyword in the associated PVL file that the WKT or WKB geometry value will be found in.

The following is an example of a GisIntersect strategy:

Object = Strategy
  Name = DeactiveNonintersecting
  Type = GisIntersect
  Description = "Define bounding box (lower left, upper right (lon lat)) 
                       around Degas"
  GisType = "WKT"
  GisGeometry =  "LINESTRING(225.00 30.00, 240.00 45.00)"
  BoundingBox = True
  ComputeRatio = True
EndObject
      

Return to List of Strategy Types

GisOverlap Strategy Description

The GisOverlap strategy provides users with the ability to determine Resources that have overlapping GIS footprints. This Strategy algorithm is useful to identify images that have common coverage which is the first requirement for determining stereo (see also StereoPair strategy).

Parameters to this algorithm can define the ratio (or percentage) of common overlap for the two GIS geometries. As implied, all Resources must have GIS geometries in order to participate in the overlap determination.

Overlap candidates are determined from other Resources for every incoming active Resource and a list of Resources that are determined from another isisminer Strategy configuration. The configuration can either be specified directly in the GisOverlap Strategy object (an embedded IsisMiner object called a CandidateMiner) or come from an external file called CandidateConfigFile. Candidate overlap Resources can be either part of the incoming Resource list (active and inactive) or from a completely different source.

Once overlaps are determined for a Resource, the candidate and its source are merged/combined into a single Resource and added to an Asset of the primary Resource. Keywords from both Resources are combined into a single Resource by renaming each keyword from both Resources and propagating them to the merged Asset Resource. An A is appended to the source Resource keywords and a B is appended to the candidate Resource keywords before they are merged to distinguish keywords. The suffix of keyword sources can be specified by the user using the MergeSuffixA and MergeSuffixB keywords, respectively. The identity of the merged Resource is a combination of the two Resource identities separated by an underscore character.

Once the list of overlapping Resources is determined, the user can opt to immediately process the overlaps. If a second isisminer configuration is provided called OverlapMiner or OverlapConfigFile, the list of overlapping Resources that are contained combined Resources and geometries is run though the Strategy list in the OverlapMiner configuration. This option is useful to eliminate the need for intermediate storage which could cause memory issues.

GisOverlap Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the GisOverlap strategy.
Type Required Identifies the GisOverlap strategy - must be GisOverlap or it will not be recognized as a valid Strategy by isisminer.
OverlapMinimum Optional Specifies the minimum common overlap ratio of the two geometries with respect to the primary Resource to be valid. If the computed ratio is less than this value, the Resource will not be deemed a valid overlap due to lack of common coverage. Defaults to 0.0.
OverlapMaximum Optional Specifies the maximum common overlap ratio of the two geometries with respect to the primary Resource to be valid. If the computed ratio is greater than this value, the Resource will not be deemed a valid overlap due to too much common coverage. Defaults to 1.0.
OverlapRatioKey Optional Provides the name of the keyword to retain the common overlap ratio. If not provided, it will be stored in the merged Resource as the keyword OverlapRatio.
Asset Optional Provides the name of the Asset created containing all the Resources determined to have overlap GIS geometries from the candidate list. If not provided, GisOverlap will be the default.
MergeSuffixA Optional To provide uniqueness of keywords that are common to both Resources when they are combined, a suffix is added to the primary Resource. Use this keyword to specify the keyword suffix. If not provided, A is the default.
MergeSuffixB Optional To provide uniqueness of keywords that are common to both Resources when they are combined, a suffix is added to the Resource that overlaps the primary Resource. Use this keyword to specify the keyword suffix of the overlapping Resource. If not provided, B is the default.
OverlapMerge Optional

GIS Operators

Users can determine how to merge the GIS geometries of overlapping Resources using the following options. Note the resulting geometry is stored in the combined Resource of the primary and overlapping Resources. Note also that two additional keywords are added to the combined Resources that provide the centroid coordinate of the resulting geometry. These keywords are GisOverlapCentroidX and GisOverlapCentroidY that provide the center coordinate of the geometry X (typically longitude) and Y (latitude). These become useful for computing common stereo data.
Parameter Description
Intersection Combines the two GIS footprints as an intersection. This will result in a GIS geometry that contains the area of common overlap.
Union Combines the two GIS footprints as a union. This will create a geometry made of the area of both Resource geometries.
Centroid This option creates a point geometry of the centroid computed from the intersection of the two geometries.
ResourceA Propagates the geometry of the primary Resource used in the overlap determination.
ResourceB Propagates the geometry of the matching/overlapping Resource used in the overlap determination. It is the Resource found in the list used to match the primary Resource.
None No geometry will be created. This is the default.
CandidateMiner Optional (Object) To select the Resource list used to evaluate overlapping geometries, users have several ways to create the list. A complete isisminer configuration object called (and named) CandidateMiner can be provided that will be executed on the full Resource list to generate a list of candidate Resources for comparisons of overlaps. This configuration must be a compliant, properly formatted isisminerobject with the only difference being the name of the object. It may also be provided in a file called CandidateConfigFile. Either a CandidateMiner object or configuration file may be provided, but not both. If no candidate options are provided, the current active list of Resources will be used.
CandidateConfigFile Optional (File) This option can be used in lieu of a CandidateMiner object to specify a set of Strategies to select overlap candidates. You may also provide arguments to that can be translated when executed to resolve the file name by providing a keyword called CandidateConfigFileArgs that contains keywords that translate occurrences of %1 to %n strings in the name of the file.
OverlapMiner Optional (Object) For efficiency purposes only, users can immediately process the overlap list of a given Resource at the time it is completely determined. Simply provide an isisminer configuration object called (and named) OverlapMiner that will be executed on the overlap Resource at the time each one is fully determined. This configuration must be a compliant, properly formatted isisminerobject with the only difference being the name of the object. It may also be provided in a file called OverlapConfigFile. Either an OverlapMiner object or a configuration file may be provided, but not both. If no overlap options are provided, no action is taken on the overlap list.
OverlapConfigFile Optional (File) This option can be used in lieu of a OverlapMiner object to specify a set of Strategies to process overlap Resource results. You may also provide arguments to that can be translated when executed to resolve the file name by providing a keyword called OverlapConfigFileArgs that contains keywords that translate occurrences of %1 to %n strings in the name of the file.

The following is an example using the GisOverlap strategy. In this example we are reading image data from an SQLite database containing WAC and NAC MESSENGER data created from caminfo runs. It determines all the images overlapping a square geometry around Debussy crater.

Object = IsisMiner

  Object = Strategy
  Name = MDISSelection
    Type = DatabaseReader

    DbConfig = wac_nac_overlaps.conf
    DbProfile = MDIS
    Mode = Select

    #  Select strings that need double quotes must use " since they cannot
    #  be stored in keyword values.  These strings will be prescanned before
    #  issuing to the DB and they will be properly replaced.
    Query = "select  i.SourceProductId, i.YearDoy, i.File,
              g.StartTime, i.EtStartTime, i.ExposureDuration, i.ObservationId,
              g.CenterLongitude, g.CenterLatitude, i.ObservationType,
              g.PixelResolution, g.MeanGroundResolution,
              g.IncidenceAngle, g.EmissionAngle, g.PhaseAngle, 
              g.SubSolarGroundAzimuth, g.SubSpacecraftGroundAzimuth, 
              ParallaxX, ParallaxY, ShadowX, ShadowY,
              p.GisFootprint
                from Polygon p
                INNER JOIN Image i
                ON p.SourceProductId=i.SourceProductId 
                INNER JOIN Geometry g
                ON p.SourceProductId=g.SourceProductId 
                INNER JOIN Statistics s
                ON p.SourceProductId=s.SourceProductId"

  # Resource targets are for reading multiple rows from the DB.  The fields
  # from each row are created as Resources.
    Target = Resource
    Identity = "%1"
    IdentityArgs = "SourceProductId"

  #  Can specify a GIS keyword that will be converted on input
    GisGeometryRef = GisFootprint
    GisType        = WKB
    RemoveGisKeywordAfterImport = True
  EndObject

 Object = Strategy
    Type = GisIntersect
    Name = SelectDebussy
    GisType = WKT 
    #  Define bounding box (lower left, upper right (lon lat)) around Debussy
    #  crater
    GisGeometry = "LINESTRING(11.56 -34.91, 13.83 -33.03)"
    BoundingBox = True
    ComputeRatio = True
    RatioRef = DebussyOverlapRatio
    ShowProgress = True
  EndObject

# Since there is no CandidateMiner specified so the current active list is 
#  processed, recursively. Meaning each Resource is checked for overlap
#  of all other active Strategies
  Object = Strategy
    Type = GisOverlap
    Name = DebussyOverlaps
    Description = "Check for image overlaps near Debussy"
    ShowProgress = True
    OverlapMinimum = 0.5
    OverlapMerge = Intersection
  EndObject

  Object = Strategy
    Type = AssetSidebar
    Name = WriteOverlapFiles
    Description = "Write CSV files of overlap data"
    Asset = GisOverlap

    Object = IsisMiner
      Object = Strategy
        Type = CsvWriter
        Name = WriteOverlaps
        Description = "Write individual files with overlap data"

        SkipEmptyLists = True
        CsvFile = "overlaps/%1.csv"
        CsvFileArgs = "Identity"
        Keywords = ("SourceProductIdA", "SourceProductIdB",
                    "OverlapRatioA", "OverlapRatioB",
                    "YearDoyA", "YearDoyB", 
                    "CenterLongitudeA", "CenterLatitudeA",
                    "CenterLongitudeB", "CenterLatitudeB",
                    "GisOverlapCentroidX", "GisOverlapCentroidY",
                    "PixelResolutionA", "PixelResolutionB", 
                    "EmissionAngleA", "EmissionAngleB", "IncidenceAngleA",
                    "IncidenceAngleB", "PhaseAngleA", "PhaseAngleB")
      EndObject
    EndObject

  EndObject

EndObject


Object = Database
  DefaultProfile = "MDIS"
  Group = Profile
    Name = "MDIS"
    Dbname = "Messenger_Monochrome_DOY_2011095_TO_2014304_gis.db"
    Type = SQLite
    Description = "MDIS Sqlite database"
  EndGroup
EndObject

End
      

Return to List of Strategy Types

GISUnion Strategy Description

The GIS Union strategy provides users with the ability to create unions between GIS geometries. Each geometry which the Strategy operates on is unioned with the other supplied geometries, and the output is a union which best fits all of the input geometries.

If the union geometry is present and good, an entry will be added to the Resource with the parameter RatioRef as the key and the overlap ratio as the value. The double parameters OverlapMinimum and OverlapMaximum indicate the acceptable range for the overlap ratio. Geometries with overlap ratios which do not fit the constraints will be discarded.

GisUnion Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the GisUnion strategy.
Type Required Identifies the GisUnion strategy - must be GisUnion or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the GisUnion strategy.
OverlapMinimum Optional User provides a value for the minimum valid overlap ratio. Defaults to 0.0.
OverlapMaximum Optional User provides a value for the maximum valid overlap ratio. Defaults to 1.0.
RatioRef Optional User provides a keyword name which will be used as the column of overlap ratios stored in the Resource. Defaults to UnionOverlapRatio.

The following is an example of a GisUnion strategy:

Object = Strategy
  Name = UnionHighOverlap
  Type = GisUnion
  Description = "Union remaining geometries which have a high overlap ratio."
  OverlapMinimum = 0.65
  OverlapMaximum = 1.0
  RatioRef = "HighOverlapRatio"
EndObject
      

Return to List of Strategy Types

IsNumeric Strategy Description

The IsNumeric strategy provides users with the ability to check if Resource keyword values are numeric. Resources with a non-numeric value are deactivated.

A keyword value is considered numeric if it can be converted to a floating point number. If the keyword does not exist in the Resource, then its value is not considered numeric.

IsNumeric Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the IsNumeric strategy.
Type Required Identifies the IsNumeric strategy - must be IsNumeric or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the IsNumeric strategy.
Keyword Required Specifies the name (or names - you can list more than one keyword to check) of an existing Resource keyword that will be included if it is numeric.

The following is an example of an IsNumeric strategy:

Object = Strategy
  Name = RemoveNotApplicable
  Type = IsNumeric
  Description = "Deactivate all responses which did not respond to rating."
  Keyword = Rating
EndObject
      

Return to List of Strategy Types

Limit Strategy Description

The Limit strategy provides users with the ability to check if Resource keyword values are within user-defined limits. Values that are outside of the limits result in Resource deactivation.

If keyword values are within the limits, the Resource disposition is set to activated. If the keyword is not within the limits provided, the Resource disposition is set to deactivated.

This Strategy uses the Calculator strategy. Any equation constructed that is valid for the Calculator strategy is valid for Limit. This includes the use of Equations group to use more than one equation. The difference here is the value of the last equation is used to evaluate a true (!0) or false (0) limit for the Resource. Also note, that a side effect of using the Equations group is that any new keyword created by an equation remains in the Resource.

Limit Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the Limit strategy.
Type Required Identifies the Limit strategy - must be Limit or it will not be recognized as a valid Strategy by isisminer.
Equation Optional Specifies an equation using Resource keywords that will be checked for true (!0) or false (0) results.
Equations Optional This group can be used instead of the Equation keyword to define a series of equations executed in sequence. The last resulting computation will be used to check for true (!0) or false (0) results. Either the Equation keyword or Equations group must be specified.

Following are few examples of Limit strategies:

Object = Strategy 
  Type = Limit
  Name = EmissionAngle
  Equation = "((emissionangle > 30) && (emissionangle < 80))"
EndObject
      
This shows an example of using a series of equations to determine limits:
Object = Strategy
  # This algorithm will select only images where the center latitude
  # and longitudes are within the defined region provided in the GLOBALS
  # parameter file.
  Name = SelectRegion
  Type = Limit
  Description = "Find all images with center image geometry in the
                 defined region"

  # This set of equations will result in 1 if test are good, meaning the
  # clat/clon is in the defined region. If any test fails, InRegion = 0
  # (false) and the Resource is disabled. Note this also defines the GIS
  #  bounding box.
  Group = Equations
    InRegion = 1
    InRegion = "InRegion * (CenterLongitude >  RegionLowerLeftLongitude)"
    InRegion = "InRegion * (CenterLatitude  >  RegionLowerLeftLatitude)"
    InRegion = "InRegion * (CenterLongitude <= RegionUpperRightLongitude)"
    InRegion = "InRegion * (CenterLatitude  <= RegionUpperRightLatitude)"
  EndGroup
EndObject
      

Return to List of Strategy Types

NumericalSort Strategy Description

The NumericalSort strategy provides users with the ability to sort Resources according to a numerically-valued keyword (SortKey). Resources may be sorted in Ascending or Descending order according to the value specified in Order.

SortKey must have a numerical value for each Resource that will be sorted. See the IsNumeric strategy.

Order may be specified as either Ascending or Descending. If not provided, the Resources will be sorted in Ascending order.

NumericalSort Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the NumericalSort strategy.
Type Required Identifies the NumericalSort strategy - must be NumericalSort or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the NumericalSort strategy.
SortKey Required Specifies the name of an existing Resource keyword that will be used to sort the Resources. The keyword's values for each Resource must be numerical.
Order Optional The order, ascending or descending, that Resources can be sorted in. Defaults to Ascending.

The following is an example of a NumericalSort strategy:

Object = Strategy
  Name = RankSort
  Type = NumericalSort
  Description = "Sorts the Resources in ascending order by Rank" 
  SortKey = Rank
  Order = Ascending
EndObject
      

Return to List of Strategy Types

PDS Table Creator Strategy Description

The PdsTableCreator strategy is used to create and write a PDS table file from column Resources. When the PDS table format file is created the column Resources are converted to PdsColumns, and all active Resources are written to the file.

Records are delimited by a new line. Fields are delimited as specified by the Delimiter parameter, which defaults to a comma.

PdsTableCreator Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the PdsTableCreator strategy.
Type Required Identifies the PdsTableCreator strategy - must be PdsTableCreator or it will not be recognized as a valid Strategy by isisminer.
PdsFormatFile Required The output format file. Can use arguments specified by PdsFormatFileArgs.
PdsFormatFileArgs Optional Replace placeholders in PdsFormatFile of the form %1, %2,..., %99, with the arguments specified. Each placeholder is replaced by its corresponding argument in order.
PdsTableFile Required The output Pds table file. Can use arguments specified by PdsTableFileArgs.
PdsTableFileArgs Optional Replace placeholders in PdsTableFile of the form %1, %2,..., %99, with the arguments specified. Each placeholder is replaced by its corresponding argument in order.
Mode Required The mode in which to write the Pds Table file. If Create, a new file is created or an existing file with the same name is overwritten. If Append, the output Pds Table file will be appended to the file. Defaults to Create.
Delimiter Optional The character or string used to separate fields in a record. Defaults to a comma.
DefaultValue Optional If a Resource does not have a keyword that is specified by the Keywords parameter, it will write this value to the column in the corresponding row Pds Table file instead. Defaults to NULL.

The following is an example of a PdsTableCreator strategy:

Object = Strategy
  Name = TestAppendTableWriter
  Type = PdsTableCreator
  PdsFormatFile = "%1/TestGeneralFormat.txt"
  PdsFormatFileArgs = "inputdir"
  PdsTableFile = "%1/TestAppend.TAB"
  PdsTableFileArgs = "outputdir"
  Mode = Append
  Delimiter = ","
  DefaultValue = "NULL"
EndObject
      

Return to List of Strategy Types

PDS Table Format Strategy Description

The PdsTableFormat strategy is used to create and write a PDS table format descriptor file for a PDS table. When the PDS table format file is created, the columns are created according to the byte size of the Resource, and then that data is stored in the format file. The Resources in the column are delimited by the delimiter.

Records are delimited by a new line. Fields are delimited as specified by the Delimiter parameter, which defaults to a comma.

PdsTableFormat Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the PdsTableFormat strategy.
Type Required Identifies the PdsTableFormat strategy - must be PdsTableFormat or it will not be recognized as a valid Strategy by isisminer.
PdsFormatFile Required The output format file. Can use arguments specified by PdsFormatFileArgs.
PdsFormatFileArgs Optional Replace arguments in the PdsFormatFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
KeywordFormatFile Optional Name of a file containing individual PDS keyword file formatting directives. Provides user with customization options. Defaults to $ISISROOT/appdata/translations/pdsExportRootGen.typ.
KeywordFormatFileArgs Optional Replace arguments in the KeywordFormatFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
Delimiter Optional The character or string used to separate fields in a record. Defaults to a comma.
Column Optional The names of the columns to be written to the format files created.
DataType Optional The type of data within the columns.
Unit Optional The unit associated with the data in the column.
DataDescription Optional User provides a description of the data stored in the column.
{Column Name}_BYTES Optional Specifies the size of the column, named in Column, in order to make it more efficient, in order to eliminate the need to traverse the entire table to determine the largest element in the column.
Example: POINT_ID_BYTES = 32

The following is an example of a PdsTableFormat strategy:

Object = Strategy 
  Name = PdsTableFormat
  Type = PdsTableFormat
  PdsFormatFile = "POINTCLOUDTAB.FTM"
  PdsFormatFileArgs = "outputdir"
  Delimiter = ","
  Column = ("POINT_ID", "STATUS") 
  DataType = ("CHARACTER", "CHARACTER")
  Unit = ("NONE", "NONE") 
  DataDescription = ("Unique point identifier.", "Status of point") 
  POINT_ID_BYTES = 32 
  STATUS_BYTES= 32
EndObject 
      

Return to List of Strategy Types

PDS Table Reader Strategy Description

The PdsTableReader strategy loads a PDS formatted table using a label containing a description of the columns in the table (contained in PdsLabelFile).

If the Identity keyword is not provided or its value is an empty string, each Resource created will be named the row number. Rows are read (starting at 0) from the PDS table data in PdsTableFIle.

PdsTableReader Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the PdsTableReader strategy.
Type Required Identifies the PdsTableReader strategy - must be PdsTableReader or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the PdsTableReader strategy
PdsLabelFile Required User provides file containing label describing the columns in the table.
PdsLabelFileArgs Optional Replace arguments in the PdsLabelFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
PdsTableFile Optional User may need to provide the name of the table file containing data. This may not be needed if the base file name is the same as the label file but ends in .tab.
PdsTableFileArgs Optional Replace arguments in the PdsTableFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
Identity Optional The name given to each Resource. Intended to be used with the IdentityArgs parameter to give each Resource a unique name. If Identity does not have a value, the Resource names are set as PvlN, N being the position of the PVL file in FromList.
IdentityArgs Optional Replace arguments in the Identity keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
TableName Optional The name of the table object in the label file may use different names. Users will need to provide the name of the PVL object containing the descriptions of each column. Defaults to TABLE.
UseFormattedName Optional Some of the tables will contain column names that contain underscores and some special characters. If set to True, all white space characters are converted to spaces. Spaces are then compressed to one space. Any left/right parentheses are removed. Then the conversion to camel-case is performed. Camel case always converts the first character in the name to uppercase. Each space or underscore character is removed and the character following it is converted to uppercase. All other characters are converted to lowercase.

The following is an example of an PdsTableReader strategy:

Object = Strategy
  Name             = PdsTableReaderTestDefault
  Type             = PdsTableReader
  PdslabelFile     = "%1/index.lbl"
  PdslabelFileArgs = "inputdir"
  TableName        = "INDEX_TABLE"
  Identity         = "%1"
  IdentityArgs     = File_Name
  UseFormattedName = False
  Description      = "Test the default functionality of PdsTableReader"
EndObject

      

Return to List of Strategy Types

PVL Reader Strategy Description

The PvlReader strategy creates Resources from a list of paths to PVL files (contained in FromList). Each PVL file becomes a Resource where the keywords in the PVL file become the keywords of the Resource. Resources may be created against any provided constraints ( Includes, Excludes, and KeyListFile), which describe which keywords become part of the created Resource.

If the Identity keyword is not provided or its value is an empty string, each Resource created will be named as PvlN. N is the position number (starting at 0) of the PVL file contained in FromList. The basename, Pvl, can be changed by setting PvlBaseName.

For example, if FromList contains PVL Files ~/somepvl.pvl and ~/someotherpvl.pvl, their default Identity values would be Pvl0 and Pvl1, respectively.

The Includes keyword operates on PVL Objects and Groups (containers). When provided by itself, the Includes keyword will exclusively include the keywords belonging to the containers specified. Since Groups must be contained in Objects, you cannot include a Group without also including its containing Object. Similarly, you must include the Root Object since all PVL files have a Root Object container. For example:

Object = Outer
    Id = 001
    Group = ExtraInfo
        Name = outside
        Time = today
    EndGroup
    Group = OtherInfo
        Info = "No information present"
    EndGroup
    Group = Important
        Level = High
    EndGroup
    Object = Inner
        Data = "Some test data"
    EndObject
EndObject
      
To include the ExtraInfo Group, the Includes keyword should be specified, which will include the Id, Name, and Time keywords in the created Resource.
Includes = (Root, Outer, ExtraInfo)

The Excludes keyword operates on PVL Objects and Groups (containers). When provided by itself, the Excludes keyword will exclude the keywords belonging to the containers specified, while all other containers and keywords will be included.

When both Includes and Excludes are provided, an Object and all of its keywords and Groups will be included except for any excluded Groups (Inclusions are applied to Objects while exclusions are applied to Groups).

Referring to the Includes example, to include the Outer Object except for its OtherInfo Group, you would specify the following:

Includes = (Root, Outer)
Excludes = OtherInfo
      

The KeyListFile keyword operates on PVL Keywords and expects a file containing PVL Keywords. If provided, only the keywords in the file will be included in the created Resource. Note that these keywords will only be included if their containers are included or not excluded.

PvlReader Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the PvlReader strategy.
Type Required Identifies the PvlReader strategy - must be PvlReader or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the PvlReader strategy.
FromList Required User provides file containing a list of paths to PVL files.
FromListArgs Optional Replace arguments in the FromList keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
Identity Optional The name given to each Resource. Intended to be used with the IdentityArgs parameter to give each Resource a unique name. If Identity does not have a value, the Resource names are set as PvlN, N being the position of the PVL file in FromList.
IdentityArgs Optional Replace arguments in the Identity keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
Includes Optional Used to exclusively include Objects and Groups when creating a Resource from a read PVL file.
Excludes Optional Used to ignore/exclude Objects and Groups when creating a Resource from a read PVL file.
KeyListFile Optional Used to exclusively include any PVL Keywords in the provided file when creating a Resource from a read PVL file.
KeyListFileArgs Optional Replace arguments in the KeyListFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names.
PvlBaseName Optional The base of the default name of each Resource. The default Resource names are the base name and a number. For example: Pvl0, Pvl1, etc. Defaults to Pvl.

The following is an example of an PvlReader strategy:

Object = Strategy
  Name          = PvlReader
  Type          = PvlReader
  FromList      = "%1/testpvl.pvl"
  FromListArgs  = "inputdir"
  Identity      = "%1"
  IdentityArgs  = Id
  KeyListFile   = "%1/keystoinclude.lis"
  KeyListFileArgs = "inputdir"
  Description   = "A PvlReader Strategy that includes the keywords contained in keystoinclude.lis
                             when creating the Resource from testpvl.pvl"
EndObject
      

Return to List of Strategy Types

Resource Manager Strategy Description

The ResourceManager strategy provides users with the ability to apply simple operations to Resources. These operations are ResetDiscard, ToggleDiscard, DeleteDiscard, DeleteAsset, and HasAsset. For more information, please see the table below.

For DeleteAsset and HasAsset, the Asset is specified in the Operations Keyword as
Operations = HasAsset::AssetName or Operations = DeleteAsset::AssetName.

ResourceManager Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the ResourceManager strategy.
Type Required Identifies the ResourceManager strategy - must be ResourceManager or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the ResourceManager strategy.
Operations Required

Operations

This specifies which operation to perform on the Resource. A list of operations can also be specified as Operations={Operation1, Operation2, ...}.
Parameter Description
ResetDiscard Re-activates all discarded Resources.
ToggleDiscard Activates all deactivated Resources Resources and deactivates all activated Resources.
DeleteDiscard Removes all deactivated Resources.
DeleteAsset Deletes a given Asset from all Resources that have it. Must be of form DeleteAsset::AssetName.
HasAsset Discards Resources without a given Asset. Must be of form HasAsset::AssetName.

The following is an example of a ResourceManager strategy:

Object = Strategy
  Name = SwitchActivated
  Type = ResourceManager
  Description = "Swap activated (Included) and deactivated (Excluded) entries in a Resource."
  Operations = ToggleDiscard
EndObject
      

A second example with multiple management operations

# Clear the entire Resource list (useful in Sidebar Strategies)
Object = Strategy
  Name = ClearResources
  Type = ResourceManager
  Description = "Delete all Resources"
  Operations = ( ResetDiscard, ToggleDiscard, DeleteDiscard )
EndObject
        

Return to List of Strategy Types

Run Command Strategy Description

The RunCommand strategy allows shell commands to be executed for each Resource. In addition, there is frequently a need to run commands prior to and after individual Resource processing. For this purpose, groups called PreCommands and PostCommands contain commands that are executed before and after, respectively, processing each Resource.

Commands can be formulated using the keyword argument substitution options available for using global parameter pools for replacing arguments in commands with Resource keyword values. One keyword, CommandArgs, is available to list all the Resource and global pool variables that are to be used in replacement. All commands can use a %1 to %n where n is the total number of keywords list in the CommandArgs keyword. This one list of keywords is used to substitute for all commands.

RunCommand Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the RunCommand strategy.
Type Required Identifies the RunCommand strategy - must be RunCommand or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the RunCommand strategy.
Command Optional Provides a single command that will be executed once for each active Resource.
Commands Optional This is a Group in the RunCommand strategy that can contain more than one command to execute for each Resource. This can exist with the Command keyword but contents of this group will be executed after the Command content.
PreCommands Optional The contents of this group contains optional commands that will be run once before any Resources (and their commands) are executed. For example, this group is useful for creating directories prior to processing any Resource commands.
PostCommands Optional The contents of this group contains optional commands that will be run after all Resources (and their commands) are executed. For example, this group is useful for running an ISIS application after processing all Resources.
CommandArgs Optional This keyword contains keywords that will be used to substitute any occurrences of keyword arguments (i.e., %1, %2,...) in all commands regardless of which group the command is from.

The following is an example of a RunCommand strategy. The &quot; text is required and should be included in the configuration of your command. The &quot; and &apos; are substituted prior to issuing the command to the shell for execution. This is required because of PVL syntax rules.

Object = Strategy
  Name         = RunManyCommands
  Type         = RunCommand
  Description  = "Tests running many command"

  CommandArgs = (outputdir, SourceProductId, StartTime)
  # Ensure the output file exists
  Group = PreCommands
    Command0 = "touch %1/runmanycommands.txt"
  EndGroup

  # Append the Resource name and start time
  Group = Commands
    Command1 = "echo &quot;Resource name %2 and date %3.&quot; >> %1/runmanycommands.txt"
  EndGroup
EndObject

      

Return to List of Strategy Types

Sidebar Strategy Description

The Sidebar strategy allows complete isisminer configurations to be run within a single Strategy. The configuration can be loaded from a file or from an embedded IsisMiner object. The configuration can be run without affecting the active state of Resources with the SavePoint keyword. It can be run without deleting or creating Resources with the SaveDelete keyword.

Sidebar strategies are useful for running Strategies that have the potential to throw out Resources or alter a current state that may be undesirable. For these cases, you can use a Sidebar that saves all the Resources, activated and deactivated, their current states and preserves any operations that may alter or discard Resources. It can also be used to process an entirely new set of Resources while retaining the current state of existing Resources.

Sidebar strategies can be inserted anywhere in the sequence of Strategies but are best placed strategically at critical points in the processing sequence where preservation of the current active Resource state is needed.

Sidebar Strategy Object Keywords

Keyword Disposition Description
Name Required User-defined name of the Sidebar strategy.
Type Required Identifies the Side strategy - must be Sidebar or it will not be recognized as a valid Strategy by isisminer.
Description Optional User provides a description of the Sidebar strategy.
IsisMiner Optional (Object) PVL Object containing an IsisMiner definition to be run on the list of Resources. Either this or StrategyConfigFile must be specified.
StrategyConfigFile Optional (File) File containing run parameters on the Resources. If this file is not given, an IsisMiner object must be specified.
StrategyConfigFileArgs Optional Replace placeholders in StrategyConfigFile of the form %1, %2, etc. up to %99 with the arguments specified. Each placeholder is replaced by its corresponding argument in order.
SavePoint Optional If True, the activeness of all the Resources will be retained after the sidebar configuration has run. If False, any Resources that are discarded during the configuration will remain discarded and lost to subsequent processing. Defaults to True.
SaveDelete Optional If True, any Resources that are deleted by the side configuration will be restored. In addition, any Resources that are created during the configuration will be deleted. If False, deleted Resources will remain deleted and created Resources will remain. Defaults to the value of SavePoint.

The following is an example of a Sidebar strategy:

Object = Strategy
  Name = SetUpResources
  Type = SideBar
  SavePoint = True
  SaveDelete = False
  StrategyConfigFile = "%1/setup.conf"
  StrategyConfigFileArgs = "inputdir"
EndObject
      

Return to List of Strategy Types

StereoPair Strategy Description

The StereoPair strategy provides users with the ability to determine, from Resources that have overlapping GIS footprints, stereo pairs that pass a series of constraints. This Strategy algorithm takes direct advantage of both the GisOverlap and Calculator strategies in characteristics of those Strategies are incorporated into this Strategy. The configuration for the GisOverlap and Calculator strategies (i.e., all keyword parameters) are included in the StereoPair configuration as their functionality is used in the determination of ranked (best) stereo pairs. GisOverlap performs the preliminary requirement of image overlap determination and ratio limits, disposition of overlap geometry and merging of the stereo pair Resources into one Resource. The Calculator strategy computes the stereo strength for a pair from the results of ImageStrength and StereoStrength keyword sets. Each keyword in the groups evaluates to values greater than or equal to 0 and less than or equal to 1 where 1 is the highest value for a keyword (unless a multiplier is used).

Parameters for this algorithm define two levels of stereo constraints that are described in [Becker 2015]. The first level of constraints are typically for constraining photometric (incidence, emission, phase) and geometric angles. The constraints for these limits are located in the ImageStrength group. Each keyword there should exist in each Resource or have a keyword map specified in the KeywordMap group. The second level constraints contained in the StereoStrength group are stereo limits and are applied to values computed from the stereo algorithm with contributions from both pair Resources. The format for keywords in both of these groups are four element vectors but only 3 values are required. For example, to limit the emission angle between 0 and 50 degrees, the form of the keyword in the ImageStrength group is Emission = (0.0, 50, 15.0 [, 1.0]). The first value in this keyword is the minimum value, the second is the maximum value and the third value is the value for emission that provides the best stereo conditions. The last optional value is a multiplier of the computed strength, or rank, for each keyword limit that can be used to weight the computed strength of the limit beyond the 0 - 1 range.

The rank of each keyword is computed using the following scheme based upon this form of limit keywords in both ImageStrength and StereoStrength: LimitKey = (t0, t1, t2, t3) where all t0 through t3 values are numerical numbers and t0 is the minimum, t1 is maximum, t2 is the best value in the range and t3 is a weighting factor. The formula to compute the rank of each keyword is as follows:

range = max(abs(t2-t0), abs(t1-t2))
rank = 1 - abs(t2-value)/range
rank *= t3          
      
The end result of rank is then stored in a keyword comprised of the name of the keyword with Rank appended to it. For example if the Emission keyword above is provided as a limit, and an image has an Emission angle value of 30 degrees, the rank computed and stored in EmissionRank = 1 - abs(15 - 30)/max(abs(15-0),abs(50-15)) = 0.57 * 1 (t3). All keyword ranks for a stereo Resource are summed for each group and the values are available in ImageStrengthRank and StereoStrengthRank. The ImageStrengthRank is computed as the average of the two image's rank. These values are typically summed in the Equation keyword in the Strategy definition to specify how to rank the strength of an individual stereo pair. These values can be sorted and reported to users (using an AssetSidebar). The typical equation used to compute the rank of a stereo pair, StereoPairRank in this example, is a combination of the two ranks:
#  Now specify how the stereo quality is determined.
Equation = "imagestrengthrank * 1.0 + stereostrengthrank * 1.0"
Result = StereoPairRank
      
The StereoPairRank will contain an indication of the strength of a stereo pair. This keyword and image keywords are combined in a Resource made up of both Resources by merging the keywords from the Resources. The keywords from each Resource have a unique suffix appended to the name of the keyword before they are added. The GisOverlap keywords MergeSuffixA and MergeSuffixB may be used to specify keyword suffixes for each image as they are merged. MergeSuffixA is used to identify the primary stereo image, MergeSuffixB the secondary or matched stereo image. Defaults for merge suffix keywords are A and B respectively.

If data is provided by caminfo, then the values required by this Strategy are provided at the center image pixel coordinate. This may not provide accurate results, particularly for the StereoStrength group of keywords, because the values are likely not of the latitude/longitude within the overlapping region of the images. However, the computation of the StereoAngle could be more accurate if you were to take advantage of GisOverlap OverlapMerge options. The centroid (longitude/latitude) values for all of these options is computed and stored in new keywords GisOverlapCentroidX (longitude) and GisOverlapCentroidY (latitude) making this coordinate the same point in both stereo image pairs. By setting a parameter in the KeywordMap group as CenterLongitude = GisOverlapCentroidX and CenterLatitude = GisOverlapCentroidY the StereoAngle is computed from a common surface point. If desired, the user can specify whether to use the stereo angle in the computation of the VerticalPrecision stereo keyword rather than the ParallaxHeightRatio. Take note the equation to compute StereoPairRank can be changed as desired or weights can be provided in the fourth limit keyword value to emphasize or de-emphasize any limit.

StereoPair Strategy Object Keywords

Keyword Disposition Description
OverlapMinimum Optional Specifies the minimum common overlap ratio of the two geometries with respect to the primary Resource to be valid. If the computed ratio is less than this value, the Resource will not be deemed a valid overlap due to lack of common coverage. Defaults to 0.0.
OverlapMaximum Optional Specifies the maximum common overlap ratio of the two geometries with respect to the primary Resource to be valid. If the computed ratio is greater than this value, the Resource will not be deemed a valid overlap due to too much common coverage. Defaults to 1.0.
OverlapRatioKey Optional Provides the name of the keyword to retain the common overlap ratio. If not provided, it will be stored in the merged Resource as the keyword OverlapRatio.
Asset Optional Provides the name of the Asset created containing all the Resources determined to have overlap GIS geometries from the candidate list. If not provided, GisOverlap will be the default.
MergeSuffixA Optional To provide uniqueness of keywords that are common to both Resources when they are combined, a suffix is added to the primary Resource. Use this keyword to specify the keyword suffix. If not provided, A is the default.
MergeSuffixB Optional To provide uniqueness of keywords that are common to both Resources when they are combined, a suffix is added to the Resource that overlaps the primary Resource. Use this keyword to specify the keyword suffix of the overlapping Resource. If not provided, B is the default.
OverlapMerge Optional

GIS Operators

Users can determine how to merge the GIS geometries of overlapping Resources using the following options. Note the resulting geometry is stored in the combined Resource of the primary and overlapping Resources. Note also that two additional keywords are added to the combined Resources that provide the centroid coordinate of the resulting geometry. These keywords are GisOverlapCentroidX and GisOverlapCentroidY that provide the center coordinate of the geometry X (typically longitude) and Y (latitude). These become useful for computing common stereo data.
Parameter Description
Intersect Combines the two GIS footprints as an intersection. This will result in a GIS geometry that contains the area of common overlap.
Union Combines the two GIS footprints as a union. This will create a geometry made of the area of both Resource geometries.
Centroid This option creates a point geometry of the centroid computed from the intersection of the two geometries.
ResourceA Propagates the geometry of the primary Resource used in the overlap determination.
ResourceB Propagates the geometry of the matching/overlapping Resource used in the overlap determination. It is the Resource found in the list used to match the primary Resource.
None No geometry will be created. This is the default.
CandidateMiner Optional (Object) To select the Resource list used to evaluate stereo geometries, users have several ways to create the list. A complete isisminer configuration object called (and named) CandidateMiner can be provided that will be executed on the full Resource list to generate a list of candidate Resources for comparisons of stereo potential. This configuration must be a compliant, properly formatted isisminerobject with the only difference being the name of the object. It may also be provided in a file called CandidateConfigFile. Either a CandidateMiner object or configuration file may be provided, but not both. If no candidate options are provided, the current active list of Resources will be used.
CandidateConfigFile Optional (File) This option can be used in lieu of a CandidateMiner object to specify a set of Strategies to select stereo candidates. You may also provide arguments to that can be translated when executed to resolve the file name by providing a keyword called CandidateConfigFileArgs that contains keywords that translate occurrences of %1 to %n strings in the name of the file.
OverlapMiner Optional (Object) For efficiency purposes only, users can immediately process the stereo list of a given Resource at the time it is completely determined. Simply provide an isisminer configuration object called (and named) OverlapMiner that will be executed on the stereo Resource at the time each one is fully determined. This configuration must be a compliant, properly formatted IsisMiner object with the only difference being the name of the object. It may also be provided in a file called OverlapConfigFile. Either an OverlapMiner object or configuration file may be provided, but not both. If no overlap options are provided, no action is taken on the stereo list.
OverlapConfigFile Optional (File) This option can be used in lieu of a OverlapMiner object to specify a set of Strategies to process stereo Resource results. You may also provide arguments to that can be translated when executed to resolve the file name by providing a keyword called OverlapConfigFileArgs that contains keywords that translate occurrences of %1 to %n strings in the name of the file.
PixelPrecisionMatch Optional This parameter specifies the expected matching precision of image pairs. This value is used to compute the expected vertical precision of the stereo data. Specifically, it specifies the ability to match features in the stereo pairs, measured in pixels. If not provided, the default is 0.2.
ImageStrength Recommended (Group) This group contains keywords that define individual image parameter minimum, maximum, best and (optional) multiplier/scale value. The keyword names contained in this group may be mapped to other keywords using the KeywordMap group. (Mapped) Keywords specified in this group must exist in both the primary and matching stereo images. The Resource is not evaluated for stereo potential. The intended values here are used to screen any images that do not satisfy individual image parameters.

Individual Image Strength Values

This group may contain any set of keyword limits/ranks as long as they map (use KeywordMap group if necessary) to valid Resource keywords. Use this group to eliminate images that do not make good stereo candidates because of undesirable lighting or observation conditions. Here are the recommended values.
Parameter Description
Incidence Restriction of incidence angle. The incidence angle is the angle between the local surface normal vector (evaluated on a sphere) and the vector to the Sun. Values should be between 40 and 65 degrees (avoid emission shading at low end, shadows at high end).
Example: Incidence = (  30.0,  70.0,  50.0 )
Emission Restriction of emission angle. The emission angle is the angle between the spacecraft intercept point and the local surface normal vector (evaluated on a sphere). Values should be between 0 and 40 degrees (avoid foreshortening and surface hiding).
Example: Emission = (   0.0,  55.0,  15.0 )
Phase Optional restriction of phase angle. The emission angle is the angle between the spacecraft and the illumination source (typically the Sun). Values should be between 15 and 120 degrees. Values should be between 15 and 120 degrees.
Example: Phase = ( 15.0, 120.00, 30.0 )
Resolution Optional restriction of resolution differences in image pairs. The ground sample distance (GSD, in m/pixel) at the center or other representative point. Image resolution must be chosen based on the desired resolution of the output DTM. Because typical stereo matching methods do not produce independent height estimates over distances smaller than about 3 to 5 image pixels, the image GSD needs to be 3 to 5 times smaller than the desired DTM GSD. The recommended difference in resolution is 1/3 of target DTM ground sampling distance (pixel scale).
Example: Resolution  = ( 275.0, 500.0, 300.0 )
StereoStrength Required (Group) This group contains keywords that define stereo strength parameter minimum, maximum, best and (optional) multiplier/scale value. The keyword names contained in this group may be mapped to other keywords using the KeywordMap group. (Mapped) Keywords specified in this group are computed from the stereo algorithm and stored in the combined stereo image pair Resource constructed from the Resource pair. Here is a table of stereo values computed by the algorithm and the required keywords to compute each value.

Computed Stereo Strength Values

Stereo values computed and stored in the combined/merged stereo pair Resource. Examples included for some of these values are recommended according to the LPSC abstract cited earlier. If any of the minimum or maximum values are exceeded, the pair is discarded.
Parameter Description
ImageStrengthRank This value is computed as the average of the ImageStrength from the two images that make up the stereo pair. It can be used in combination with the StereoStrengthRank to determine the final best stereo potential (see the Equation keyword).
ParallaxHeightRatio The strength of a stereo pair is easily visualized as the angle between the emission vectors of the two images, provided that both vectors are relatively close to vertical. This parameter can be generalized (e.g., to cover cases in which both images are oblique) in terms of the parallax/height ratio. Physically, it represents the amount of parallax difference that would be measured between an object in the two images, for unit height. The keywords ParallaxX and ParallaxY are required in both pair Resources but can be mapped to different keywords in the KeywordMap group.
Example: ParallaxHeightRatio = ( 0.1, 1.0, 0.4 )
ShadowTipDistance Differences in illumination between two images reduce the accuracy of stereo matching. Compatibility of the images can be measured in terms of the Shadow-Tip Distance. This measurement is defined as the distance between the tips of the shadows in the two images for a hypothetical vertical post of unit height, and it is computed analogously to ParallaxHeightRatio. The keywords ShadowX and ShadowY are required in both pair Resources but can be mapped to different keywords in the KeywordMap group.
Example: ShadowTipDistance = ( 0.0, 1.0, 0.5 )
ResolutionRatio Computed as the highest resolution/lowest resolution. The keyword Resolution is required in both pair Resources but can be mapped to different keywords in the KeywordMap group.
Example: ResolutionRatio  = (  1.0,  2.5,  1.0 )
DeltaSolarAzimuth This optional value is computed as the angular difference in degrees of the SubSolarGroundAzimuth angles in each Resource. The keyword SubSolarGroundAzimuth is required in both pair Resources but can be mapped to different keywords in the KeywordMap group.
Example: DeltaSolarAzimuth    = (  0.0, 90.0,  45.0 )
DeltaSpacecraftAzimuth This optional value is computed as the angular difference in degrees of the SubSpacecraftGroundAzimuth angles in each Resource. The keyword SubSpacecraftGroundAzimuth is required in both pair Resources but can be mapped to different keywords in the KeywordMap group. The difference between this angle and the convergence angle is that this angle is measured from the center of the body and not a point on the surface as the StereoAngle.
Example: DeltaSpacecraftAzimuth  = (  0.0, 65.0,  25.0 )
StereoAngle Sometimes called the convergence angle, this optional value is computed as the angle between the two spacecraft to the surface in degrees. Note that if using the center image latitude/longitude coordinates, the ground points are typically not the same point on the surface and thus, not as reliable. However, if using the GisOverlapCentroidY and GisOverlapCentroidX values, where CenterLongitude and CenterLatitude can be mapped to these values in the KeywordMap group, this is a common point in both images and will produce more accurate results. The keywords CenterRadius, CenterLatitude, CenterLongitude, TargetCenterDistance, SubspacecraftLatitude and SubspacecraftLongitude are required in order to compute this value. However, all of these keywords can be mapped to different keywords in the KeywordMap group.
Example:  StereoAngle  = ( 10.0,  65.0,  25.0 )
UseStereoAngle Optional Specify what to use as the VerticalPrecision of a stereo pair. You can use the stereo angle (True) or the ParallaxHeightRatio. It may be more precise, particularly when using the GIS intersection of common area centroid parameters, to use the StereoAngle computed from the common latitude/longitude point rather than the center image pixel latitude/longitude of each image - which is likely not the same point but may be close enough. Defaults to True, to use the StereoAngle.
Equation or Equations (Group) Required This keyword or set of keywords (if using Equations group) are used to compute the strength (rank) of the stereo pair based upon any value in the combined pair Resource, but is typically applied to stereo ranking values. The resulting value of the calculation provided by the Calculator strategy is stored in the combined Resource as the value specified in the Result keyword, required when using the single keyword form or any keyword in the Equations group if you choose to use that method. This example equation is typically as follows:
#  Now specify how the stereo quality is determined. Based upon
#  ImageStrength and StereoStrength, the maximum (best) rank is 8.0.
Equation = "ImageStrengthRank * 1.0 + StereoStrengthRank * 1.0"
Result = StereoPairRank
          

The following is an example using the StereoPair strategy. In this example we are reading image data from an SQLite database containing WAC and NAC MESSENGER data created from caminfo runs. It determines stereo from overlapping images in the vicinity of Degas crater on Mercury.

Object = IsisMiner

  Name = DegasStereoPair
  RequiredParameters = ( inputdir, outputdir )

  Object = Strategy
    Name        = ReadDegasPerimeter
    Type        = CsvReader
    Description = "Read Degas data"

    CsvFile        = "%1/degaus_stereo_perimeter.csv"
    CsvFileArgs    = "inputdir"

     #Ignore the first several lines including the header.
    HasHeader      = True

    Identity = "%1"
    IdentityArgs = "SourceProductId"

  #  Can specify a GIS keyword that will be converted on input
    GisGeometryKey = GisFootprint
    GisType        = WKB
    RemoveGisKeywordAfterImport = False
  EndObject

  Object = Strategy
    Type = StereoPair
    Name = DegasStereoPairs
    Description = "Find all stereo pairs in the Degas crater region"
    OverlapMerge = None
    ShowProgress = True
    #Debug = True

    PixelPrecisionMatch  = 0.2
    Asset = StereoPairs
`
    #  Now specify how the stereo quality is determined.
    Equation = "imagestrengthrank * 1.0 + stereostrengthrank * 1.0"
    Result = StereoPairRank

    #  First level contraints/strengths
    Group = ImageStrength
      Incidence       = (  30.0,  65.0,  50.0 )
      Emission        = (   0.0,  55.0,  15.0 )
      Phase           = (  15.0, 120.00, 30.0 )
      Resolution      = ( 25.0, 5000.0, 200.0 )
    EndGroup

    #  Second level constraints/strengths
    Group = StereoStrength
      ResolutionRatio      = (  1.0,  2.5,  1.0 )
      ParallaxHeightRatio  = (  0.1,  1,    0.4 )
      ShadowTipDistance    = (  0.0,  2.58, 0.0 )
      OverlapRatio         = (  0.1,  1.0,  0.5 )
    EndGroup

    #  This shows the keyword mappings to values required for stereo
    #  matching and constraint values
    Group = KeywordMap
      Resolution = PixelResolution
      Emission   = EmissionAngle
      Incidence  = IncidenceAngle
      Phase      = PhaseAngle
    EndGroup

    #  Select candidates from input list.  Matches over itself
    Object = CandidateMiner
      Object = Strategy
        Name = PairCandidates
        Type = ResourceManager
        Description = "Choose all Resources to determine stereo"
        Operations = ResetDiscard
      EndObject
    EndObject
  EndObject

  Object = Strategy
    Type = AssetSidebar
    Name = WriteStereoFiles
    Description = "Process all stereo pair Assets"
    Asset = StereoPairs
      
    Object = IsisMiner

      Object = Strategy
        Name = SortStereoPairs
        Type = NumericalSort
        Description = "Sort stereo pairs by StereoPairRank"
        SortKey = StereoPairRank
        Order = Descending
      EndObject

      Object = Strategy
        Type = CsvWriter
        Name = WritePairs
        Description = "Write individual files with stereo pair data"

        SkipEmptyLists = True
        CsvFile = "%1/%2.csv"
        CsvFileArgs = ( "outputdir", "Identity" )
        Keywords = ("SourceProductIdA", "SourceProductIdB", "StereoPairRank",
                    "ImageStrengthRank", "StereoStrengthRank",
                    "ParallaxHeightRatio", "ShadowTipDistance", "ResolutionRatio", 
                    "DeltaSolarAzimuth", "DeltaSpacecraftAzimuth", 
                    "VerticalPrecision", "StereoAngle",
                    "ImageStrengthRankA", "ImageStrengthRankB",
                    "IncidenceRankA", "IncidenceRankB",
                    "EmissionRankA", "EmissionRankB",
                    "PhaseRankA", "PhaseRankB",
                    "OverlapRatioA", "OverlapRatioB",
                    "YearDoyA", "YearDoyB", 
                    "CenterLongitudeA", "CenterLatitudeA",
                    "CenterLongitudeB", "CenterLatitudeB",
                    "PixelResolutionA", "PixelResolutionB", 
                    "EmissionAngleA", "EmissionAngleB",
                    "IncidenceAngleA", "IncidenceAngleB",
                    "PhaseAngleA", "PhaseAngleB")
      EndObject
    EndObject
  EndObject
EndObject
End

      

Return to List of Strategy Types


Categories


History

Kris Becker2013-02-19 Original Version
Kris Becker2015-02-24 Update/complete documentation and regression tests
Jeffrey Covington2015-02-26 Added documentation for the IsNumeric strategy
Ian Humphrey2015-03-03 Added documentation and tests for the NumericalSort strategy
Makayla Shepherd2015-03-04 Added documentation for the Limit strategy
Jeffrey Covington2015-03-04 Added documentation for the CsvWriter strategy
Jeffrey Covington2015-03-06 Added documentation for the CsvReader strategy
Makayla Shepherd2015-03-13 Added documentation for PdsTableFormat strategy
Ian Humphrey2015-03-16 Added documentation for CnetReader strategy
Kristin Berry2015-03-23 Added documentation and tests for the ResourceManager strategy
Ian Humphrey2015-03-24 Added documentation for PvlReader strategy
Jeannie Backer2015-03-27 Added a List of Strategies table of contents with links to each individual description. Added links from individual Strategy descriptions back to table of contents.
Makayla Shepherd2015-03-27 Added documentation for PdsTableCreator strategy. Fixed documentation for PdsTableFormat strategy.
Jeffrey Covington2015-03-31 Added documentation for the Calculator strategy
Kristin Berry2015-04-07 Merged GisBasic strategy with GisIntersect strategy. Added documentation and tests for GisIntersect strategy. Modified Strategy:::applyToIntersectedGeometry in Strategy.cpp to deal with the case where no Resources are active.
Jeffrey Covington2015-04-09 Added documentation for the Sidebar strategy
Jeffrey Covington2015-04-16 Updated documentation for the CSV strategies.
Kris Becker2015-04-27 Added GisOverlap strategy and documentation for it; added ShowProgress keywords to allow users to selectively show the progress status percentage/bar for any Strategy that contains this keyword; updated Resource and Strategy code with new additions/modifications.
Jeffrey Covington2015-04-28 Added documentation for the Database strategy.
Kris Becker2015-04-29 Renamed the Database strategy to DatabaseReader
Andrew Stebenne2015-05-01 Added documentation and app tests for the GisUnion strategy.
Jeffrey Covington2015-05-01 Updated the DatabaseReader strategy documentation.
Ian Humphrey2015-05-01 Added documentation for the AssetSidebar strategy
Makayla Shepherd2015-05-21 Added keyword format option (KeywordFormatFile) to PdsTableFormat strategy.
Kris Becker2015-05-25 Added the ability to code Strategy objects using plugins and added the StereoPair and RunCommand (plugin) Strategies.
Kris Becker2015-06-05 Updated Calculator strategy to include all global parameters in variable resolution; fixed StereoPair computation of stereo angle; implemented check in StrategyFactory to validate RequiredParameters are define in the global variable pool and updated app test configuration files to use this feature.
Kris Becker2015-06-12 Improved GisOverlap and StereoPair strategies. StereoAngle can now be computed from the GIS centroid of the overlapping region, providing a much higher degree of accuracy. Improved global variable pool management whilst traversing through Strategy/Resource depths in the mining process. Ensure Asset is cleared when Mode = Create is used in AssetSideBar strategy.
Kris Becker2015-06-17 Fixed bug in creation of GEOS SRTree when only one geometry occurs (requires two or more). Changed name of IsisMiner objects within the GisOverlap strategy (since we now have two of them) to CandidateMiner and OverlapMiner. Added feature to simplify geometry whilst preserving topology using GEOS' Douglas-Peucker algorithm when reading in geometries. It is automatically detected in Strategies like CsvReader and DatabaseReader. Added many new parameters to various Strategies.
Kris Becker2015-06-28 Add ability to process the set of GisOverlap results for each Resource as they are matched. This is handled in the OverlapMiner Strategy object.
Kris Becker2015-07-07 Added more content to documentation and added an example demonstrating a real application of the isisminer application.
Kris Becker2015-09-27 Refactored the Resource class to contain all but the active status so that copies can be maintained in separate instances. Reworked the AssetSidebar strategy to take advantage of this work.
Kris Becker2015-10-11 Fixed bug in argument scanning when the number of arguments reached 10 or more as %1 was replacing %10, etc... Scan and replace i reverse order corrected this problem. Enhanced the Calculator strategy to provide argument replacement in the Initializers group.
Kris Becker2015-11-01 Modified CsvReader strategy to have the provided header length determine the number of keywords created/Resource. This allows the jigsaw residual file to read the rejected column.

Parameter Groups

Inputs

Name Description
CONFIG Configuration of Strategies that are to be executed.
GLOBALS Optional PVL file containing parameters that are loaded into global variable pool.
PARAMETERS Optional user global parameters.
X

Inputs: CONFIG


Description

This file contains a series of Strategy objects enclosed in an object named IsisMiner. The main documentation contains many different examples of the format of this file.

Type filename
File Mode input
Filter *.conf
Close Window
X

Inputs: GLOBALS


Description

All keywords contained in this file are loaded in the global variable pool. Unique keywords that do not exist in Resources can be used as keyword value substitutions in your Strategies.

Type filename
File Mode input
Default None
Close Window
X

Inputs: PARAMETERS


Description

This parameter allows the user to add global parameters to each Strategy so that they can be used in translation of keyword ARG values. This parameter has the form:

PARAMETERS = "key1:value1[@key2:value2...]"
Defaults to None.

Type string
Default None
Close Window

Example 1

From a global image dataset, find all stereo pairs within a 10x10 degree map section

Description

This example has been used to process MESSENGER MDIS global image dataset. The purpose of this isisminer run is to

  1. identify all images whose center pixel longitude/latitude coordinate are contained within a defined 10x10 degree map section,
  2. from that list, find all images that overlap each satisfying a specified overlap ratio, and
  3. determine all images that satisfy stereo conditions for each image in the map section.
The design of isisminer is intended to do to all these things in a single run. But do not try it on the complete dataset with hundreds of thousands of images; you have been warned. We have partitioned the Mercury global equatorial map in 10x10 degree sections resulting in 648 small quads and created small files that define each section. Of the approximately 274,000 images, this provides a more reasonable set of images to find stereo pairs/sets of about 300-400 pairs per section.

File naming conventions used for these section definition files are EquiYYYHXXXED10 where

  • YYY is the center latitude of the section,
  • H is the hemisphere (i.e. H = N for northern or H = S for southern),
  • XXX is the center longitude of the section,
  • E indicates positive east longitude, and
  • D10 indicates the spacing is 10 degrees in both latitude and longitude.

The section in this example includes Degas crater located at 37.08 degrees latitude and 232.75 positive east longitude. This falls in the section defined in the file Equi035N235ED10.def. Here is the contents of this file:

# Define sub region in 10 degree increments below the NP
# Define lower left and upper right coordinates
Region = "Equi035N235ED10"

RegionGisGeometry = "LINESTRING(230 30, 240 40)"
RegionBoundingBox = True

RegionCenterLongitude     = 235
RegionCenterLatitude      = 35

RegionLowerLeftLongitude  = 230
RegionLowerLeftLatitude   = 30

RegionUpperLeftLongitude  = 230
RegionUpperLeftLatitude   = 40

RegionUpperRightLongitude = 240
RegionUpperRightLatitude  = 40

RegionLowerRightLongitude = 240
RegionLowerRightLatitude  = 30

End
          
This file is provided in the GLOBALS parameter to isisminer. This will make all keywords in the file available for use in keyword argument substitution, CSV output, and Calculator and Limit Strategies.

To process the global image dataset requires treating data at the poles slightly different when identifying stereo. The incidence angles are consistently high, up to 90 degrees. This requires a slight change to the stereo configuration so two files are required to process the data - one for equatorial data between 65N and 65S degrees and one for polar from 65N/S to 90 N/S degrees. The isisminer configuration file must consider this when processing stereo and it is driven by the map section input file as to which one is selected. The easiest way to do this is to provide the name of the stereo configuration file as a PARAMETER to isisminer. The main isisminer CONFIG file is named mdis_overlaps_and_stereo.conf and its contents with comments is shown in the mdis_overlaps_and_stereo.conf file below.

This configuration file uses the GISOverlap strategy to identify all images that overlap one another and creates a CSV file of those images and a lot of relevant data for computing stereo. The stereo image pairings/sets are determined using a special processing feature of GisOverlaps. It has a provision for immediately processing the result of the overlap determination for each image. Just so happens that the output CSV file contains all the information needed to compute stereo. As each image overlap set is determined and written to the output CSV file, GisOverlap runs the stereo configuration file on that CSV file using the StereoPair strategy. Its configuration is contained in the file mdis_global_stereo_equatorial.conf and is provided in the command line to isisminer in PARAMETERS using the StereoConfig parameter. The contents of mdis_global_stereo_equatorial.conf is shown below.

Finally, there is a complete configuration that has command line control depending upon which latitude region is being processed and input/output directories. To process the entire global dataset of MDIS images collected by MESSENGER, 267,769 of them, will require vast computing Resources. It is highly recommended that a processing cluster be used if available. Below is the command line used to process this map section.

The results summarized here show the output of the command and a summary of the results below.

Creating Strategies...
Finished creating 24 strategies...

Running DatabaseReader::MDISSelectRegionOct2014 (TimeIn:  02:46:51.626)
Description: Strategy::MDISSelectRegionOct2014 is running a DatabaseReader algorithm.
248586 of 0 processed in DatabaseReader::MDISSelectRegionOct2014 (TimeOut: 03:18:36.966)
ElapsedTime(s): 1905

Running DatabaseReader::MDISSelectRegionMay2015 (TimeIn:  03:18:36.966)
Description: Strategy::MDISSelectRegionMay2015 is running a DatabaseReader algorithm.
19183 of 0 processed in DatabaseReader::MDISSelectRegionMay2015 (TimeOut: 03:20:59.204)
ElapsedTime(s): 142

Running IsNumeric::FindBadOnes (TimeIn:  03:20:59.204)
Description: Identify images with invalid center pixel geometry
267474 of 267769 processed in IsNumeric::FindBadOnes (TimeOut: 03:21:00.209)
ElapsedTime(s): 0

Running ResourceManager::DeleteBadOnes (TimeIn:  03:21:00.209)
Description: Delete Resources with bad values in required keywords
295 of 0 processed in ResourceManager::DeleteBadOnes (TimeOut: 03:21:00.247)
ElapsedTime(s): 0

Running Limit::SelectRegion (TimeIn:  03:21:00.247)
Description: Find all MDIS images with center image geometry in the defined region
557 of 267474 processed in Limit::SelectRegion (TimeOut: 03:21:20.489)
ElapsedTime(s): 20

Running RunCommand::MakeOutputDirectory (TimeIn:  03:21:20.489)
Description: Ensure output directories exists
Running Mkdir1 PRE command: mkdir -p /scratch/kbecker/MESSENGER/Region10D/Global/Images
Running Mkdir2 PRE command: mkdir -p /scratch/kbecker/MESSENGER/Region10D/Global/Overlaps/Equi035N235ED10
2 of 0 processed in RunCommand::MakeOutputDirectory (TimeOut: 03:21:20.976)
ElapsedTime(s): 0

Running CsvWriter::WriteRegionImages (TimeIn:  03:21:20.976)
Description: Write images that are in the defined region to a file
557 of 557 processed in CsvWriter::WriteRegionImages (TimeOut: 03:21:21.398)
ElapsedTime(s): 0

Running GisOverlap::ImageOverlap (TimeIn:  03:21:21.398)
Description: Find all overlaps images in specfied region
557 of 557 processed in GisOverlap::ImageOverlap (TimeOut: 04:42:28.181)
ElapsedTime(s): 4866

Session complete in 01:55:36.555 of elapsed time
          
This run completed in about 2 hours. This is typical for most regions when run on machines with similar CPU processing power. There were 557 images identified in this region and the results are stored in /scratch/kbecker/MESSENGER/Region10D/Global/ImagesEqui035N235ED10_images.csv. For each image, and average of several hundred images were determined to overlap. There are now 557 files in the directory /scratch/kbecker/MESSENGER/Region10D/Global/Overlaps/Equi035N235ED10 that have a base name as the image name and _overlaps (example: EW1067331895I_overlaps.csv has 213 overlapping images) with a total of 230,389 image overlaps found. In the directory /scratch/kbecker/MESSENGER/Region10D/Global/Stereo/Equi035N235ED10, there are 308 files (249 had no stereo images or did not pass level1 image constraints for stereo) with similar file naming conventions used in the Overlaps directory (example: EW1067331895I_stereopairs.csv has 11 stereo images) where a total of 7,497 stereo images were identified.

Command Line

isisminer config=/scratch/kbecker/MESSENGER/Region10D/Global/mdis_overlaps_and_stereo.conf parameters='DBDir:/work/projects/messenger/DB@outputdir:/scratch/kbecker/MESSENGER/Region10D/Global@stereoconfig:/scratch/kbecker/MESSENGER/Region10D/Global/mdis_global_stereo_equatorial.conf' globals=/scratch/kbecker/MESSENGER/Stereo/Region10D/Deffiles/Equi035N235ED10.def
This command executes isisminer with the required parameters for our example.

Data Files

Links open in a new window.
Global map region section definition file This file contains an example of the keywords used to describe a small section of Mercury as defined in equirectangular map coordinates. These parameters are contained in a file called Equi035N235ED10.def. This file is provided in the GLOBALS parameters making the contents available in the global variable pool. The latitude/longitude boundaries in the file are used to find images where the center latitude/longitude coordinates are contained in the region.
CONFIG file for identifying images that overlap an image The contents of this file contain a sequence of Strategy object definitions used to process the global MESSENGER MDIS image dataset. The objectives of this configuration is to
  1. identify all images in a given map section
  2. from the global image dataset, find all images that intersect the regional images
  3. from overlapping images, find all images pairs that satisfy stereo parameters (defined in the mdis_global_stereo_equatorial.conf)
Results of all objectives are written to CSV files for future processing and/or refinements.
Sidebar configuration to process stereo This file contains a sequence of Strategy object definitions used to process overlapping images for stereo potential. This configuration is designed to be run independently or as the GisOverlap optional set of algorithms. The input source is a file containing individual image overlaps. This overlap file is indicated by the user with the input parameter called PARAMETERS.