ISIS Application Documentation
isisminer | Standard View | TOC | Home |
Run a series of algorithms (Strategies) that perform various operations on input sources (Resources)
Description
Categories
Groups
Examples
History
The isisminer application assists in the identification, manipulation, and output of data from a variety of data sources. One main source of input is caminfo. Processing in the isisminer program is controlled by the contents of a PVL-based, object-structured, configuration file provided in the CONFIG parameter. There are two main PVL objects used in the configuration of an isisminer program run: the IsisMiner object and the Strategy object.
The IsisMiner object contains a series of Strategy objects. Strategy objects are associated with a defined algorithm that performs some operation on data called a Resource. Strategy objects contain parameters that are needed by the algorithm to process Resource data. The order of Strategy objects is important as the algorithms are executed in the order they occur in the IsisMiner object. Each Strategy object must be associated with a known Strategy Type. Other objects may exist anywhere in the CONFIG file but are ignored when parsing Strategy objects unless they are part of the algorithm. One common use for other objects in the CONFIG file is to include database profiles that are used to provide access information for the DatabaseReader strategy (see below).
Strategy algorithms operate on a Resource. A Resource is a data container that stores a set of PVL keyword/values that can have data storage extensions (called Assets and GisGeometries). These Assets can contain anything created by Strategy objects for further analysis or processing and could be unique to a set of Strategies. Resources have three major components: keyword/values, GIS geometries and Assets. Most of the data manipulations occur on keyword/value elements. A Resource may contain a GIS geometry that is used in algorithms that work on geometric regions using GIS operators (GIS Intersect, GIS Overlap, GIS Union, StereoPair ). Currently, GIS geometries must be in well-known-text (WKT) or well-known-binary (WKB) as conversions are restricted to text representations of geometries (format requirements are the same as functions like ST_GeomFromWKT and ST_GeomFromWKB, respectively). Resources may also contain Assets. Assets are typically another list of Resources created from Strategies that are attached to a (parent) Resource. If Assets are represented as Resource lists, they can be processed by almost all Strategy algorithms using the AssetSidebar strategy (see below).
Resources can come from many sources. One of the most common input sources are databases. SQL queries can be specified directly in the DatabaseReader strategy. Each row returned when the query is executed is converted into a Resource. Other data sources are comma separated values (CSV) files (using the CsvReader strategy) where each row is a Resource and each column is a keyword/value. The keyword name in the case of a CSV file is the name of the column as specified in the header. Users can also read in ISIS control networks and process each Control Measure as a Resource. And, of course, ISIS cube files and other PVL file formats are also input sources (see PvlReader strategy). Strategies are designed to operate on a single Resource. This allows for a very narrow scope of design considerations to make coding Strategies quite easy.
Some very clever and sophisticated Strategy configurations can be created but to make them generally useful, the user may need to change some parameters within Strategies, such as output directories or file names that can be modified at runtime without having to edit the CONFIG files directly. To allow some control options, there are two optional application parameters provided: GLOBALS and PARAMETERS.
The GLOBALS parameter can specify the name of a file that contains PVL keywords and values. The complete contents of all the keywords found in the file, regardless of Object/Group structure are loaded immediately when isisminer first runs. The keywords contained in this file can be used as substitutions in all or part of many keyword value specifications used within most Strategies (described below).
The PARAMETERS input parameter is provided so that users can create simple keyword/value pairs on the command line that can be used in substitutions of part of all keyword values used in Strategies in the same manner GLOBALS keywords are substituted. For any keyword that is contained in both the GLOBALS and PARAMETERS keyword set, keywords in PARAMETERS always take precedence. This allows users to override keywords in GLOBALS as needed during keyword lookup and replacement operations to fully determine values of keywords in real time when Strategies are executed. Since equal signs are not allowed in ISIS application parameters, keyword names and values are separated by a colon. Multiple keywords can be specified and are separated by the @ symbol. For example, PARAMETERS="inputdir:$PWD@outputdir:$PWD/Results" will create two keywords, inputdir and outputdir, with inputdir set to the full path of the current directory running the job and outputdir set to a subdirectory in the current directory called Results.
Essentially, the GLOBALS and PARAMETERS elements creates a hierarchical lookup for keyword substitutions where PARAMETERS are searched first, then GLOBALS, for argument substitution of elements within a keyword value. At times, when individual Resources drive output operations (e.g., AssetSidebars), the individual Resource keywords become available for substitution as well and are the first set of keywords searched before PARAMETERS and GLOBALS. Argument substitutions are specified using the %n symbol in keyword values. Here, the n is a number from 1 to the number of values corresponding to an associated keyword with the same name but with Args appended to it. The nth value of this keyword is itself a keyword where its value is substituted in the corresponding %n occurrence in the original keyword. For example, in the CSVReader strategy, the name of a file to read the contents from is specified in the CsvFile keyword. To allow arbitrary specification of the input file without editing the CONFIG file to change it for each use of the CONFIG file, the PARAMETERS option can be used to specify the path to a named file or the entire file name by creating a keyword called CsvFileArgs. This is the general form for creating a keyword value substitution that uses two associated keywords with the first being any keyword and the second keyword having Args appended to the first keyword which contains other keywords that will be used to substitute arguments with.
See the complete stereo example provided for isisminer that demonstrates many of the concepts discussed here and in the Strategy descriptions below. It will also provide output and describe the results of a MESSENGER MDIS stereo processing run.
The following section describes all the Strategy algorithms currently available in IsisMiner. Strategies iterate through all active Resources. One of the features in isisminer that provides a lot of flexibility in the system is that Resources are never deleted unless users provide explicit instructions to do so (see ResourceManager). Each Resource has an inherent state that indicates if it is active, meaning it is available for processing, or inactive. Strategies like the Limit or Filter will evaluate values of keywords for boolean equations or strings, respectively, where true conditions keep Resource states as active, and false conditions set their states to inactive. Inactive Resources are not processed in subsequent Strategies unless explicitly reactivated (again using the ResourceManager). Strategies typically only use active Resources when normally processed. Exceptional is when a Strategy provides options to select an additional processing set of Resources (see GisOverlap, StereoPair and Sidebar). Those Resources accept a complete isisminer configuration where the object specification can be embedded within the Strategy object or a file name can be provided that contains a complete isisminer configuration (much like provided in CONFIG). This design allows users to progressively select Resources for processing and reactivate the entire list at any point in the execution of Strategies. The Strategies below provide the user with many options to help identify only data with specified/desired characteristics for evaluation and subsequent processing.
There are some keywords available that can be used in every Strategy beyond their unique keywords described below. Many of these keywords provide additional processing control or options that produce information in the form of text to the standard output stream. The following tables provide descriptions of the keywords that can be used in all Strategies. Some will apply only under certain conditions which are described in the context of the descriptions. All of these keywords are optional except where indicated.
Some miscellaneous keywords are available that provide general information and/or assist users in debugging Strategies that appear to result in unexpected behavior. Here are other keywords that users can set and are available in all Strategies.
Keyword | Disposition | Description |
---|---|---|
Program | Constant | Name of the program (isisminer). |
Version | Constant | A text string that provides the version of the application. |
Revision | Constant | The revision number of the program as provided in the source code repository. |
Runtime | Constant | A string that provides the time that the application was started. It is of the form YYYY-MM-DDThh:mm:ss. |
CONFIG | Constant | Provides the value (name of the file) given by the user in the CONFIG parameter. |
PARAMETERS | Constant | Provides the value (keyword/value) given by the user in the PARAMETERS parameter. |
GLOBALS | Constant | Provides the value (name of the file) given by the user in the GLOBALS parameter. |
ApplyToDiscarded | Optional | When True, all Resources regardless of active or discard state, are processed in the current Strategy. This is handy when you want all Resources to be processed and would otherwise require use of a Sidebar strategy to set states for all Resources. |
ShowProgress | Optional | When True, a progress bar as seen in ISIS applications is created and updated as each Resource is processed in a Strategy. This keyword can be set in any Strategy definition. This is useful to ensure processing is occurring in long Strategy algorithms. |
Debug | Optional | If set to True in any Strategy definition, information in the form of print statements are sent to standard output stream that report certain information about processing that is occurring in a Strategy. This can result in a large amount of information that developers use to determine Strategy behavior. The reported form and content will vary widely and is solely at the discretion of the developer. |
The following table describes keywords that pertain to manipulation of GIS geometries. They are available to all Strategies that use GIS geometries but only apply where indicated.
Keyword | Disposition | Description |
---|---|---|
GisGeometry | Optional | Specifies the source of the geometry. The value of this keyword varies according to the type of GIS geometry being processed. The geometry type is specified in the GisType keyword. |
GisGeometryArgs | Optional | Allows user to provide keywords that are used as substitution values in the GisGeometry keyword. The value in GisGeometry may contain strings of the form %n where n corresponds to the array index, starting from 1, of the values in the GisGeometryArgs keyword. Each value in the GisGeometryArgs array of values is expected to name a valid keyword that can be resolved using the hierarchical parameter search. |
GisGeometryKey | Optional | This is a shorthand method to specify a keyword in the Resource which fully contains the GisGeometry. It will override any value specified in GisGeometry and must name a valid Resource keyword containing a GIS geometry. GisGeometryArgs can be used to substitute arguments that may be contained in the value retrieved in the keyword specified by GisGeometryKey. |
GisType | Required | Specifies the type of GIS geometry to be processed (converted from text form). Valid types are WKT, WKB, CUBE or ISISCUBE. |
RemoveGisKeywordAfterImport | Optional | If set to True, then the keyword referenced in GisGeometryKey is deleted from the Resource after its value is retrieved for processing. This is an attempt to reduce memory if this value is no longer needed as these strings can be quite large. |
GisSimplifyTolerance | Optional | This keyword, if present, specifies the tolerance in units of the GIS geometry (typically in units of degrees) to use when applying a Douglas-Peucker algorithm to simplify an imported geometry after conversion from text. The tolerance in degrees can be converted to pixels by using a simple formula of circumference at the equator. For example, for Mercury the radius is 2440 km. At the equator the circumference is 2 * pi * 2440 = 15,331 km. Each degree of contains 15,331/360 = 42 km. If GisSimplifyTolerance = 0.005, the tolerance is ~0.212 km (212 meters) at the equator. This tolerance was chosen because our target shape model/map is ~200 meters/pixel. In a set of three consecutive points along the boundary, the minimum distance from the middle point of the set to the line segment that connects the first and third points is computed. If this distance is less than the tolerance, then these boundary points are eliminated. |
GisGeometryPointsKey | Optional | If provided when GisSimplifyTolerance is specified, this is the base name of a set of keywords that describe the simplified geometry. Three keywords are created from the GIS geometry simplification process. If the value of GisGeometryPointsKey = GisSimplePoints, then the keywords GisSimplePoints contains the total number of points after simplification. GisSimplePointsOriginal contains the total number of points prior to simplification and GisSimplePointsTolerance is the value of GisSimplifyTolerance. These keywords are written to each Resource from which GIS geometries are converted from text. |
GisMethod | Optional | When applying geometry operators in Strategies, the GEOS system provides a very efficient, geometry-based, RTree search algorithm to make multiple search queries very fast. If the search is limited to one geometry operating on all active Resource geometries, then creating the RTree is unnecessary and the operation can be applied to every geometry. This will be faster than creating the RTree from all Resource geometries and then applying the RTree search for a single geometry. Set the value of this keyword to direct if you want to use direct GIS operators. Or use rtree for use of the RTree search. Defaults to direct. This value does not apply to GisOverlap or StereoPair. |
This Strategy can be used to add Assets to any active Resource. By providing the Operation keyword with a value of Create, the user can then process an Asset to be added to the active Resources (with the PvlReader or CsvReader strategies, for example). Note that when using create as a value for the Operation keyword, the specified Strategy in the AssetSidebar strategy will not be applied to any existing Assets contained in the active Resources. In fact, any Asset that exists in a Resource of the same name will be deleted and replaced with the results of the AssetSidebar processing. Note that some Strategies create Assets directly in the course of executing its algorithm and do not require an AssetSidebar to create them.
When creating Assets, users have several options to create the Resource list for the Asset. Using the CreateSource keyword with a value of Copy, a copy of all Resources in the input list is generated prior to executing the Strategies to process the Assets. The active/deactive status of each new Resource in this list is managed independently from the list outside of the AssetSidebar processing it. However, keywords, other Assets, and the GIS Geometry remain identical to those of the parent Resource. To create a completely independent copy of the Resources, set CreateSource to Clone. This option clones all the status and data of each Resource and creates a completely new Resource. To start from scratch and create an Asset by other means, use None. Defaults to Copy the Resource list.
This Strategy may also be used to apply a specified Strategy to any active Resource's Assets. For Operation set to Process, the specified Strategy in the AssetSidebar will be applied to the active Resources' Assets and replace the processed Asset in active Resources as new Assets. This mode is particularly useful for processing Assets created directly by other Strategies.
Keyword | Disposition | Description | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
Name | Required | User sets the name of the AssetSidebar strategy. | ||||||||
Type | Required | Identifies the AssetSidebar strategy - must be AssetSidebar or it will not be recognized as a valid Strategy by isisminer. | ||||||||
Description | Optional | User provides a description of the AssetSidebar strategy. | ||||||||
Asset | Required | Specify the name of the Asset to create or append to the active Resource. | ||||||||
Operation | Optional | Specifies how Assets will be processed (see the AssetSidebar description). Valid values are process and create. Use process if you want to process an existing list and preserve the resulting contents. In this mode, you can delete them all after processing the Resource lists that exist for each Resource (a list of Resources attached to a Resource). | ||||||||
CreateSource | Optional |
Asset Creation Options
|
||||||||
IsisMiner | Optional | PVL object containing an IsisMiner definition to run on the Assets. Either this or StrategyConfigFile must be specified. | ||||||||
StrategyConfigFile | Optional | The CONFIG file to run on the Assets. Either this or an IsisMiner object must be specified. | ||||||||
StrategyConfigFileArgs | Optional | Replace placeholders in StrategyConfigFile of the form %1, %2, etc. up to %99 with the arguments specified. Each placeholder is replaced by its corresponding argument in order. |
The following is an example of an AssetSidebar strategy
Object = Strategy Name = TestCreateAsset Type = AssetSidebar Asset = AssetA Operation = Create CreateSource = None Description = "Adds AssetA (Assets read from test.csv) to the active Resources" Object = IsisMiner Object = Strategy Name = ReadAssetTest Type = CsvReader CsvFile = "test.csv" HasHeader = True SkipLines = 0 IgnoreComments = False Delimiter = ", " EndObject EndObject EndObject
Return to List of Strategy Types
There are two ways equations can be specified. The basic method requires two keywords to compute, execute, and store the results in each Resource. The Equation keyword stores the complete equation to execute for each Resource and uses the Result keyword value to name the keyword to store the result of the equation.
An alternative form uses a PVL Group named Equations which contains a sequence of keywords whose values are equations. A keyword of the same name is created in each Resource with the value of each successfully computed equation. Values of previous equations can be used in subsequent equations. No other keyword may exist in the Equations group as every keyword is expected to be a valid equation.
Currently this Strategy supports scalars in the form of integers and doubles. In addition, it supports the circle constant pi and Euler's constant e. Functions, such as sin and cos, are followed by parentheses and a list of arguments. Operators such as * and + are placed in between their arguments. If a token is not recognized as a scalar, operator, or function, then it is assumed to be a keyword, and will be replaced by the corresponding value in the Resource.
All trigonometric functions expect angles in radians, not degrees so use the rads() function!
The following table shows all currently supported operators and functions sorted by precedence (0 is the highest precedence). Functions have a precedence of 1 and operators have a precedence of 2 or higher, according to order of operations. All examples are valid equations using 'x' and 'y' as variables:
Precedence Level |
Operator | Description | Example |
---|---|---|---|
0 | { [ ( ) ] } | Parentheses, brackets, or braces | y*(x+[30-{x/pi}]) |
1 | -- or neg | Negative sign | --x + y or neg(x) + y |
1 | abs | Absolute value | abs(y - x) |
1 | min | Minimum | x + min(5, y) + min(y, z) |
1 | max | Maximum | 20 * max(x,y) |
1 | sin | Sine | x * sin(123/321) |
1 | cos | Cosine | cos(.02*50) |
1 | tan | Tangent | tan(x/y) |
1 | csc | Cosecant | 12.3 + csc(x) |
1 | sec | Secant | sin(pi/60) + (sec(y))^2 |
1 | cot | Cotangent | x + cot(x) - 42 |
1 | asin | Arcsine | 0.006 ^ asin(x*5) |
1 | acos | Arccosine | acos(1/[2*pi]) |
1 | atan | Arctangent | atan(x/e) |
1 | atan2 | Arctangent2 | atan2(--10) |
1 | sinh | Hyperbolic sine | 55 + sinh(y) |
1 | cosh | Hyperbolic cosine | cosh(x^pi) |
1 | tanh | Hyperbolic tangent | tanh(x) |
1 | log or ln | Natural log | ln(abs(1/[y-x])) |
1 | log10 | Log base 10 | 99 + log10(x-160) |
1 | sqrt | Square root | sqrt(abs[1000 - y]) |
1 | rads | Convert degrees to radians | x / cos(rads(x)) |
1 | degs | Convert radians to degrees | degs(acos(x)) |
1 | and | Bitwise and | and(x, y) |
1 | or | Bitwise or | or(x, y) |
1 | mod | Modulus (Note: the arguments are rounded to the nearest integer) | mod(x, y) |
1 | fmod | Float modulus, remainder of division | fmod(x, y) |
2 | ^ | Exponentiation | x ^ 3 |
3 | * | Multiplication | 10 * x |
3 | / | Division | y / x |
4 | << | Bitwise left shift | x << 4 |
4 | >> | Bitwise right shift | y + (x >> 3) |
5 | + | Addition | 123 + 0.004 + x |
5 | - | Subtraction | 10 - (--x) |
6 | & | Bitwise and | x & y |
6 | | | Bitwise or | x | y |
6 | && | Logical and | x && y |
6 | || | Logical or | x || y |
6 | % | Float modulus, remainder of division | x % y |
6 | > | Greater than (Note: result is 0 or 1) | x > y |
6 | < | Less than (Note: result is 0 or 1) | x * (x < y) |
6 | <= | Less than or equal (Note: result is 0 or 1) | x <= 0.505 |
6 | >= | Greater than or equal (Note: result is 0 or 1) | x * (x >= 101) |
6 | == | Equal to (Note: result is 0 or 1) | x == (y/2) |
6 | != | Not equal to (Note: result is 0 or 1) | x != y |
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the Calculator strategy. |
Type | Required | Identifies the Calculator strategy - must be Calculator or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the Calculator strategy. |
Initializers | Optional | PVL Group containing Resource keywords to be initialized with initial values. The PVL names are the keywords and the values are the initializers. In some cases, results need to be initialized if they do not already exist in the Resources. |
Equation | Optional | A single equation to be stored in Result. If this and Equations is specified, this equation will be executed and stored first. |
Result | Optional | The keyword where the result of Equation will be stored. |
Equations | Optional | The PVL Group where all keyword values are an equation. The names of the keywords containing the equations in the Equations group are used to store the results of the calculation in each active Resource as a new keyword. If Equation is specified, these equations will be executed after the equation specified in the Equation keyword value. |
The following is an example of a Calculator strategy:
Object = Strategy Name = Trigonometry Type = Calculator Description = "Calculate trigonometric functions of angle." Group = Initializers Sine = 0 Cosine = 0 Tangent = 0 EndGroup Group = Equations Sine = "sin(angle)" Cosine = "cos(angle)" Tangent = "tan(angle)" EndGroup EndObject
Return to List of Strategy Types
If the Identity keyword is not provided or its value is an empty string, each Resource is named with a unique number (starting at 0).
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the CnetReader strategy. |
Type | Required | Identifies the CnetReader strategy - must be CnetReader or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the CnetReader strategy. |
CnetFile | Required | User provided control network file. |
CnetFileArgs | Optional | Used for providing any arguments to be substituted in CnetFile's value. |
Identity | Optional | The name given to each Resource. Intended to be used with the IdentityArgs parameter to give each Resource a unique name. If Identity does not have a value, the Resource names are set as unique serial numbers starting at 0. |
IdentityArgs | Optional | Replace placeholders in Identity of the form %1, %2,..., %99, with the arguments specified. Each placeholder is replaced by its corresponding argument in order. The arguments are assumed to be keywords. |
The following is an example of a CnetReader strategy:
Object = Strategy Name = ControlNetworkReader Type = CnetReader CnetFile = "%1/Alph_VIS.net" CnetFileArgs = "inputdir" Identity = "%1_%2" IdentityArgs = (PointId, SerialNumber) Description = "Read from a control network file" EndObject
Return to List of Strategy Types
The CSV file must have the following properties. The rows must be delimited by a new line. The columns must be delimited by a single character. The number of columns in each row must be the same for all rows. If IgnoreComments is set to True, then Comments must be on their own line and be preceded by a #.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the CsvReader strategy. |
Type | Required | Identifies the CsvReader strategy - must be CsvReader or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the CsvReader strategy. |
CsvFile | Required | The CSV file that the Resources will be created from. The rows must be delimited by a new line. The columns must be delimited by a single character. |
CsvFileArgs | Optional | Replace arguments in CsvFile of the form %1, %2, etc. up to %99 with the values of keywords specified in this array of keywords. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
HasHeader | Optional | If True, then the first row of the CSV file will be assumed to be the header and used to create keywords of column header names in each Resource (row). If False, the first row is assumed to be data and the keyword will be named using the combination of the ColumnBaseName appended with the column index number. Defaults to True. |
Header | Optional | The list of keywords that the values in each column will be stored in. The column values will be associated with keywords from left to right unless Index is used. |
Index | Optional | The column indices in the CSV file that the keywords in Header will use for their values. The first entry will be the value for the first keyword in Header, the second entry will be the value for the second keyword, etc. |
SkipLines | Optional | The number of lines to be skipped at the beginning of the CSV file. Defaults to 0. |
IgnoreComments | Optional | If True, lines beginning with # will be ignored. If False, all text is processed, regardless of the presence of a # character. Defaults to False. |
Delimiter | Optional | The delimiter used to separate columns of the CSV file. It must be a single character. Defaults to a comma, [ , ]. |
RowBaseName | Optional | The default base keyword name to use as the Identity of each Resource. The default Resource names are the base name appended with the row index number. For example Row0, Row1, etc. Defaults to Row. |
Identity | Optional | The name given to each Resource. Intended to be used with the IdentityArgs parameter to give each Resource a unique name. If Identity is not specified, the Resource names are created according to the RowBaseName description. |
IdentityArgs | Optional | Replace arguments in Identity of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
ColumnBaseName | Optional | The base name used for the default keyword names identifying each column read from the CSV file. If no header is specified, the keyword names will be the value of this keyword with the column index number appended to it, starting at zero. For example Column0, Column1, Column2, etc. Defaults to Column. |
GisGeometry | Optional | The Geometry to be added to the Resources. Must be in either well-known-text or well-known-binary format. |
GisGeometryKey | Optional | The keyword used to reference the Geometry. The name of the column in each row in the CSV file that contains that contains a GIS geometry in well-known text (WKT) or well-known binary (B) format. When converted from its text representation to binary form, it will be added to the Resource (row) as its defining geometry. This keyword, if provided, takes precedence over GisGeometry and GisGeometryRef. Note that special GIS keywords described above will apply to the conversion of these GIS columns to their binary form. |
GisGeometryRef | Optional | Identical to GisGeometryKey |
GisGeometryArgs | Optional | Replace arguments in the GIS geometry keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
GisType | Optional | The text format of the Geometry. If a Geometry is specified, this keyword is required. Supports WKT (well-known-text) and WKB (well-known-binary). |
RemoveGisKeywordAfterImport | Optional | If True, the Geometry keyword will be removed from the Resources after the Geometry has been added to the Resources. These GIS geometry definitions can be quite large consisting of many characters. To help manage memory, remove these keywords once they are converted to reduce memory requirements. |
The following is an example of a CsvReader strategy:
Object = Strategy Name = ReadFilterData Type = CsvReader CsvFile = "%1/csvreader_data.csv" CsvFileArgs = "inputdir" HasHeader = False SkipLines = 0 IgnoreComments = False Delimiter = "," Identity = "%1_%2" IdentityArgs = ( SourceProductId, StereoSource ) EndObject
Return to List of Strategy Types
Resource geometries can be written to the CSV file as keywords using the GisGeometryKey parameter in the format specified by the GisType parameter. The two supported formats are well-known binary (WKB) and well-known text (WKT).
Note: Writing a geometry to a CSV file in WKT format while using the default comma delimiter may result in the CSV file being unreadable by CsvReader. Change the delimiter to a different character if using WKT format.
Rows in the CSV file will be delimited by a new line. Columns are delimited by the character specified by the Delimiter parameter, which defaults to a comma.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the CsvWriter strategy. |
Type | Required | Identifies the CsvWriter strategy - must be CsvWriter or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the CsvWriter strategy. |
CsvFile | Required | The output CSV file. Can use arguments specified by CsvFileArgs. |
CsvFileArgs | Optional | Replace arguments in the CsvFile keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
Mode | Optional | The mode in which to write the CSV file. If Create, a new file is created or an existing file with the same name is overwritten. If Append, the output CSV file will be appended to the file. Defaults to Create. |
Keywords | Required | A list of the keywords in the Resources to be written to the CSV file. The keyword names are written to the header of the CSV file as the name of each column. Each Resource is written as a row with the corresponding keyword values written in each column. Only the keywords specified will be written, all other keywords in the Resources are ignored. |
Header | Optional | Option to write a header to the CSV file. Defaults to True, which writes the values specified by the Keywords parameter to the header of the CSV file. False does not write a header to the CSV file, which is useful for appending to another CSV file with the same header. |
Delimiter | Optional | The character or string used to separate fields in a row. Defaults to a comma. |
DefaultValue | Optional | If a Resource does not have a keyword that is specified by the Keywords parameter, it will write this value to the column in the corresponding row CSV file instead. Defaults to NULL. |
SkipEmptyLists | Optional | If True, it will not add empty Resources to the CSV file. Defaults to False. |
GisGeometryKey | Optional | The keyword to write the geometry to. If the keyword already exists in a Resource, then the keyword value will be overwritten with the text version of the GIS geometry. The geometry will be written in the text format specified by GisType. Takes precedence over GisGeometryRef. |
GisGeometryRef | Optional | Identical to GisGeometryKey. |
GisType | Optional | The text format to write the geometry in. The two supported formats are WKT (well-known-text format) and WKB (well-known-binary format). Defaults to WKB. Note that WKT uses commas, which may render the CSV file unreadable if the delimiter is set to a comma. |
The following is an example of a CsvWriter strategy:
Object = Strategy Type = CsvWriter Name = mdismla CSVFile = "mdis_mla_ridelong.lis" Mode = Create Header = True Keywords = (YearDoy, SourceProductId, StartTime, EtStartTime, ExposureDuration, CenterLongitude, CenterLatitude, PixelResolution, MeanGroundResolution, IncidenceAngle, EmissionAngle, PhaseAngle, SubSolarGroundAzimuth, SubSpacecraftGroundAzimuth, ParallaxX, ParallaxY, ShadowX, ShadowY) Delimiter = "," DefaultValue = "NULL" EndObject
Return to List of Strategy Types
The access details of the database can be stored in a configuration file in a profile of the Database object. The configuration file containing the Database object is specified by the DbConfig parameter. The database profile is specified by the DbProfile parameter. A special parameter called DbFile is provided for SQLite files. This shortened specification for SQLite files eliminates the need for creating an elaborate Database configuration for file-based databases.
Generally, any valid SQL query can be used that returns rows from the results of the query issued to any database. Any database that is supported by the Qt system configuration shipped with each ISIS release can accept SQL queries. This typically includes PostgreSQL, MySQL and SQLite. When specifying the query in the Query, some special quoting may be needed in order to comply with SQL syntax. Before a query is sent to the database, all occurrences of " and ' in the query string are replaced with double quotes [ " ] and single quotes [ ' ], respectively.
Keyword | Disposition | Description |
---|---|---|
Name | Required | The user-defined name for the strategy. |
Type | Required | The type of the Strategy. This parameter must be DatabaseReader or the Strategy will not be identified as a DatabaseReader strategy. |
Description | Optional | The user-provided description for the Strategy. |
DbConfig | Required | The configuration file containing the Database object of the database. |
DbConfigArgs | Optional | Replace arguments in the DbConfig keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
DbProfile | Required | The Name of the database Profile to use in the Database Object in the configuration file specified by the DbConfig parameter. |
DbFile | Optional | A SQLite database file to use as the database. Takes precedence over a database identified by DbConfig and DbProfile. |
DbFileArgs | Optional | Replace arguments in the DbFile keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
Query | Required | The SQL query to be sent to the database. Double quotes should be escaped by using ". |
QueryArgs | Optional | Replace arguments in the Query keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
QueryStore | Optional | The Keyword to store the Query used to create the Resource in the new Resources. If this parameter is not specified, the Keyword will not be stored in the Resources. |
Target | Required | Specify how resulting query rows will be stored. If this parameter is set to Resource, the Query will be executed once and each retrieved row will be stored as a new Resource. If this parameter is set to Asset, the Query will be executed once for each Resource, and all rows retrieved for the Query will be stored in each Resource as an Asset containing the list of new Resources, one Resource per row. |
Asset | Optional | If the Target parameter is set to Asset, this parameter is the name of the Asset added to each Resource. If the Target parameter is set to Asset, this parameter is required. |
RowBaseName | Optional | The base of the default Name of each new Resource. The default Resource Names are created from the base and a number. The default base is Row. |
Identity | Optional | The Name of each new Resource. This parameter is intended to be used with the IdentityArgs parameter to give each Resource a unique name. If the Name is not unique or it is empty, the default Name will be used. |
IdentityArgs | Optional | Replace arguments in the Identity keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
GisGeometry | Optional | The default Geometry to be added to each new Resource. It must be in either well-known-text or well-known-binary format. If the default Geometry is used and this parameter is not specified, no Geometry will be added to the new Resources. |
GisGeometryKey | Optional | The Keyword in the new Resources containing the Geometry to be added to those Resources. If the Keyword does not exist, the default Geometry is used. This parameter takes precedence over GisGeometryRef. |
GisGeometryRef | Optional | Identical to GisGeometryKey. |
GisGeometryArgs | Optional | Replace arguments in the GisGeometry keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
GisType | Optional | The text format of the Geometry to be added to each new Resource. If this parameter is set to WKT, the Geometry is interpreted as well-known-text. If WKB, it is interpreted as well-known-binary. This parameter is required if GisGeometry, GisGeometryKey, or GisGeometryRef are specified. |
RemoveGisKeywordAfterImport | Optional | Whether the Keyword containing the Geometry should be removed from the new Resources. If this parameter is set to True, the Geometry Keyword will be removed. If it is set to False, the Keyword will remain in the new Resources. Defaults to False. |
The following is an example of a valid configuration file named mdis.conf with a DatabaseReader strategy within an IsisMiner object and a Database object:
Object = IsisMiner Object = Strategy Name = MDISSelection Type = DatabaseReader DbConfig = mdis.conf DbProfile = MDIS Query = "select i.SourceProductId, i.YearDoy, g.StartTime, i.EtStartTime, i.ExposureDuration, g.CenterLongitude, g.CenterLatitude, i.ObservationType, g.PixelResolution, g.MeanGroundResolution, g.IncidenceAngle, g.EmissionAngle, g.PhaseAngle, g.SubSolarGroundAzimuth, g.SubSpacecraftGroundAzimuth, g.ParallaxX, g.ParallaxY, g.ShadowX, g.ShadowY, p.GisFootprint from Polygon p INNER JOIN Image i ON p.SourceProductId=i.SourceProductId INNER JOIN Geometry g ON p.SourceProductId=g.SourceProductId INNER JOIN Statistics s ON p.SourceProductId=s.SourceProductId where (i.Center=747.7 or i.Center=748.7) order by g.PixelResolution" Target = Resource Identity = "%1" IdentityArgs = "SourceProductId" GisGeometryRef = GisFootprint GisType = WKB RemoveGisKeywordAfterImport = True EndObject EndObject Object = Database DefaultProfile = "MDIS" Group = Profile Name = "MDIS" Dbname = "Messenger_DOY_2011095_TO_2013011_gis.db" Type = SQLite Description = "MDIS Sqlite database" EndGroup EndObject
Return to List of Strategy Types
All comparisons are case-insensitive. The keyword used to filter Resources is specified in the Keyword key in the Filter Strategy object definition found in the Strategy object. Only the first value in the named Resource Keyword will be checked unless the [optional] CheckAll keyword is set to True.
Values contained in the Include keyword in the Strategy object will be compared with values in the Resource key specified in the Keyword key. If a value exists in both keywords, then the Resource is retained (as active). Values contained in the Exclude keyword in the Strategy object that exist in the Resource key will deactivate a Resource.
Both Include and Exclude keywords may exist in the Filter Strategy object. In this case, values are first checked in the Include list then the Exclude list. Whichever occurs first, that action is taken for the Resource. If none of the keyword values exist in either Include or Exclude keywords, then disposition of the Resource depends on the lists. If the Include list is not empty and the Exclude list is empty, the Resource is deactivated. Otherwise, the Resource is retained as active.
If needed, users can provide a regular expression string that will use Qt's QRegExp class implementation to apply to keyword values. If given, this test is applied after the Include and before the Exclude test to determine if the value satisfies the regular expression. If after all the tests are applied and this string was supplied, the Resource is discarded since it did not succeed in its test.
If neither Include nor Exclude keywords exist, then no action is taken for the Resources.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the Filter strategy. |
Type | Required | Identifies the Filter strategy - must be Filter or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the Filter strategy. |
Keyword | Required | Specifies the name of an existing Resource keyword that will be checked for Include and Exclude values. |
CheckAll | Optional | Boolean flag to check all values in the keyword (True) or just the first value (False). Defaults to True. |
Include | Optional | Specify one or more values to check for in the Keyword for existence. If it exists, the keyword is retained (as activated) unless the Exclude parameter is also present - in that case, the rules of Exclude apply. If it does not exist, it is deactivated regardless of the Exclude parameter. |
RegExp | Optional | A single regular expression string can be given in this parameter to test values in the keyword. The syntax in the regular expression must strictly follow the requirements of the Qt QRegExp class. If the string satisfies the regular expression, it is retained as active or it will be discarded. |
WildCardMode | Optional | When using a RegExp, this parameters accepts a value of True or False to signal use of the QRegExp::WildcardUnix mode for regular expressions. If True, QRegExp::WildcardUnix mode is set. If False, the default QRegExp::RegExp mode is used. |
Exclude | Optional | Specify one or more values to look for in the Keyword. If found, the Keyword is deactivated. Otherwise, if Include is not specified, the Keyword is retained. |
The following is an example of a Filter strategy:
Object = Strategy Name = TossIgnores Type = Filter Description = "Deactivate all ControlMeasures that are ignored" Keyword = Ignore Exclude = True EndObject
Return to List of Strategy Types
Keyword | Description | |
---|---|---|
Name | Required | User-defined name of the GisIntersect strategy. |
Type | Required | Identifies the GisIntersect strategy - must be GisIntersect or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the GisIntersect strategy. |
GisType | Required | Specifies the type of the geometry. Valid options are WKT, WKB, pvlWKT, pvlWKB, or IsisCube. WKT and WKB are common GIS file formats that indicate well-known-text and well-known-binary, respectively. If GisType is pvlWKT or pvlWKB, GisGeometry must be the path to a PVL file which contains a valid WKT or WKB geometry. |
GisGeometry | Required | The actual geometry or geometry source. This can either be directly equal to a WKB or WKT value like POLYGON ((0.00 22.50, 90.00 22.50, 90.00 65.00, 0.00 65.00, 0.00 22.50)) or can be set to the path to a PVL file containing a valid geometry. If GisGeometry is set to a file name, the associated file must be a PVL file with a valid WKT or WKB geometry, or be an ISIS cube. |
GisGeometryArgs | Optional | Replace arguments in the GisGeometry keyword of the form %1, %2,..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
BoundingBox | Optional | Specify whether or not to use a bounding box from the source. Possible values are True or False. Default value is False. |
ComputeRatio | Optional | Specify whether or not to compute the overlap ratio and add it to the Resource. Options are True or False. Default value is False. |
RatioRef | Optional | Specify which Keyword to insert the calculated overlap ratio in, if ComputeRatio = True. |
GisGeometryRef | Optional | Used when using a file name as a GisGeometry source. This indicates the Keyword in the associated PVL file that the WKT or WKB geometry value will be found in. |
The following is an example of a GisIntersect strategy:
Object = Strategy Name = DeactiveNonintersecting Type = GisIntersect Description = "Define bounding box (lower left, upper right (lon lat)) around Degas" GisType = "WKT" GisGeometry = "LINESTRING(225.00 30.00, 240.00 45.00)" BoundingBox = True ComputeRatio = True EndObject
Return to List of Strategy Types
Parameters to this algorithm can define the ratio (or percentage) of common overlap for the two GIS geometries. As implied, all Resources must have GIS geometries in order to participate in the overlap determination.
Overlap candidates are determined from other Resources for every incoming active Resource and a list of Resources that are determined from another isisminer Strategy configuration. The configuration can either be specified directly in the GisOverlap Strategy object (an embedded IsisMiner object called a CandidateMiner) or come from an external file called CandidateConfigFile. Candidate overlap Resources can be either part of the incoming Resource list (active and inactive) or from a completely different source.
Once overlaps are determined for a Resource, the candidate and its source are merged/combined into a single Resource and added to an Asset of the primary Resource. Keywords from both Resources are combined into a single Resource by renaming each keyword from both Resources and propagating them to the merged Asset Resource. An A is appended to the source Resource keywords and a B is appended to the candidate Resource keywords before they are merged to distinguish keywords. The suffix of keyword sources can be specified by the user using the MergeSuffixA and MergeSuffixB keywords, respectively. The identity of the merged Resource is a combination of the two Resource identities separated by an underscore character.
Once the list of overlapping Resources is determined, the user can opt to immediately process the overlaps. If a second isisminer configuration is provided called OverlapMiner or OverlapConfigFile, the list of overlapping Resources that are contained combined Resources and geometries is run though the Strategy list in the OverlapMiner configuration. This option is useful to eliminate the need for intermediate storage which could cause memory issues.
Keyword | Disposition | Description | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Name | Required | User-defined name of the GisOverlap strategy. | ||||||||||||||
Type | Required | Identifies the GisOverlap strategy - must be GisOverlap or it will not be recognized as a valid Strategy by isisminer. | ||||||||||||||
OverlapMinimum | Optional | Specifies the minimum common overlap ratio of the two geometries with respect to the primary Resource to be valid. If the computed ratio is less than this value, the Resource will not be deemed a valid overlap due to lack of common coverage. Defaults to 0.0. | ||||||||||||||
OverlapMaximum | Optional | Specifies the maximum common overlap ratio of the two geometries with respect to the primary Resource to be valid. If the computed ratio is greater than this value, the Resource will not be deemed a valid overlap due to too much common coverage. Defaults to 1.0. | ||||||||||||||
OverlapRatioKey | Optional | Provides the name of the keyword to retain the common overlap ratio. If not provided, it will be stored in the merged Resource as the keyword OverlapRatio. | ||||||||||||||
Asset | Optional | Provides the name of the Asset created containing all the Resources determined to have overlap GIS geometries from the candidate list. If not provided, GisOverlap will be the default. | ||||||||||||||
MergeSuffixA | Optional | To provide uniqueness of keywords that are common to both Resources when they are combined, a suffix is added to the primary Resource. Use this keyword to specify the keyword suffix. If not provided, A is the default. | ||||||||||||||
MergeSuffixB | Optional | To provide uniqueness of keywords that are common to both Resources when they are combined, a suffix is added to the Resource that overlaps the primary Resource. Use this keyword to specify the keyword suffix of the overlapping Resource. If not provided, B is the default. | ||||||||||||||
OverlapMerge | Optional |
GIS Operators
|
||||||||||||||
CandidateMiner | Optional (Object) | To select the Resource list used to evaluate overlapping geometries, users have several ways to create the list. A complete isisminer configuration object called (and named) CandidateMiner can be provided that will be executed on the full Resource list to generate a list of candidate Resources for comparisons of overlaps. This configuration must be a compliant, properly formatted isisminerobject with the only difference being the name of the object. It may also be provided in a file called CandidateConfigFile. Either a CandidateMiner object or configuration file may be provided, but not both. If no candidate options are provided, the current active list of Resources will be used. | ||||||||||||||
CandidateConfigFile | Optional (File) | This option can be used in lieu of a CandidateMiner object to specify a set of Strategies to select overlap candidates. You may also provide arguments to that can be translated when executed to resolve the file name by providing a keyword called CandidateConfigFileArgs that contains keywords that translate occurrences of %1 to %n strings in the name of the file. | ||||||||||||||
OverlapMiner | Optional (Object) | For efficiency purposes only, users can immediately process the overlap list of a given Resource at the time it is completely determined. Simply provide an isisminer configuration object called (and named) OverlapMiner that will be executed on the overlap Resource at the time each one is fully determined. This configuration must be a compliant, properly formatted isisminerobject with the only difference being the name of the object. It may also be provided in a file called OverlapConfigFile. Either an OverlapMiner object or a configuration file may be provided, but not both. If no overlap options are provided, no action is taken on the overlap list. | ||||||||||||||
OverlapConfigFile | Optional (File) | This option can be used in lieu of a OverlapMiner object to specify a set of Strategies to process overlap Resource results. You may also provide arguments to that can be translated when executed to resolve the file name by providing a keyword called OverlapConfigFileArgs that contains keywords that translate occurrences of %1 to %n strings in the name of the file. |
The following is an example using the GisOverlap strategy. In this example we are reading image data from an SQLite database containing WAC and NAC MESSENGER data created from caminfo runs. It determines all the images overlapping a square geometry around Debussy crater.
Object = IsisMiner Object = Strategy Name = MDISSelection Type = DatabaseReader DbConfig = wac_nac_overlaps.conf DbProfile = MDIS Mode = Select # Select strings that need double quotes must use " since they cannot # be stored in keyword values. These strings will be prescanned before # issuing to the DB and they will be properly replaced. Query = "select i.SourceProductId, i.YearDoy, i.File, g.StartTime, i.EtStartTime, i.ExposureDuration, i.ObservationId, g.CenterLongitude, g.CenterLatitude, i.ObservationType, g.PixelResolution, g.MeanGroundResolution, g.IncidenceAngle, g.EmissionAngle, g.PhaseAngle, g.SubSolarGroundAzimuth, g.SubSpacecraftGroundAzimuth, ParallaxX, ParallaxY, ShadowX, ShadowY, p.GisFootprint from Polygon p INNER JOIN Image i ON p.SourceProductId=i.SourceProductId INNER JOIN Geometry g ON p.SourceProductId=g.SourceProductId INNER JOIN Statistics s ON p.SourceProductId=s.SourceProductId" # Resource targets are for reading multiple rows from the DB. The fields # from each row are created as Resources. Target = Resource Identity = "%1" IdentityArgs = "SourceProductId" # Can specify a GIS keyword that will be converted on input GisGeometryRef = GisFootprint GisType = WKB RemoveGisKeywordAfterImport = True EndObject Object = Strategy Type = GisIntersect Name = SelectDebussy GisType = WKT # Define bounding box (lower left, upper right (lon lat)) around Debussy # crater GisGeometry = "LINESTRING(11.56 -34.91, 13.83 -33.03)" BoundingBox = True ComputeRatio = True RatioRef = DebussyOverlapRatio ShowProgress = True EndObject # Since there is no CandidateMiner specified so the current active list is # processed, recursively. Meaning each Resource is checked for overlap # of all other active Strategies Object = Strategy Type = GisOverlap Name = DebussyOverlaps Description = "Check for image overlaps near Debussy" ShowProgress = True OverlapMinimum = 0.5 OverlapMerge = Intersection EndObject Object = Strategy Type = AssetSidebar Name = WriteOverlapFiles Description = "Write CSV files of overlap data" Asset = GisOverlap Object = IsisMiner Object = Strategy Type = CsvWriter Name = WriteOverlaps Description = "Write individual files with overlap data" SkipEmptyLists = True CsvFile = "overlaps/%1.csv" CsvFileArgs = "Identity" Keywords = ("SourceProductIdA", "SourceProductIdB", "OverlapRatioA", "OverlapRatioB", "YearDoyA", "YearDoyB", "CenterLongitudeA", "CenterLatitudeA", "CenterLongitudeB", "CenterLatitudeB", "GisOverlapCentroidX", "GisOverlapCentroidY", "PixelResolutionA", "PixelResolutionB", "EmissionAngleA", "EmissionAngleB", "IncidenceAngleA", "IncidenceAngleB", "PhaseAngleA", "PhaseAngleB") EndObject EndObject EndObject EndObject Object = Database DefaultProfile = "MDIS" Group = Profile Name = "MDIS" Dbname = "Messenger_Monochrome_DOY_2011095_TO_2014304_gis.db" Type = SQLite Description = "MDIS Sqlite database" EndGroup EndObject End
Return to List of Strategy Types
If the union geometry is present and good, an entry will be added to the Resource with the parameter RatioRef as the key and the overlap ratio as the value. The double parameters OverlapMinimum and OverlapMaximum indicate the acceptable range for the overlap ratio. Geometries with overlap ratios which do not fit the constraints will be discarded.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the GisUnion strategy. |
Type | Required | Identifies the GisUnion strategy - must be GisUnion or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the GisUnion strategy. |
OverlapMinimum | Optional | User provides a value for the minimum valid overlap ratio. Defaults to 0.0. |
OverlapMaximum | Optional | User provides a value for the maximum valid overlap ratio. Defaults to 1.0. |
RatioRef | Optional | User provides a keyword name which will be used as the column of overlap ratios stored in the Resource. Defaults to UnionOverlapRatio. |
The following is an example of a GisUnion strategy:
Object = Strategy Name = UnionHighOverlap Type = GisUnion Description = "Union remaining geometries which have a high overlap ratio." OverlapMinimum = 0.65 OverlapMaximum = 1.0 RatioRef = "HighOverlapRatio" EndObject
Return to List of Strategy Types
A keyword value is considered numeric if it can be converted to a floating point number. If the keyword does not exist in the Resource, then its value is not considered numeric.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the IsNumeric strategy. |
Type | Required | Identifies the IsNumeric strategy - must be IsNumeric or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the IsNumeric strategy. |
Keyword | Required | Specifies the name (or names - you can list more than one keyword to check) of an existing Resource keyword that will be included if it is numeric. |
The following is an example of an IsNumeric strategy:
Object = Strategy Name = RemoveNotApplicable Type = IsNumeric Description = "Deactivate all responses which did not respond to rating." Keyword = Rating EndObject
Return to List of Strategy Types
If keyword values are within the limits, the Resource disposition is set to activated. If the keyword is not within the limits provided, the Resource disposition is set to deactivated.
This Strategy uses the Calculator strategy. Any equation constructed that is valid for the Calculator strategy is valid for Limit. This includes the use of Equations group to use more than one equation. The difference here is the value of the last equation is used to evaluate a true (!0) or false (0) limit for the Resource. Also note, that a side effect of using the Equations group is that any new keyword created by an equation remains in the Resource.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the Limit strategy. |
Type | Required | Identifies the Limit strategy - must be Limit or it will not be recognized as a valid Strategy by isisminer. |
Equation | Optional | Specifies an equation using Resource keywords that will be checked for true (!0) or false (0) results. |
Equations | Optional | This group can be used instead of the Equation keyword to define a series of equations executed in sequence. The last resulting computation will be used to check for true (!0) or false (0) results. Either the Equation keyword or Equations group must be specified. |
Following are few examples of Limit strategies:
Object = Strategy Type = Limit Name = EmissionAngle Equation = "((emissionangle > 30) && (emissionangle < 80))" EndObjectThis shows an example of using a series of equations to determine limits:
Object = Strategy # This algorithm will select only images where the center latitude # and longitudes are within the defined region provided in the GLOBALS # parameter file. Name = SelectRegion Type = Limit Description = "Find all images with center image geometry in the defined region" # This set of equations will result in 1 if test are good, meaning the # clat/clon is in the defined region. If any test fails, InRegion = 0 # (false) and the Resource is disabled. Note this also defines the GIS # bounding box. Group = Equations InRegion = 1 InRegion = "InRegion * (CenterLongitude > RegionLowerLeftLongitude)" InRegion = "InRegion * (CenterLatitude > RegionLowerLeftLatitude)" InRegion = "InRegion * (CenterLongitude <= RegionUpperRightLongitude)" InRegion = "InRegion * (CenterLatitude <= RegionUpperRightLatitude)" EndGroup EndObject
Return to List of Strategy Types
SortKey must have a numerical value for each Resource that will be sorted. See the IsNumeric strategy.
Order may be specified as either Ascending or Descending. If not provided, the Resources will be sorted in Ascending order.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the NumericalSort strategy. |
Type | Required | Identifies the NumericalSort strategy - must be NumericalSort or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the NumericalSort strategy. |
SortKey | Required | Specifies the name of an existing Resource keyword that will be used to sort the Resources. The keyword's values for each Resource must be numerical. |
Order | Optional | The order, ascending or descending, that Resources can be sorted in. Defaults to Ascending. |
The following is an example of a NumericalSort strategy:
Object = Strategy Name = RankSort Type = NumericalSort Description = "Sorts the Resources in ascending order by Rank" SortKey = Rank Order = Ascending EndObject
Return to List of Strategy Types
Records are delimited by a new line. Fields are delimited as specified by the Delimiter parameter, which defaults to a comma.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the PdsTableCreator strategy. |
Type | Required | Identifies the PdsTableCreator strategy - must be PdsTableCreator or it will not be recognized as a valid Strategy by isisminer. |
PdsFormatFile | Required | The output format file. Can use arguments specified by PdsFormatFileArgs. |
PdsFormatFileArgs | Optional | Replace placeholders in PdsFormatFile of the form %1, %2,..., %99, with the arguments specified. Each placeholder is replaced by its corresponding argument in order. |
PdsTableFile | Required | The output Pds table file. Can use arguments specified by PdsTableFileArgs. |
PdsTableFileArgs | Optional | Replace placeholders in PdsTableFile of the form %1, %2,..., %99, with the arguments specified. Each placeholder is replaced by its corresponding argument in order. |
Mode | Required | The mode in which to write the Pds Table file. If Create, a new file is created or an existing file with the same name is overwritten. If Append, the output Pds Table file will be appended to the file. Defaults to Create. |
Delimiter | Optional | The character or string used to separate fields in a record. Defaults to a comma. |
DefaultValue | Optional | If a Resource does not have a keyword that is specified by the Keywords parameter, it will write this value to the column in the corresponding row Pds Table file instead. Defaults to NULL. |
The following is an example of a PdsTableCreator strategy:
Object = Strategy Name = TestAppendTableWriter Type = PdsTableCreator PdsFormatFile = "%1/TestGeneralFormat.txt" PdsFormatFileArgs = "inputdir" PdsTableFile = "%1/TestAppend.TAB" PdsTableFileArgs = "outputdir" Mode = Append Delimiter = "," DefaultValue = "NULL" EndObject
Return to List of Strategy Types
Records are delimited by a new line. Fields are delimited as specified by the Delimiter parameter, which defaults to a comma.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the PdsTableFormat strategy. |
Type | Required | Identifies the PdsTableFormat strategy - must be PdsTableFormat or it will not be recognized as a valid Strategy by isisminer. |
PdsFormatFile | Required | The output format file. Can use arguments specified by PdsFormatFileArgs. |
PdsFormatFileArgs | Optional | Replace arguments in the PdsFormatFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
KeywordFormatFile | Optional | Name of a file containing individual PDS keyword file formatting directives. Provides user with customization options. Defaults to $ISISROOT/appdata/translations/pdsExportRootGen.typ. |
KeywordFormatFileArgs | Optional | Replace arguments in the KeywordFormatFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
Delimiter | Optional | The character or string used to separate fields in a record. Defaults to a comma. |
Column | Optional | The names of the columns to be written to the format files created. |
DataType | Optional | The type of data within the columns. |
Unit | Optional | The unit associated with the data in the column. |
DataDescription | Optional | User provides a description of the data stored in the column. |
{Column Name}_BYTES | Optional |
Specifies the size of the column, named in Column, in order to make it more efficient,
in order to eliminate the need to traverse the entire table to determine the largest element
in the column.
Example: POINT_ID_BYTES = 32 |
The following is an example of a PdsTableFormat strategy:
Object = Strategy Name = PdsTableFormat Type = PdsTableFormat PdsFormatFile = "POINTCLOUDTAB.FTM" PdsFormatFileArgs = "outputdir" Delimiter = "," Column = ("POINT_ID", "STATUS") DataType = ("CHARACTER", "CHARACTER") Unit = ("NONE", "NONE") DataDescription = ("Unique point identifier.", "Status of point") POINT_ID_BYTES = 32 STATUS_BYTES= 32 EndObject
Return to List of Strategy Types
If the Identity keyword is not provided or its value is an empty string, each Resource created will be named the row number. Rows are read (starting at 0) from the PDS table data in PdsTableFIle.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the PdsTableReader strategy. |
Type | Required | Identifies the PdsTableReader strategy - must be PdsTableReader or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the PdsTableReader strategy |
PdsLabelFile | Required | User provides file containing label describing the columns in the table. |
PdsLabelFileArgs | Optional | Replace arguments in the PdsLabelFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
PdsTableFile | Optional | User may need to provide the name of the table file containing data. This may not be needed if the base file name is the same as the label file but ends in .tab. |
PdsTableFileArgs | Optional | Replace arguments in the PdsTableFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
Identity | Optional | The name given to each Resource. Intended to be used with the IdentityArgs parameter to give each Resource a unique name. If Identity does not have a value, the Resource names are set as PvlN, N being the position of the PVL file in FromList. |
IdentityArgs | Optional | Replace arguments in the Identity keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
TableName | Optional | The name of the table object in the label file may use different names. Users will need to provide the name of the PVL object containing the descriptions of each column. Defaults to TABLE. |
UseFormattedName | Optional | Some of the tables will contain column names that contain underscores and some special characters. If set to True, all white space characters are converted to spaces. Spaces are then compressed to one space. Any left/right parentheses are removed. Then the conversion to camel-case is performed. Camel case always converts the first character in the name to uppercase. Each space or underscore character is removed and the character following it is converted to uppercase. All other characters are converted to lowercase. |
The following is an example of an PdsTableReader strategy:
Object = Strategy Name = PdsTableReaderTestDefault Type = PdsTableReader PdslabelFile = "%1/index.lbl" PdslabelFileArgs = "inputdir" TableName = "INDEX_TABLE" Identity = "%1" IdentityArgs = File_Name UseFormattedName = False Description = "Test the default functionality of PdsTableReader" EndObject
Return to List of Strategy Types
If the Identity keyword is not provided or its value is an empty string, each Resource created will be named as PvlN. N is the position number (starting at 0) of the PVL file contained in FromList. The basename, Pvl, can be changed by setting PvlBaseName.
For example, if FromList contains PVL Files ~/somepvl.pvl and ~/someotherpvl.pvl, their default Identity values would be Pvl0 and Pvl1, respectively.
The Includes keyword operates on PVL Objects and Groups (containers). When provided by itself, the Includes keyword will exclusively include the keywords belonging to the containers specified. Since Groups must be contained in Objects, you cannot include a Group without also including its containing Object. Similarly, you must include the Root Object since all PVL files have a Root Object container. For example:
Object = Outer Id = 001 Group = ExtraInfo Name = outside Time = today EndGroup Group = OtherInfo Info = "No information present" EndGroup Group = Important Level = High EndGroup Object = Inner Data = "Some test data" EndObject EndObjectTo include the ExtraInfo Group, the Includes keyword should be specified, which will include the Id, Name, and Time keywords in the created Resource.
Includes = (Root, Outer, ExtraInfo)
The Excludes keyword operates on PVL Objects and Groups (containers). When provided by itself, the Excludes keyword will exclude the keywords belonging to the containers specified, while all other containers and keywords will be included.
When both Includes and Excludes are provided, an Object and all of its keywords and Groups will be included except for any excluded Groups (Inclusions are applied to Objects while exclusions are applied to Groups).
Referring to the Includes example, to include the Outer Object except for its OtherInfo Group, you would specify the following:
Includes = (Root, Outer) Excludes = OtherInfo
The KeyListFile keyword operates on PVL Keywords and expects a file containing PVL Keywords. If provided, only the keywords in the file will be included in the created Resource. Note that these keywords will only be included if their containers are included or not excluded.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the PvlReader strategy. |
Type | Required | Identifies the PvlReader strategy - must be PvlReader or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the PvlReader strategy. |
FromList | Required | User provides file containing a list of paths to PVL files. |
FromListArgs | Optional | Replace arguments in the FromList keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
Identity | Optional | The name given to each Resource. Intended to be used with the IdentityArgs parameter to give each Resource a unique name. If Identity does not have a value, the Resource names are set as PvlN, N being the position of the PVL file in FromList. |
IdentityArgs | Optional | Replace arguments in the Identity keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
Includes | Optional | Used to exclusively include Objects and Groups when creating a Resource from a read PVL file. |
Excludes | Optional | Used to ignore/exclude Objects and Groups when creating a Resource from a read PVL file. |
KeyListFile | Optional | Used to exclusively include any PVL Keywords in the provided file when creating a Resource from a read PVL file. |
KeyListFileArgs | Optional | Replace arguments in the KeyListFile keyword of the form %1, %2, ..., %99, with the values of the assumed keywords specified in this keyword parameter. Each argument is replaced by its corresponding keyword value in order. The keywords are assumed to be Resource or global keyword names. |
PvlBaseName | Optional | The base of the default name of each Resource. The default Resource names are the base name and a number. For example: Pvl0, Pvl1, etc. Defaults to Pvl. |
The following is an example of an PvlReader strategy:
Object = Strategy Name = PvlReader Type = PvlReader FromList = "%1/testpvl.pvl" FromListArgs = "inputdir" Identity = "%1" IdentityArgs = Id KeyListFile = "%1/keystoinclude.lis" KeyListFileArgs = "inputdir" Description = "A PvlReader Strategy that includes the keywords contained in keystoinclude.lis when creating the Resource from testpvl.pvl" EndObject
Return to List of Strategy Types
For DeleteAsset and HasAsset, the Asset is specified in the Operations Keyword as
Operations = HasAsset::AssetName or Operations = DeleteAsset::AssetName.
Keyword | Disposition | Description | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Name | Required | User-defined name of the ResourceManager strategy. | ||||||||||||
Type | Required | Identifies the ResourceManager strategy - must be ResourceManager or it will not be recognized as a valid Strategy by isisminer. | ||||||||||||
Description | Optional | User provides a description of the ResourceManager strategy. | ||||||||||||
Operations | Required |
Operations
|
The following is an example of a ResourceManager strategy:
Object = Strategy Name = SwitchActivated Type = ResourceManager Description = "Swap activated (Included) and deactivated (Excluded) entries in a Resource." Operations = ToggleDiscard EndObject
A second example with multiple management operations
# Clear the entire Resource list (useful in Sidebar Strategies) Object = Strategy Name = ClearResources Type = ResourceManager Description = "Delete all Resources" Operations = ( ResetDiscard, ToggleDiscard, DeleteDiscard ) EndObject
Return to List of Strategy Types
Commands can be formulated using the keyword argument substitution options available for using global parameter pools for replacing arguments in commands with Resource keyword values. One keyword, CommandArgs, is available to list all the Resource and global pool variables that are to be used in replacement. All commands can use a %1 to %n where n is the total number of keywords list in the CommandArgs keyword. This one list of keywords is used to substitute for all commands.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the RunCommand strategy. |
Type | Required | Identifies the RunCommand strategy - must be RunCommand or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the RunCommand strategy. |
Command | Optional | Provides a single command that will be executed once for each active Resource. |
Commands | Optional | This is a Group in the RunCommand strategy that can contain more than one command to execute for each Resource. This can exist with the Command keyword but contents of this group will be executed after the Command content. |
PreCommands | Optional | The contents of this group contains optional commands that will be run once before any Resources (and their commands) are executed. For example, this group is useful for creating directories prior to processing any Resource commands. |
PostCommands | Optional | The contents of this group contains optional commands that will be run after all Resources (and their commands) are executed. For example, this group is useful for running an ISIS application after processing all Resources. |
CommandArgs | Optional | This keyword contains keywords that will be used to substitute any occurrences of keyword arguments (i.e., %1, %2,...) in all commands regardless of which group the command is from. |
The following is an example of a RunCommand strategy. The " text is required and should be included in the configuration of your command. The " and ' are substituted prior to issuing the command to the shell for execution. This is required because of PVL syntax rules.
Object = Strategy Name = RunManyCommands Type = RunCommand Description = "Tests running many command" CommandArgs = (outputdir, SourceProductId, StartTime) # Ensure the output file exists Group = PreCommands Command0 = "touch %1/runmanycommands.txt" EndGroup # Append the Resource name and start time Group = Commands Command1 = "echo "Resource name %2 and date %3." >> %1/runmanycommands.txt" EndGroup EndObject
Return to List of Strategy Types
Sidebar strategies are useful for running Strategies that have the potential to throw out Resources or alter a current state that may be undesirable. For these cases, you can use a Sidebar that saves all the Resources, activated and deactivated, their current states and preserves any operations that may alter or discard Resources. It can also be used to process an entirely new set of Resources while retaining the current state of existing Resources.
Sidebar strategies can be inserted anywhere in the sequence of Strategies but are best placed strategically at critical points in the processing sequence where preservation of the current active Resource state is needed.
Keyword | Disposition | Description |
---|---|---|
Name | Required | User-defined name of the Sidebar strategy. |
Type | Required | Identifies the Side strategy - must be Sidebar or it will not be recognized as a valid Strategy by isisminer. |
Description | Optional | User provides a description of the Sidebar strategy. |
IsisMiner | Optional (Object) | PVL Object containing an IsisMiner definition to be run on the list of Resources. Either this or StrategyConfigFile must be specified. |
StrategyConfigFile | Optional (File) | File containing run parameters on the Resources. If this file is not given, an IsisMiner object must be specified. |
StrategyConfigFileArgs | Optional | Replace placeholders in StrategyConfigFile of the form %1, %2, etc. up to %99 with the arguments specified. Each placeholder is replaced by its corresponding argument in order. |
SavePoint | Optional | If True, the activeness of all the Resources will be retained after the sidebar configuration has run. If False, any Resources that are discarded during the configuration will remain discarded and lost to subsequent processing. Defaults to True. |
SaveDelete | Optional | If True, any Resources that are deleted by the side configuration will be restored. In addition, any Resources that are created during the configuration will be deleted. If False, deleted Resources will remain deleted and created Resources will remain. Defaults to the value of SavePoint. |
The following is an example of a Sidebar strategy:
Object = Strategy Name = SetUpResources Type = SideBar SavePoint = True SaveDelete = False StrategyConfigFile = "%1/setup.conf" StrategyConfigFileArgs = "inputdir" EndObject
Return to List of Strategy Types
Parameters for this algorithm define two levels of stereo constraints that are described in [Becker 2015]. The first level of constraints are typically for constraining photometric (incidence, emission, phase) and geometric angles. The constraints for these limits are located in the ImageStrength group. Each keyword there should exist in each Resource or have a keyword map specified in the KeywordMap group. The second level constraints contained in the StereoStrength group are stereo limits and are applied to values computed from the stereo algorithm with contributions from both pair Resources. The format for keywords in both of these groups are four element vectors but only 3 values are required. For example, to limit the emission angle between 0 and 50 degrees, the form of the keyword in the ImageStrength group is Emission = (0.0, 50, 15.0 [, 1.0]). The first value in this keyword is the minimum value, the second is the maximum value and the third value is the value for emission that provides the best stereo conditions. The last optional value is a multiplier of the computed strength, or rank, for each keyword limit that can be used to weight the computed strength of the limit beyond the 0 - 1 range.
The rank of each keyword is computed using the following scheme based upon this form of limit keywords in both ImageStrength and StereoStrength: LimitKey = (t0, t1, t2, t3) where all t0 through t3 values are numerical numbers and t0 is the minimum, t1 is maximum, t2 is the best value in the range and t3 is a weighting factor. The formula to compute the rank of each keyword is as follows:
range = max(abs(t2-t0), abs(t1-t2)) rank = 1 - abs(t2-value)/range rank *= t3The end result of rank is then stored in a keyword comprised of the name of the keyword with Rank appended to it. For example if the Emission keyword above is provided as a limit, and an image has an Emission angle value of 30 degrees, the rank computed and stored in EmissionRank = 1 - abs(15 - 30)/max(abs(15-0),abs(50-15)) = 0.57 * 1 (t3). All keyword ranks for a stereo Resource are summed for each group and the values are available in ImageStrengthRank and StereoStrengthRank. The ImageStrengthRank is computed as the average of the two image's rank. These values are typically summed in the Equation keyword in the Strategy definition to specify how to rank the strength of an individual stereo pair. These values can be sorted and reported to users (using an AssetSidebar). The typical equation used to compute the rank of a stereo pair, StereoPairRank in this example, is a combination of the two ranks:
# Now specify how the stereo quality is determined. Equation = "imagestrengthrank * 1.0 + stereostrengthrank * 1.0" Result = StereoPairRankThe StereoPairRank will contain an indication of the strength of a stereo pair. This keyword and image keywords are combined in a Resource made up of both Resources by merging the keywords from the Resources. The keywords from each Resource have a unique suffix appended to the name of the keyword before they are added. The GisOverlap keywords MergeSuffixA and MergeSuffixB may be used to specify keyword suffixes for each image as they are merged. MergeSuffixA is used to identify the primary stereo image, MergeSuffixB the secondary or matched stereo image. Defaults for merge suffix keywords are A and B respectively.
If data is provided by caminfo, then the values required by this Strategy are provided at the center image pixel coordinate. This may not provide accurate results, particularly for the StereoStrength group of keywords, because the values are likely not of the latitude/longitude within the overlapping region of the images. However, the computation of the StereoAngle could be more accurate if you were to take advantage of GisOverlap OverlapMerge options. The centroid (longitude/latitude) values for all of these options is computed and stored in new keywords GisOverlapCentroidX (longitude) and GisOverlapCentroidY (latitude) making this coordinate the same point in both stereo image pairs. By setting a parameter in the KeywordMap group as CenterLongitude = GisOverlapCentroidX and CenterLatitude = GisOverlapCentroidY the StereoAngle is computed from a common surface point. If desired, the user can specify whether to use the stereo angle in the computation of the VerticalPrecision stereo keyword rather than the ParallaxHeightRatio. Take note the equation to compute StereoPairRank can be changed as desired or weights can be provided in the fourth limit keyword value to emphasize or de-emphasize any limit.
Keyword | Disposition | Description | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
OverlapMinimum | Optional | Specifies the minimum common overlap ratio of the two geometries with respect to the primary Resource to be valid. If the computed ratio is less than this value, the Resource will not be deemed a valid overlap due to lack of common coverage. Defaults to 0.0. | ||||||||||||||||
OverlapMaximum | Optional | Specifies the maximum common overlap ratio of the two geometries with respect to the primary Resource to be valid. If the computed ratio is greater than this value, the Resource will not be deemed a valid overlap due to too much common coverage. Defaults to 1.0. | ||||||||||||||||
OverlapRatioKey | Optional | Provides the name of the keyword to retain the common overlap ratio. If not provided, it will be stored in the merged Resource as the keyword OverlapRatio. | ||||||||||||||||
Asset | Optional | Provides the name of the Asset created containing all the Resources determined to have overlap GIS geometries from the candidate list. If not provided, GisOverlap will be the default. | ||||||||||||||||
MergeSuffixA | Optional | To provide uniqueness of keywords that are common to both Resources when they are combined, a suffix is added to the primary Resource. Use this keyword to specify the keyword suffix. If not provided, A is the default. | ||||||||||||||||
MergeSuffixB | Optional | To provide uniqueness of keywords that are common to both Resources when they are combined, a suffix is added to the Resource that overlaps the primary Resource. Use this keyword to specify the keyword suffix of the overlapping Resource. If not provided, B is the default. | ||||||||||||||||
OverlapMerge | Optional |
GIS Operators
|
||||||||||||||||
CandidateMiner | Optional (Object) | To select the Resource list used to evaluate stereo geometries, users have several ways to create the list. A complete isisminer configuration object called (and named) CandidateMiner can be provided that will be executed on the full Resource list to generate a list of candidate Resources for comparisons of stereo potential. This configuration must be a compliant, properly formatted isisminerobject with the only difference being the name of the object. It may also be provided in a file called CandidateConfigFile. Either a CandidateMiner object or configuration file may be provided, but not both. If no candidate options are provided, the current active list of Resources will be used. | ||||||||||||||||
CandidateConfigFile | Optional (File) | This option can be used in lieu of a CandidateMiner object to specify a set of Strategies to select stereo candidates. You may also provide arguments to that can be translated when executed to resolve the file name by providing a keyword called CandidateConfigFileArgs that contains keywords that translate occurrences of %1 to %n strings in the name of the file. | ||||||||||||||||
OverlapMiner | Optional (Object) | For efficiency purposes only, users can immediately process the stereo list of a given Resource at the time it is completely determined. Simply provide an isisminer configuration object called (and named) OverlapMiner that will be executed on the stereo Resource at the time each one is fully determined. This configuration must be a compliant, properly formatted IsisMiner object with the only difference being the name of the object. It may also be provided in a file called OverlapConfigFile. Either an OverlapMiner object or configuration file may be provided, but not both. If no overlap options are provided, no action is taken on the stereo list. | ||||||||||||||||
OverlapConfigFile | Optional (File) | This option can be used in lieu of a OverlapMiner object to specify a set of Strategies to process stereo Resource results. You may also provide arguments to that can be translated when executed to resolve the file name by providing a keyword called OverlapConfigFileArgs that contains keywords that translate occurrences of %1 to %n strings in the name of the file. | ||||||||||||||||
PixelPrecisionMatch | Optional | This parameter specifies the expected matching precision of image pairs. This value is used to compute the expected vertical precision of the stereo data. Specifically, it specifies the ability to match features in the stereo pairs, measured in pixels. If not provided, the default is 0.2. | ||||||||||||||||
ImageStrength | Recommended (Group) |
This group contains keywords that define individual image parameter
minimum, maximum, best and (optional) multiplier/scale value. The
keyword names contained in this group may be mapped to other keywords
using the KeywordMap group. (Mapped) Keywords specified in
this group must exist in both the primary and matching stereo images.
The Resource is not evaluated for stereo potential. The intended
values here are used to screen any images that do not satisfy
individual image parameters.
Individual Image Strength Values
|
||||||||||||||||
StereoStrength | Required (Group) |
This group contains keywords that define stereo strength parameter
minimum, maximum, best and (optional) multiplier/scale value. The
keyword names contained in this group may be mapped to other keywords
using the KeywordMap group. (Mapped) Keywords specified in
this group are computed from the stereo algorithm and stored in the
combined stereo image pair Resource constructed from the Resource
pair. Here is a table of stereo values computed by the algorithm and
the required keywords to compute each value.
Computed Stereo Strength Values
|
||||||||||||||||
UseStereoAngle | Optional | Specify what to use as the VerticalPrecision of a stereo pair. You can use the stereo angle (True) or the ParallaxHeightRatio. It may be more precise, particularly when using the GIS intersection of common area centroid parameters, to use the StereoAngle computed from the common latitude/longitude point rather than the center image pixel latitude/longitude of each image - which is likely not the same point but may be close enough. Defaults to True, to use the StereoAngle. | ||||||||||||||||
Equation or Equations (Group) | Required |
This keyword or set of keywords (if using Equations group) are used
to compute the strength (rank) of the stereo pair based upon any
value in the combined pair Resource, but is typically applied to
stereo ranking values. The resulting value of the calculation
provided by the Calculator strategy is
stored in the combined Resource as the value specified in the
Result keyword, required when using the single keyword form
or any keyword in the Equations group if you choose to use that
method. This example equation is typically as follows:
# Now specify how the stereo quality is determined. Based upon # ImageStrength and StereoStrength, the maximum (best) rank is 8.0. Equation = "ImageStrengthRank * 1.0 + StereoStrengthRank * 1.0" Result = StereoPairRank |
The following is an example using the StereoPair strategy. In this example we are reading image data from an SQLite database containing WAC and NAC MESSENGER data created from caminfo runs. It determines stereo from overlapping images in the vicinity of Degas crater on Mercury.
Object = IsisMiner Name = DegasStereoPair RequiredParameters = ( inputdir, outputdir ) Object = Strategy Name = ReadDegasPerimeter Type = CsvReader Description = "Read Degas data" CsvFile = "%1/degaus_stereo_perimeter.csv" CsvFileArgs = "inputdir" #Ignore the first several lines including the header. HasHeader = True Identity = "%1" IdentityArgs = "SourceProductId" # Can specify a GIS keyword that will be converted on input GisGeometryKey = GisFootprint GisType = WKB RemoveGisKeywordAfterImport = False EndObject Object = Strategy Type = StereoPair Name = DegasStereoPairs Description = "Find all stereo pairs in the Degas crater region" OverlapMerge = None ShowProgress = True #Debug = True PixelPrecisionMatch = 0.2 Asset = StereoPairs ` # Now specify how the stereo quality is determined. Equation = "imagestrengthrank * 1.0 + stereostrengthrank * 1.0" Result = StereoPairRank # First level contraints/strengths Group = ImageStrength Incidence = ( 30.0, 65.0, 50.0 ) Emission = ( 0.0, 55.0, 15.0 ) Phase = ( 15.0, 120.00, 30.0 ) Resolution = ( 25.0, 5000.0, 200.0 ) EndGroup # Second level constraints/strengths Group = StereoStrength ResolutionRatio = ( 1.0, 2.5, 1.0 ) ParallaxHeightRatio = ( 0.1, 1, 0.4 ) ShadowTipDistance = ( 0.0, 2.58, 0.0 ) OverlapRatio = ( 0.1, 1.0, 0.5 ) EndGroup # This shows the keyword mappings to values required for stereo # matching and constraint values Group = KeywordMap Resolution = PixelResolution Emission = EmissionAngle Incidence = IncidenceAngle Phase = PhaseAngle EndGroup # Select candidates from input list. Matches over itself Object = CandidateMiner Object = Strategy Name = PairCandidates Type = ResourceManager Description = "Choose all Resources to determine stereo" Operations = ResetDiscard EndObject EndObject EndObject Object = Strategy Type = AssetSidebar Name = WriteStereoFiles Description = "Process all stereo pair Assets" Asset = StereoPairs Object = IsisMiner Object = Strategy Name = SortStereoPairs Type = NumericalSort Description = "Sort stereo pairs by StereoPairRank" SortKey = StereoPairRank Order = Descending EndObject Object = Strategy Type = CsvWriter Name = WritePairs Description = "Write individual files with stereo pair data" SkipEmptyLists = True CsvFile = "%1/%2.csv" CsvFileArgs = ( "outputdir", "Identity" ) Keywords = ("SourceProductIdA", "SourceProductIdB", "StereoPairRank", "ImageStrengthRank", "StereoStrengthRank", "ParallaxHeightRatio", "ShadowTipDistance", "ResolutionRatio", "DeltaSolarAzimuth", "DeltaSpacecraftAzimuth", "VerticalPrecision", "StereoAngle", "ImageStrengthRankA", "ImageStrengthRankB", "IncidenceRankA", "IncidenceRankB", "EmissionRankA", "EmissionRankB", "PhaseRankA", "PhaseRankB", "OverlapRatioA", "OverlapRatioB", "YearDoyA", "YearDoyB", "CenterLongitudeA", "CenterLatitudeA", "CenterLongitudeB", "CenterLatitudeB", "PixelResolutionA", "PixelResolutionB", "EmissionAngleA", "EmissionAngleB", "IncidenceAngleA", "IncidenceAngleB", "PhaseAngleA", "PhaseAngleB") EndObject EndObject EndObject EndObject End
Name | Description |
---|---|
CONFIG | Configuration of Strategies that are to be executed. |
GLOBALS | Optional PVL file containing parameters that are loaded into global variable pool. |
PARAMETERS | Optional user global parameters. |
This file contains a series of Strategy objects enclosed in an object named IsisMiner. The main documentation contains many different examples of the format of this file.
Type | filename |
---|---|
File Mode | input |
Filter | *.conf |
All keywords contained in this file are loaded in the global variable pool. Unique keywords that do not exist in Resources can be used as keyword value substitutions in your Strategies.
Type | filename |
---|---|
File Mode | input |
Default | None |
This parameter allows the user to add global parameters to each Strategy so that they can be used in translation of keyword ARG values. This parameter has the form:
PARAMETERS = "key1:value1[@key2:value2...]"Defaults to None.
Type | string |
---|---|
Default | None |
From a global image dataset, find all stereo pairs within a 10x10 degree map section
This example has been used to process MESSENGER MDIS global image dataset. The purpose of this isisminer run is to
File naming conventions used for these section definition files are EquiYYYHXXXED10 where
The section in this example includes Degas crater located at 37.08 degrees latitude and 232.75 positive east longitude. This falls in the section defined in the file Equi035N235ED10.def. Here is the contents of this file:
# Define sub region in 10 degree increments below the NP # Define lower left and upper right coordinates Region = "Equi035N235ED10" RegionGisGeometry = "LINESTRING(230 30, 240 40)" RegionBoundingBox = True RegionCenterLongitude = 235 RegionCenterLatitude = 35 RegionLowerLeftLongitude = 230 RegionLowerLeftLatitude = 30 RegionUpperLeftLongitude = 230 RegionUpperLeftLatitude = 40 RegionUpperRightLongitude = 240 RegionUpperRightLatitude = 40 RegionLowerRightLongitude = 240 RegionLowerRightLatitude = 30 EndThis file is provided in the GLOBALS parameter to isisminer. This will make all keywords in the file available for use in keyword argument substitution, CSV output, and Calculator and Limit Strategies.
To process the global image dataset requires treating data at the poles slightly different when identifying stereo. The incidence angles are consistently high, up to 90 degrees. This requires a slight change to the stereo configuration so two files are required to process the data - one for equatorial data between 65N and 65S degrees and one for polar from 65N/S to 90 N/S degrees. The isisminer configuration file must consider this when processing stereo and it is driven by the map section input file as to which one is selected. The easiest way to do this is to provide the name of the stereo configuration file as a PARAMETER to isisminer. The main isisminer CONFIG file is named mdis_overlaps_and_stereo.conf and its contents with comments is shown in the mdis_overlaps_and_stereo.conf file below.
This configuration file uses the GISOverlap strategy to identify all images that overlap one another and creates a CSV file of those images and a lot of relevant data for computing stereo. The stereo image pairings/sets are determined using a special processing feature of GisOverlaps. It has a provision for immediately processing the result of the overlap determination for each image. Just so happens that the output CSV file contains all the information needed to compute stereo. As each image overlap set is determined and written to the output CSV file, GisOverlap runs the stereo configuration file on that CSV file using the StereoPair strategy. Its configuration is contained in the file mdis_global_stereo_equatorial.conf and is provided in the command line to isisminer in PARAMETERS using the StereoConfig parameter. The contents of mdis_global_stereo_equatorial.conf is shown below.
Finally, there is a complete configuration that has command line control depending upon which latitude region is being processed and input/output directories. To process the entire global dataset of MDIS images collected by MESSENGER, 267,769 of them, will require vast computing Resources. It is highly recommended that a processing cluster be used if available. Below is the command line used to process this map section.
The results summarized here show the output of the command and a summary of the results below.
Creating Strategies... Finished creating 24 strategies... Running DatabaseReader::MDISSelectRegionOct2014 (TimeIn: 02:46:51.626) Description: Strategy::MDISSelectRegionOct2014 is running a DatabaseReader algorithm. 248586 of 0 processed in DatabaseReader::MDISSelectRegionOct2014 (TimeOut: 03:18:36.966) ElapsedTime(s): 1905 Running DatabaseReader::MDISSelectRegionMay2015 (TimeIn: 03:18:36.966) Description: Strategy::MDISSelectRegionMay2015 is running a DatabaseReader algorithm. 19183 of 0 processed in DatabaseReader::MDISSelectRegionMay2015 (TimeOut: 03:20:59.204) ElapsedTime(s): 142 Running IsNumeric::FindBadOnes (TimeIn: 03:20:59.204) Description: Identify images with invalid center pixel geometry 267474 of 267769 processed in IsNumeric::FindBadOnes (TimeOut: 03:21:00.209) ElapsedTime(s): 0 Running ResourceManager::DeleteBadOnes (TimeIn: 03:21:00.209) Description: Delete Resources with bad values in required keywords 295 of 0 processed in ResourceManager::DeleteBadOnes (TimeOut: 03:21:00.247) ElapsedTime(s): 0 Running Limit::SelectRegion (TimeIn: 03:21:00.247) Description: Find all MDIS images with center image geometry in the defined region 557 of 267474 processed in Limit::SelectRegion (TimeOut: 03:21:20.489) ElapsedTime(s): 20 Running RunCommand::MakeOutputDirectory (TimeIn: 03:21:20.489) Description: Ensure output directories exists Running Mkdir1 PRE command: mkdir -p /scratch/kbecker/MESSENGER/Region10D/Global/Images Running Mkdir2 PRE command: mkdir -p /scratch/kbecker/MESSENGER/Region10D/Global/Overlaps/Equi035N235ED10 2 of 0 processed in RunCommand::MakeOutputDirectory (TimeOut: 03:21:20.976) ElapsedTime(s): 0 Running CsvWriter::WriteRegionImages (TimeIn: 03:21:20.976) Description: Write images that are in the defined region to a file 557 of 557 processed in CsvWriter::WriteRegionImages (TimeOut: 03:21:21.398) ElapsedTime(s): 0 Running GisOverlap::ImageOverlap (TimeIn: 03:21:21.398) Description: Find all overlaps images in specfied region 557 of 557 processed in GisOverlap::ImageOverlap (TimeOut: 04:42:28.181) ElapsedTime(s): 4866 Session complete in 01:55:36.555 of elapsed timeThis run completed in about 2 hours. This is typical for most regions when run on machines with similar CPU processing power. There were 557 images identified in this region and the results are stored in /scratch/kbecker/MESSENGER/Region10D/Global/ImagesEqui035N235ED10_images.csv. For each image, and average of several hundred images were determined to overlap. There are now 557 files in the directory /scratch/kbecker/MESSENGER/Region10D/Global/Overlaps/Equi035N235ED10 that have a base name as the image name and _overlaps (example: EW1067331895I_overlaps.csv has 213 overlapping images) with a total of 230,389 image overlaps found. In the directory /scratch/kbecker/MESSENGER/Region10D/Global/Stereo/Equi035N235ED10, there are 308 files (249 had no stereo images or did not pass level1 image constraints for stereo) with similar file naming conventions used in the Overlaps directory (example: EW1067331895I_stereopairs.csv has 11 stereo images) where a total of 7,497 stereo images were identified.
Global map region section definition file | This file contains an example of the keywords used to describe a small section of Mercury as defined in equirectangular map coordinates. These parameters are contained in a file called Equi035N235ED10.def. This file is provided in the GLOBALS parameters making the contents available in the global variable pool. The latitude/longitude boundaries in the file are used to find images where the center latitude/longitude coordinates are contained in the region. |
---|---|
CONFIG file for identifying images that overlap an image |
The contents of this file contain a sequence of Strategy object
definitions used to process the global MESSENGER MDIS image dataset.
The objectives of this configuration is to
|
Sidebar configuration to process stereo | This file contains a sequence of Strategy object definitions used to process overlapping images for stereo potential. This configuration is designed to be run independently or as the GisOverlap optional set of algorithms. The input source is a file containing individual image overlaps. This overlap file is indicated by the user with the input parameter called PARAMETERS. |
Kris Becker | 2013-02-19 | Original Version |
Kris Becker | 2015-02-24 | Update/complete documentation and regression tests |
Jeffrey Covington | 2015-02-26 | Added documentation for the IsNumeric strategy |
Ian Humphrey | 2015-03-03 | Added documentation and tests for the NumericalSort strategy |
Makayla Shepherd | 2015-03-04 | Added documentation for the Limit strategy |
Jeffrey Covington | 2015-03-04 | Added documentation for the CsvWriter strategy |
Jeffrey Covington | 2015-03-06 | Added documentation for the CsvReader strategy |
Makayla Shepherd | 2015-03-13 | Added documentation for PdsTableFormat strategy |
Ian Humphrey | 2015-03-16 | Added documentation for CnetReader strategy |
Kristin Berry | 2015-03-23 | Added documentation and tests for the ResourceManager strategy |
Ian Humphrey | 2015-03-24 | Added documentation for PvlReader strategy |
Jeannie Backer | 2015-03-27 | Added a List of Strategies table of contents with links to each individual description. Added links from individual Strategy descriptions back to table of contents. |
Makayla Shepherd | 2015-03-27 | Added documentation for PdsTableCreator strategy. Fixed documentation for PdsTableFormat strategy. |
Jeffrey Covington | 2015-03-31 | Added documentation for the Calculator strategy |
Kristin Berry | 2015-04-07 | Merged GisBasic strategy with GisIntersect strategy. Added documentation and tests for GisIntersect strategy. Modified Strategy:::applyToIntersectedGeometry in Strategy.cpp to deal with the case where no Resources are active. |
Jeffrey Covington | 2015-04-09 | Added documentation for the Sidebar strategy |
Jeffrey Covington | 2015-04-16 | Updated documentation for the CSV strategies. |
Kris Becker | 2015-04-27 | Added GisOverlap strategy and documentation for it; added ShowProgress keywords to allow users to selectively show the progress status percentage/bar for any Strategy that contains this keyword; updated Resource and Strategy code with new additions/modifications. |
Jeffrey Covington | 2015-04-28 | Added documentation for the Database strategy. |
Kris Becker | 2015-04-29 | Renamed the Database strategy to DatabaseReader |
Andrew Stebenne | 2015-05-01 | Added documentation and app tests for the GisUnion strategy. |
Jeffrey Covington | 2015-05-01 | Updated the DatabaseReader strategy documentation. |
Ian Humphrey | 2015-05-01 | Added documentation for the AssetSidebar strategy |
Makayla Shepherd | 2015-05-21 | Added keyword format option (KeywordFormatFile) to PdsTableFormat strategy. |
Kris Becker | 2015-05-25 | Added the ability to code Strategy objects using plugins and added the StereoPair and RunCommand (plugin) Strategies. |
Kris Becker | 2015-06-05 | Updated Calculator strategy to include all global parameters in variable resolution; fixed StereoPair computation of stereo angle; implemented check in StrategyFactory to validate RequiredParameters are define in the global variable pool and updated app test configuration files to use this feature. |
Kris Becker | 2015-06-12 | Improved GisOverlap and StereoPair strategies. StereoAngle can now be computed from the GIS centroid of the overlapping region, providing a much higher degree of accuracy. Improved global variable pool management whilst traversing through Strategy/Resource depths in the mining process. Ensure Asset is cleared when Mode = Create is used in AssetSideBar strategy. |
Kris Becker | 2015-06-17 | Fixed bug in creation of GEOS SRTree when only one geometry occurs (requires two or more). Changed name of IsisMiner objects within the GisOverlap strategy (since we now have two of them) to CandidateMiner and OverlapMiner. Added feature to simplify geometry whilst preserving topology using GEOS' Douglas-Peucker algorithm when reading in geometries. It is automatically detected in Strategies like CsvReader and DatabaseReader. Added many new parameters to various Strategies. |
Kris Becker | 2015-06-28 | Add ability to process the set of GisOverlap results for each Resource as they are matched. This is handled in the OverlapMiner Strategy object. |
Kris Becker | 2015-07-07 | Added more content to documentation and added an example demonstrating a real application of the isisminer application. |
Kris Becker | 2015-09-27 | Refactored the Resource class to contain all but the active status so that copies can be maintained in separate instances. Reworked the AssetSidebar strategy to take advantage of this work. |
Kris Becker | 2015-10-11 | Fixed bug in argument scanning when the number of arguments reached 10 or more as %1 was replacing %10, etc... Scan and replace i reverse order corrected this problem. Enhanced the Calculator strategy to provide argument replacement in the Initializers group. |
Kris Becker | 2015-11-01 | Modified CsvReader strategy to have the provided header length determine the number of keywords created/Resource. This allows the jigsaw residual file to read the rejected column. |