ISIS Documentation

ISIS Application and Category Test How-to

Guide to writing an application and category tests

Overview

Isis Application Test files are used to ensure that an application is working correctly. Isis Category Test files are used to ensure that a category is working correctly. These tests are especially important when porting to a new operating system, or changing compilers. They are also important when changes are made to the entire system (such as refactoring) or to other files that directly affect the application. If something is changed that affects the outcome of the application, the test will fail, making the programmer aware that something has changed and needs to be examined. This guide focuses on the specific task of adding tests to an existing application or category.

Questions & Answers

What do I need to know?

You need to be familiar with what the application or category does in order to test it properly. Also a basic knowledge of make would be helpful but not required.

What is make?

Make is a programming tool used to automate the build process. Make is used used to build C/C++ projects.

Make Basics

Targets

Make is a text file that is broken down into what is known as targets. Targets represent a series of actions as a single word. Targets are distinguished by the colon ( : ) after the target name

An action is a single command that is terminated by a semicolon ( ; ). If a command must be on multiple lines, each line must end with a backslash ( \ ).

            
              clean:
              	ratio num=isisTruth.cub+1 \
              	den=isisTruth.cub+2 \
              	to=temp.cub;

If there is any character, after the backslash (even a space), the command will end at that line. Make will not show an error but the command will not run as intended.

Each line in a target must be preceded with a tab character. Make will show an error if there is not a tab at the front of each line in a target.

Variables

Make allows for variables to be used to store values. These variables can be set at the command line by using the syntax VARIABLE=value. For example, setting the variable clean would be CLEAN=yes. Variables are usually in all capital letters. Variables are accessed using the $( ) syntax. For example, using the clean variable would be $(CLEAN).

Writing a Test for an application

Test Structure

A test is composed of input, output, truth and make files. All of the files the test needs as input are placed in the input directory. Everything the test generates goes into the output directory. When the test is working correctly the output files representing the base truth files of the test are placed in the truth directory. The make file contains the commands to run the test.

Test Results

Tests are to ensure that the programs are running correctly. This is done by comparing the output files generated by the test with the files designated as the truth for the test. If the files have the same contents the test shows an OK status. If the files are different in some way the test shows a FAILED status.

Setup Tests

Go to the directory where the application directory is located
Type make testdir

This creates a directory call tsts
Places a Makefile for creating tests in the tsts directory

Setup Individual Test

Within the tsts directory type make newtest TEST=testname

Where testname is the name of the test. Testname should decribe what the test is testing for.
e.g. if the program has a variable mode that can be ALL or NONE, the test for mode all would be named all and the test for mode none would be none.
The command would be make newtest TEST=all
This creates a directory called testname and places a template makefile, and an input directory in that directory

Type cd testname to change directories to the test

          
            APPNAME =

            include $(ISISROOT)/make/isismake.tsts

            commands:
            > /dev/null;

APPNAME - name of the application being tested
include - includes all targets from the isismake.tsts file
commands - defines the commands target needed to run the test
/dev/null - all actions are placed before or on this line

Write Test

APPNAME - type in the application name after the equals. Capitalization and spelling count here.

The excetable for the test resides two directories above the test. APPNAME is adjusted during the run of the test to take care of this. APPNAME should just be the name of the program. Each time the test is run, the current version of the executable is used. Running the test will not remake the executable.

Write in commands before the > /dev/null.

Remember

if multiple commands are needed than each needs > /dev/null; at the end of the commands
beginning of each line needs a tab character
if a command goes onto multiple lines, end of each line needs a \ with nothing after it

Type $(APPNAME) where the need of the current application is being used. Other program names should just be the name of the program.
Copy any input files to the input directory
Set all paths in the test so that the input files have $(INPUT) before the filename
Set all paths in the test so that the output files have $(OUTPUT) before the filename

Files that are to be compared for the test must be in the output directory

If the test is testing standardout, the output must be redirected to a text file. Change the /dev/null to the name of a text file for comparison. See cathist for an example of this.
To generate the output files for the test type make output
Add variables as needed to the test. Variables are defined in the next section
When the output file looks good type make truthdata

if the test is operating system dependent test than do make ostruthdata

To run the test, type make test . At this point the test shoud pass. If not, revise the test till it does.

Test Variables

Tests deal with files. Variables for the tests are strings attached to the end of file names. For example if the file is test.txt then the variable IGNORELINES would be test.txt.IGNORELINES. The file portion must match spelling and capitalization of the file it is assoiciated with. The variable portion (.IGNORELINES) needs to be in all capital letters. For each file in the test if the variable is not set than a default value is used.

Possible variables for file types are:

Cub files (.cub)

.TOLERANCE

Defines the difference in dn values allowed between the cubes generated by the test and the truth cubes, to allow the test to pass
Value - number, usually a small decimal value
Defaults - 0, meaning that the two dn values have to match exactly
For an example, see the mpp test in the cam2map application

.IGNORESPECIAL

Designates whether or not to ignore special pixels in comparing two cubes
Value - yes or no value
Defaults - no, special pixels will be used in the comparison of two cube files

Text Files (.txt)

.SKIPLINES

Defines the number of lines to skip at the beginning of a text file
Value - number, integer
Defaults - 0, meaning that the whole file it to be used
For an example, see the cnet test in the coreg application

.IGNORELINES

Defines the lines of text to omit from the output file
Value - list of word at the beginning of the lines to omit
Defaults - empty list, use the whole file
For an example, see the circle test in the isisui application

Note: SKIPLINES occurs before IGNORELINES

Binary Files (.jpg .tif .bin ...)

.BINSKIP

Defines the number of bytes to skip at the start of an input file
Value - number, integer
Defaults - 0, meaning that the whole file is to be used

.BINCOUNT

Defines the number of bytes to copy to the output file
Value - number, integer
Defaults - 0, meaning that the whole file is to be used

Pvl Files (.pvl)

.SKIPLINES, .IGNORELINES

Because a pvl file is essentially a formatted text file, test variables that work on text files will work the same way on pvl files

.DIFF

See Pvl Files for setting variables when comparing pvl files.

Test Data

In order to perserve the data that is used by the test, type make checkin . This copies the contents of the input and truth directories to the testdata area. Since the output files can be generated by using the output command, these files do not need to be saved. Once the files are stored in the testdata area you can do make release to remove all of the files exept the makefile. Then the makefile can be checked into cvs. To restore these files back into the test, type make checkout . This copies the files into the proper directories. Once the files for the test are in the test data area, the test will still run even if the test files are not in the same directory as the test.

Writing a Test for a category

Category tests work the same way as application tests do. The difference is that category tests don't test a single program but a sequence of programs. Therefore the APPNAME variable should not be used. Instead use variables such as APP1NAME, APP2NAME, etc. In addition, all ISIS application names need to be followed by $(TSTARGS) to guarantee proper testing parameters. Currently, this ensures that they use the default preference file. All of the variables for testing an application also work for category tests. A brief example is shown below.

          
            APP1NAME = clem2isis
	    APP2NAME = spiceinit
	    APP3NAME = campt
	    APP4NAME = getsn

	    include $(ISISROOT)/make/isismake.tsts

	    commands:
	    	$(APP1NAME) $(TSTARGS) from= $(INPUT)/lna3819k.045 \
	    	to= $(OUTPUT)/lna3819k.045--clem2isis-spiceinit.cub > /dev/null

	    	$(APP2NAME) $(TSTARGS) from= $(OUTPUT)/lna3819k.045--clem2isis-spiceinit.cub \
	    	> /dev/nul

Comparing Files

PVL

In order to assure the most accurate testing possible, pvl (parameter value language) files are compared by parsing through the groups, objects and keywords in the file. Any file with a .pvl extension is considered a pvl file. Because pvl files contain text, the text file test variables will work on the pvl files; it is not recommended to use these test variables, however, and the result must still be a valid pvl file. The order of the keywords, objects and groups matter and may not be different in any case. In order to set tolerances and ignore keywords, a pvl tolerance file with .DIFF attached to the end will be used. For example, if if file is test.pvl then the tolerance file must be named test.pvl.DIFF. These files must only exist in the input data directory. Inside the pvl tolerance file you can specify numerical tolerances and keywords to ignore. Inside the pvl tolerance file, there should be one or two groups, Tolerances and IgnoreKeys, in the root of the file. So, a basic pvl tolerance file looks like this:

            
              Group = Tolerances
              End_Group

              Group = IgnoreKeys
              End_Group

              Group = IgnoreFilePaths
              End_Group

              End

To set a tolerance on a number, inside the Tolerances group there must be a keyword in this format:

            
              KeywordName = MaximumDifference

If the output and truth data values exceed the maximum difference (tolerance), then the files are not considered the same. In order to ignore a non-numerical value, such as a file name, inside the IgnoreKeys group there must be a keyword in this format:

            
              KeywordName = true

Note: The value of this keyword does not matter. To ignore a file's path up until the name, Put a corresponding keyword set to true inside the IgnoreFilePaths group.

            
              KeywordName = true

With these formats in mind, to give a tolerance for the keyword StandardDeviation of 0.0000000001 and ignore the keyword FileName the pvl tolerance file would look like this:

            
              Group = Tolerances
              StandardDeviation = 0.0000000001
              End_Group

              Group = IgnoreKeys
              FileName = true
              End_Group

              Group = IgnoreFilePaths
              FilePath = true
              End_Group

              End

To debug these files, or see why two pvl files compare as different, you can use pvldiff. The FROM parameter will be the output file (which will be created with the command make output), the FROM2 parameter will be the truth file and the DIFF parameter will be the DIFF file inside of the input folder (the same as the FROM2 parameter with .DIFF added on to the end). When pvldiff is run, it will output the first difference it finds that exceeds the tolerances.
For an example of the pvl test, say we have the pvl file cubelabels.pvl that looks like this:

            
              Object = IsisCube
              Object = Core
              SomeFile = /foo/bar/fakefile.cub
              StartByte   = 65537
              Format      = Tile
              TileSamples = 128
              TileLines   = 128

              Group = Dimensions
              Samples = 1024
              Lines   = 1024
              Bands   = 7
              End_Group

              Group = Pixels
              Type       = Real
              ByteOrder  = LSB
              Base       = 0.0
              Multiplier = 1.0
              End_Group
              End_Object
              End_Object

              Object = SomeObject
              SomeCalculaion = 5536.4363463414
              End_Object

              End

The SomeCalculation keyword may vary by 0.000000001 and the ByteOrder could be MSB or LSB. SomeFile could have a different path if the tester is not using the default data area. To compensate for this, you would add a file named cubelabels.pvl.DIFF in the input folder with the following:

            
              Group = Tolerances
              SomeCalculation = 0.000000001
              End_Group

              Group = IgnoreKeys
              ByteOrder = true
              End_Group

              Group = IgnoreFilePaths
              SomeFile = true
              End_Group

              End

This would ignore the value of ByteOrder always, the path of SomeFile, and the value of SomeCalculation if it were within the tolerance.

Finally, you can create unique tolerances and ignore keys for each element of an array contained within a keyword, like so:

            
              Group = Tolerances
              SomeArray = (0.000000001, 0.0, 0.025)
              AnotherArray = 0.000000001
              End_Group

              Group = IgnoreKeys
              SomeArray =(false, true, false)
              End_Group

              Group = IgnoreFilePaths
              SomeArray = (false, false, true)
              End_Group

              End

In the above case, the keyword SomeArray would have a tolerance of 0.000000001 applied to the first element, a tolerance of 0.025 applied to the third element, and the second element would be ignored entirely.

Just make sure when working with arrays that you have exactly the same number of tolerances/ignore keys for the keyword as there are elements in the array. Alternatively, you can list only one tolerance or ignore key for the keyword, and it will be applied to every element in the array. In the above case, the keyword AnotherArray could have 50 elements, but each element would have a tolerance of 0.000000001 applied to it. Having multiple tolerances and ignore keys without matching the number of array elements, however, will result in an error.

Comma-Separated Values (CSV)

A similar tool exists for comparing two comma-separated value (CSV) files as those for PVL files and cubes. When writing application tests, any file output with a ".csv" extension will automatically be compared against the truth CSV using the CSV-Diff script. The benefit of using this script over a one-to-one textual comparison comes into play when the CSV files being compared contain numerical data.

By specifying tolerance values in a DIFF file contained within the test's input directory, one can consider two CSV files "the same" within some margin of error. The DIFF fle uses a similar naming convention as that for PVL diffing: csvfile.csv.DIFF. The tolerance file must contain one tolerance per line of the form:

            
              COLUMN=VALUE

where COLUMN is the name of the tolerance value for every cell in that column, and VALUE is the actual tolerance that will be compared against the difference between the values in the two files.

The script can also be used by itself, independent of any application test. This is most useful when a test fails, and the user wishes to run the CSV diff directly in order to see the reason why. Running the script directly can be accomplished with the following usage:

            
              $ISISROOT/scripts/csvdiff.py CSV1 CSV2 [TOLERANCES]

Output will be a message stating either SUCCESS or FAILURE followed by an explanation of why. Difference failures will also include a reference to the exact line and column where the offending difference occurred. The program will terminate with an error signal when a failure occurs during processing.

Before any cells are compared the program will first ensure that the two CSV files to be compared have the same number of columns and rows, additionally checking that their headers match.

Future versions will support an option for submitting files without headers.

Modifying an Existing Application Test

Checkout the application from the cvs repository.
Follow the steps above to setup the tests.

Copy the commands from the xml file for the application test below the word commands and above the /dev/null;.

If the application test looks like this:

                
                  <test>
                  <commandLine>
                  num=\$ISISDATA/base/examples/isisTruth.cub+1
                  den=\$ISISDATA/base/examples/isisTruth.cub+2
                  to=<temp>temp1.cub</temp>
                  </commandLine>

                  <cubes>
                  <compareCube>
                  <truth>ratioTruth1.cub</truth>
                  <against>temp1.cub</against>
                  </compareCube>
                  </cubes>
                  </test>

Then the test would look like:

                
                  APPNAME =

                  include $(ISISROOT)/make/isismake.tsts

                  commands:
                  	num=\$ISISDATA/base/examples/isisTruth.cub+1
                  den=\$ISISDATA/base/examples/isisTruth.cub+2
                  to=<temp>temp1.cub</temp> > /dev/null;

Add a \ to the end of each line except the last one that has the ; at the end.

The test now looks like:

                
                  APPNAME =

                  include $(ISISROOT)/make/isismake.tsts

                  commands:
                  	num=\$ISISDATA/base/examples/isisTruth.cub+1 \
                  den=\$ISISDATA/base/examples/isisTruth.cub+2 \
                  to=<temp>temp1.cub</temp> > /dev/null;

Delete all spaces at the front of each line, add a tab to the front of each line.

The test now looks like:

                
                  APPNAME =

                  include $(ISISROOT)/make/isismake.tsts

                  commands:
                  	num=\$ISISDATA/base/examples/isisTruth.cub+1 \
                  	den=\$ISISDATA/base/examples/isisTruth.cub+2 \
                  	to=<temp>temp1.cub</temp> > /dev/null;

Set APPNAME to the name of the application being tested. Add $(APPNAME) to the front of the first line of each command that is using the application that is being tested. Programs that are being used by the test but are not being tested just need the name of the program.

The test now looks like:

                
                  APPNAME = ratio

                  include $(ISISROOT)/make/isismake.tsts

                  commands:
                  	$(APPNAME) num=\$ISISDATA/base/examples/isisTruth.cub+1 \
                  	den=\$ISISDATA/base/examples/isisTruth.cub+2 \
                  	to=<temp>temp1.cub</temp> > /dev/null;

Replace the path of input file names to be $(INPUT).

The test now looks like:

                
                  APPNAME = ratio

                  include $(ISISROOT)/make/isismake.tsts

                  commands:
                  	$(APPNAME) num=$(INPUT)/isisTruth.cub+1 \
                  	den=$(INPUT)/isisTruth.cub+2 \
                  	to=<temp>temp1.cub</temp> > /dev/null;

Replace <temp></temp> files with name in the truth tag for the application test

The test now looks like:

                
                  APPNAME = ratio

                  include $(ISISROOT)/make/isismake.tsts

                  commands:
                  	$(APPNAME) num=$(INPUT)/isisTruth.cub+1 \
                  	den=$(INPUT)/isisTruth.cub+2 \
                  	to=ratioTruth1.cub > /dev/null;

Replace the path of output file names to be $(OUTPUT)

The test now looks like:

                
                  APPNAME = ratio

                  include $(ISISROOT)/make/isismake.tsts

                  commands:
                  	$(APPNAME) num=$(INPUT)/isisTruth.cub+1 \
                  	den=$(INPUT)/isisTruth.cub+2 \
                  	to=$(OUTPUT)/ratioTruth1.cub > /dev/null;

Add variables as needed

e.g. add a tolerance to the output cube would be ratioTruth1.cub.TOLERANCE = someValue

Copy input files into the input directory

In the example above copy isisTruth.cub into the input directory from the $ISISDATA/base/examples area.

Copy the truth file from the application test to the truth directory

In the example above copy ratioTruth1.cub into the truth directory from the appTest directory of the ratio application

Type make test
The test should have the same results as the application test did.
Type make checkin to copy the data to the test data area
Type make release to clean up the test directory
Checkin the Makefile into the cvs repository

Modifying a Test

Checkout the Makefile from the cvs repository
Type make checkout to get the data for the test
Modify the test as needed
Type make test to see the current results of the test
If the truth files need to be remade type make truthdata or make ostruthdata as described above
When done, type make checkin to copy the data back to the testdata area
Type make release to remove the files
Check in the Makefile back into the cvs repository

Document History

Robert Wallace	2006-09-08	Created
Robert Wallace	2006-10-06	Updated with recent changes to testing system and added examples
Travis Addair	2011-05-13	Added section for Comparing Files, moved PVL diff discussion into this section and added new discussion for CSV file diffing.
Jesse Mapel	2016-04-05	Updated section on category tests. Fixes #3884.

User Documentation

Contributor Documentation

Quick Links

Documentation Versions

ISIS 2

ISIS Application and Category Test How-to

Guide to writing an application and category tests

Overview

Questions & Answers

What do I need to know?

What is make?

Make Basics

Targets

Variables

Writing a Test for an application

Test Structure

Test Results

Setup Tests

Setup Individual Test

Write Test

Test Variables

Test Data

Writing a Test for a category

Comparing Files

PVL

Comma-Separated Values (CSV)

Modifying an Existing Application Test

Modifying a Test

Document History