Isis 3 Documentation

Isis Application and Category Test How-to

Guide to writing an application and category tests

Overview

Isis Application Test files are used to ensure that an application is working correctly. Isis Category Test files are used to ensure that a category is working correctly. These tests are especially important when porting to a new operating system, or changing compilers. They are also important when changes are made to the entire system (such as refactoring) or to other files that directly affect the application. If something is changed that affects the outcome of the application, the test will fail, making the programmer aware that something has changed and needs to be examined. This guide focuses on the specific task of adding tests to an existing application or cateogry.

Questions & Answers

What do I need to know?

You need to be familiar with what the application or category does in order to test it properly. Also a basic knowledge of make would be helpful but not required.

What is make?

Make is a programming tool used to automate the build process. Make is used used to build C/C++ projects.

Make Basics

Targets

Make is a text file that is broken down into what is known as targets. Targets represent a series of actions as a single word. Targets are distinguished by the colon ( : ) after the target name

An action is a single command that is terminated by a semicolon ( ; ). If a command must be on multiple lines, each line must end with a backslash ( \ ).

                        
                            clean:
                            	ratio num=isisTruth.cub+1 \
                            	den=isisTruth.cub+2 \
                            	to=temp.cub;

If there is any character, after the backslash (even a space), the command will end at that line. Make will not show an error but the command will not run as intended.

Each line in a target must be preceded with a tab character. Make will show an error if there is not a tab at the front of each line in a target.

Variables

Make allows for variables to be used to store values. These variables can be set at the command line by using the syntax VARIABLE=value. For example, setting the variable clean would be CLEAN=yes. Variables are usually in all capital letters. Variables are accessed using the $( ) syntax. For example, using the clean variable would be $(CLEAN).

Writing a Test for an application

Test Structure

A test is composed of input, output, truth and make files. All of the files the test needs as input are placed in the input directory. Everything the test generates goes into the output directory. When the test is working correctly the output files representing the base truth files of the test are placed in the truth directory. The make file contains the commands to run the test.

Test Results

Tests are to ensure that the programs are running correctly. This is done by comparing the output files generated by the test with the files designated as the truth for the test. If the files have the same contents the test shows an OK status. If the files are different in some way the test shows a FAILED status.

Setup Tests

Go to the directory where the application directory is located
Type make testdir

This creates a directory call tsts
Places a Makefile for creating tests in the tsts directory

Setup Individual Test

Within the tsts directory type make newtest TEST=testname

Where testname is the name of the test. Testname should decribe what the test is testing for.
e.g. if the program has a variable mode that can be ALL or NONE, the test for mode all would be named all and the test for mode none would be none.
The command would be make newtest TEST=all
This creates a directory called testname and places a template makefile, and an input directory in that directory

Type cd testname to chage directories to the test

                    
                        APPNAME =

                        include $(ISISROOT)/make/isismake.tsts

                        commands:
                        > /dev/null;

APPNAME - name of the application being tested
include - includes all targets from the isismake.tsts file
commands - defines the commands target needed to run the test
/dev/null - all actions are placed before or on this line

Write Test

APPNAME - type in the application name after the equals. Capitalization and spelling count here.

The excetable for the test resides two directories above the test. APPNAME is adjusted during the run of the test to take care of this. APPNAME should just be the name of the program. Each time the test is run, the current version of the executable is used. Running the test will not remake the executable.

Write in commands before the > /dev/null.

Remember

if multiple commands are needed than each needs > /dev/null; at the end of the commands
beginning of each line needs a tab character
if a command goes onto multiple lines, end of each line needs a \ with nothing after it

Type $(APPNAME) where the need of the current application is being used. Other program names should just be the name of the program.
Copy any input files to the input directory
Set all paths in the test so that the input files have $(INPUT) before the filename
Set all paths in the test so that the output files have $(OUTPUT) before the filename

Files that are to be compared for the test must be in the output directory

If the test is testing standardout, the output must be redirected to a text file. Change the /dev/null to the name of a text file for comparison. See cathist for an example of this.
To generate the output files for the test type make output
Add variables as needed to the test. Variables are defined in the next section
When the output file looks good type make truthdata

if the test is operating system dependent test than do make ostruthdata

To run the test, type make test . At this point the test shoud pass. If not, revise the test till it does.

Test Variables

Tests deal with files. Variables for the tests are strings attached to the end of file names. For example if the file is test.txt then the variable IGNORELINES would be test.txt.IGNORELINES. The file portion must match spelling and capitalization of the file it is assoiciated with. The variable portion (.IGNORELINES) needs to be in all capital letters. For each file in the test if the variable is not set than a default value is used.

Possible variables for file types are:

Cub files (.cub)

.TOLERANCE

Defines the difference in dn values allowed between the cubes generated by the test and the truth cubes, to allow the test to pass
Value - number, usually a small decimal value
Defaults - 0, meaning that the two dn values have to match exactly
For an example, see the mpp test in the cam2map application

.IGNORESPECIAL

Designates whether or not to ignore special pixels in comparing two cubes
Value - yes or no value
Defaults - no, special pixels will be used in the comparison of two cube files

Text Files (.txt)

.SKIPLINES

Defines the number of lines to skip at the beginning of a text file
Value - number, integer
Defaults - 0, meaning that the whole file it to be used
For an example, see the cnet test in the coreg application

.IGNORELINES

Defines the lines of text to omit from the output file
Value - list of word at the beginning of the lines to omit
Defaults - empty list, use the whole file
For an example, see the circle test in the isisui application

Note: SKIPLINES occurs before IGNORELINES

Binary Files (.jpg .tif .bin ...)

.BINSKIP

Defines the number of bytes to skip at the start of an input file
Value - number, integer
Defaults - 0, meaning that the whole file is to be used

.BINCOUNT

Defines the number of bytes to copy to the output file
Value - number, integer
Defaults - 0, meaning that the whole file is to be used

Pvl Files (.pvl)

.SKIPLINES, .IGNORELINES

Because a pvl file is essentially a formatted text file, test variables that work on text files will work the same way on pvl files

.DIFF

See Pvl Files for setting variables when comparing pvl files.

Pvl Files

In order to assure the most accurate testing possible, pvl (parameter value language) files are compared by parsing through the groups, objects and keywords in the file. Any file with a .pvl extension is considered a pvl file. Because pvl files contain text, the text file test variables will work on the pvl files; it is not recommended to use these test variables, however, and the result must still be a valid pvl file. The order of the keywords, objects and groups matter and may not be different in any case. In order to set tolerances and ignore keywords, a pvl tolerance file with .DIFF attached to the end will be used. For example, if if file is test.pvl then the tolerance file must be named test.pvl.DIFF. These files must only exist in the truth data directories. Inside the pvl tolerance file you can specify numerical tolerances and keywords to ignore. Inside the pvl tolerance file, there should be one or two groups, Tolerances and IgnoreKeys, in the root of the file. So, a basic pvl tolerance file looks like this:

                   
Group = Tolerances
End_Group

Group = IgnoreKeys
End_Group

End

To set a tolerance on a number, inside the Tolerances group there must be a keyword in this format:

                   
KeywordName = MaximumDifference

If the output and truth data values exceed the maximum difference (tolerance), then the files are not considered the same. In order to ignore a non-numerical value, such as a file name, inside the IgnoreKeys group there must be a keyword in this format:

                   
KeywordName = true

Note: The value of this keyword does not matter.
With these formats in mind, to give a tolerance for the keyword StandardDeviation of 0.0000000001 and ignore the keyword FileName the pvl tolerance file would looks like this:

                   
Group = Tolerances
  StandardDeviation = 0.0000000001
End_Group

Group = IgnoreKeys
  FileName = true
End_Group

End

To debug these files, or see why two pvl files compare as different, you can use pvldiff. The FROM parameter will be the output file (which will be created with the command make output), the FROM2 parameter will be the truth file and the DIFF parameter will be the DIFF file inside of the truth folder (the same as the FROM2 parameter with .DIFF added on to the end). When pvldiff is run, it will output the first difference it finds that exceeds the tolerances.
For an example of the pvl test, say we have the pvl file cubelabels.pvl that looks like this:

                   
Object = IsisCube
  Object = Core
    StartByte   = 65537
    Format      = Tile
    TileSamples = 128
    TileLines   = 128

    Group = Dimensions
      Samples = 1024
      Lines   = 1024
      Bands   = 7
    End_Group

    Group = Pixels
      Type       = Real
      ByteOrder  = LSB
      Base       = 0.0
      Multiplier = 1.0
    End_Group
  End_Object
End_Object

Object = SomeObject
  SomeCalculaion = 5536.4363463414
End_Object

End

The SomeCalculation keyword may vary by 0.000000001 and the ByteOrder could be MSB or LSB. To compensate for this, you would add a file named cubelabels.pvl.DIFF in the truth folder with the following:

                   
Group = Tolerances
  SomeCalculation = 0.000000001
End_Group

Group = IgnoreKeys
  ByteOrder = true
End_Group

End

This would ignore the value of ByteOrder always and SomeCalculation if it were within the tolerance.

Test Data

In order to perserve the data that is used by the test, type make checkin . This copies the contents of the input and truth directories to the testdata area. Since the output files can be generated by using the output command, these files do not need to be saved. Once the files are stored in the testdata area you can do make release to remove all of the files exept the makefile. Then the makefile can be checked into cvs. To restore these files back into the test, type make checkout . This copies the files into the proper directories. Once the files for the test are in the test data area, the test will still run even if the test files are not in the same directory as the test.

Writing a Test for a category

Category tests work the same way as application tests do. The difference is that category tests don't test a single program but a sequence of programs. Therefore the APPNAME isn't needed for a category test and can be deleted safely. All of the variables for testing an application also work for the category test.

Modifying an Existing Application Test

Checkout the application from the cvs repository.
Follow the steps above to setup the tests.

Copy the commands from the xml file for the application test below the word commands and above the /dev/null;.

If the application test looks like this:

                                
<test>
  <commandLine>
    num=\$base/testData/isisTruth.cub+1
    den=\$base/testData/isisTruth.cub+2
    to=<temp>temp1.cub</temp>
  </commandLine>

  <cubes>
    <compareCube>
      <truth>ratioTruth1.cub</truth>
      <against>temp1.cub</against>
    </compareCube>
  </cubes>
</test>

Then the test would look like:

                                
APPNAME =

include $(ISISROOT)/make/isismake.tsts

commands:
	num=\$base/testData/isisTruth.cub+1
    den=\$base/testData/isisTruth.cub+2
    to=<temp>temp1.cub</temp> > /dev/null;

Add a \ to the end of each line except the last one that has the ; at the end.

The test now looks like:

                                
APPNAME =

include $(ISISROOT)/make/isismake.tsts

commands:
	num=\$base/testData/isisTruth.cub+1 \
    den=\$base/testData/isisTruth.cub+2 \
    to=<temp>temp1.cub</temp> > /dev/null;

Delete all spaces at the front of each line, add a tab to the front of each line.

The test now looks like:

                                
APPNAME =

include $(ISISROOT)/make/isismake.tsts

commands:
	num=\$base/testData/isisTruth.cub+1 \
	den=\$base/testData/isisTruth.cub+2 \
	to=<temp>temp1.cub</temp> > /dev/null;

Set APPNAME to the name of the application being tested. Add $(APPNAME) to the front of the first line of each command that is using the application that is being tested. Programs that are being used by the test but are not being tested just need the name of the program.
The test now looks like:
```
                                
APPNAME = ratio

include $(ISISROOT)/make/isismake.tsts

commands:
	$(APPNAME) num=\$base/testData/isisTruth.cub+1 \
	den=\$base/testData/isisTruth.cub+2 \
	to=<temp>temp1.cub</temp> > /dev/null; 
                            
```

Replace the path of input file names to be $(INPUT).

The test now looks like:

                                
APPNAME = ratio

include $(ISISROOT)/make/isismake.tsts

commands:
	$(APPNAME) num=$(INPUT)/isisTruth.cub+1 \
	den=$(INPUT)/isisTruth.cub+2 \
	to=<temp>temp1.cub</temp> > /dev/null;

Replace <temp></temp> files with name in the truth tag for the application test

The test now looks like:

                                
APPNAME = ratio

include $(ISISROOT)/make/isismake.tsts

commands:
	$(APPNAME) num=$(INPUT)/isisTruth.cub+1 \
	den=$(INPUT)/isisTruth.cub+2 \
	to=ratioTruth1.cub > /dev/null;

Replace the path of output file names to be $(OUTPUT)

The test now looks like:

                                
APPNAME = ratio

include $(ISISROOT)/make/isismake.tsts

commands:
	$(APPNAME) num=$(INPUT)/isisTruth.cub+1 \
	den=$(INPUT)/isisTruth.cub+2 \
	to=$(OUTPUT)/ratioTruth1.cub > /dev/null;

Add variables as needed

e.g. add a tolerance to the output cube would be ratioTruth1.cub.TOLERANCE = someValue

Copy input files into the input directory

In the example above copy isisTruth.cub into the input directory from the $ISIS3DATA/base/testData area.

Copy the truth file from the application test to the truth directory

In the example above copy ratioTruth1.cub into the truth directory from the appTest directory of the ratio application

Type make test
The test should have the same results as the application test did.
Type make checkin to copy the data to the test data area
Type make release to clean up the test directory
Checkin the Makefile into the cvs repository

Modifying a Test

Checkout the Makefile from the cvs repository
Type make checkout to get the data for the test
Modify the test as needed
Type make test to see the current results of the test
If the truth files need to be remade type make truthdata or make ostruthdata as described above
When done, type make checkin to copy the data back to the testdata area
Type make release to remove the files
Check in the Makefile back into the cvs repository

Document History

Robert Wallace	2006-09-08	Created
Robert Wallace	2006-10-06	Updated with recent changes to testing system and added examples

Isis 3

Isis 2

Search

Isis Application and Category Test How-to

Guide to writing an application and category tests

Overview

Questions & Answers

What do I need to know?

What is make?

Make Basics

Targets

Variables

Writing a Test for an application

Test Structure

Test Results

Setup Tests

Setup Individual Test

Write Test

Test Variables

Pvl Files

Test Data

Writing a Test for a category

Modifying an Existing Application Test

Modifying a Test

Document History