Adding a SABER test

SABER has an automated testing system which will produce a serial and a 2-processor parallel version of each test. This helps to catch errors and bugs which are sensitive to how the the grid is partitioned. To add a new test in SABER, follow these steps:

Create test YAML

The test YAML file sets the test configuration. See the other tests in the test/testinput directory for examples.

The name for the new test (and filename for the new YAML file) should begin with the name of the QUENCH executable that will run the test. Current testing executables are: convertstate, process_perts, randomization, error_covariance_training, and dirac. The last three executables are provided by the quenchErrorCovarianceToolbox application.

Create testdeps file

The testdeps is a .txt which lists any tests on which the new test will depend. If the new test has no dependencies, this will be a file with just one blank line. Add this in the test/testdeps directory.

Add test in testlist

There are several lists of tests in the test/testdeps directory. For example, there are the tier 1 tests which are the primary set of tests which are run as part of the JEDI CI (continuous integration) in github. Add the name of the new test to the appropriate list. There is no need to add the test to any CMakeLists.txt files; this will happen automatically.

Create test reference file

To create the reference file containing the expected results of the test, first complete the three steps above. Then, at the end of the test configuration YAML file, under the test: key, add the test output filename sub-key as sketched below:

test:
  reference filename: testref/<TEST_NAME>.ref
  test output filename: testref/<TEST_NAME>.test.out

And add a blank reference file with the name <TEST_NAME>.ref in test/testref of the source directory. Next, re-configure and re-build SABER (re-run ecbuild and make) to build the new test. When the new test is built, run the test, and the output will be written into the test/testref directory in the build of SABER – as opposed to the source directory – in a file with the .test.out extension.

Hint

If the test has no dependencies, you can run just the single test using the ctest -R <TEST_NAME> option. See the guide on JEDI Testing for more information on ctest.

Since there was nothing in the reference file already, running the new test to produce the output will result in a failure, but check test/testref to see if the test output file was created. If the output looks correct, and the test only failed due to the reference file mis-match, use this output file that was written in test/testref of the build dirctory to overwrite the blank reference file in the source directory.

When you are finished developing the test and are ready to commit it, remove the test output filename: testref/<TEST_NAME>.test.out line from the test YAML file before opening the pull request.