Testing

FV3-JEDI comes with an extensive suite of ctests. These tests are designed to makes sure as much as possible of the source code in FV3-JEDI is regularly exercised and that all common applications are tested. This ensures continued functionality of the code as changes are made and saves time by not requiring code reviewers to run all the tests themselves and instead be focused on design implications.

Continuous integration

FV3-JEDI has an extensive continuous integration (CI) suite running on Amazon Web Services. Each time a pull request is issued against develop in the JCSDA public repository the FV3-JEDI part of the FV3-BUNDLE package is built with Intel, GNU and Clang compilers and all the FV3-JEDI tests are executed. Unless all the tests pass a failure blocks the pull request from being merged. In addition to running all the tests the CI includes a code coverage report. The main purpose of the code coverage test is to make sure that any changes to the code are included in one of the tests that run. If the coverage of the code that is added through the pull request is less than the coverage of the existing code it will issue a failure that blocks the merge.

Adding a test to FV3-JEDI

Having the CI means that developers are responsible for adding a test that covers any code they wish to add to the system. All the testing in FV3-JEDI is controlled through fv3-jedi/test/CMakeLists.txt. Broadly there are two kinds of ctests in FV3-JEDI, and JEDI in general. The first is a so-called interface test. These kinds of tests measure individual methods (units) in each class. It might have an expected result or allow for a result to within some tolerance. The other kind of ctest is an application test, where an entire application is run. In these application tests there is a reference log file that is compared with the actual output, usually with some tolerance to allow for small differences.

To simplify adding tests to FV3-JEDI a macro function fv3jedi_add_test is provided. Using this function an interface test is added as follows:

# Add an interface test
fv3jedi_add_test( NAME    test_name
                  EXE     test_fv3jedi_testexe.x
                  INTERFACETEST )

In the above test_name is the users choice for the test name. There needs to be a corresponding file Data/testinput/test_name.yaml. The executable test_fv3jedi_testexe.x is the test exeutable that will be run. These are all pre-built executables for each of the relevant interface tests.

To add an application test the INTERFACETEST directive is dropped:

# Add an application test
fv3jedi_add_test( NAME    test_name
                  EXE     fv3jedi_application.x
                  TOL     1e-3 2)

When adding an application test the user must also provide a file Data/testinput/test_name.ref. This file is generated by running the application with the test configuration and writing the output to a log file. The below provides and example of how to generate the log and reference file:

# cd to test directory
cd build/fv3-jedi/test

# Run the application
mpirun -np 6 ../../bin/fv3jedi_forecast.x testinput/forecast.yaml forecast.log

# Generate the test reference file with grep 'Test     :' (five spaces followed by :)
grep 'Test     :' forecast.log > forecast.ref

The test will run the application and then run a Python script located at bin/compare.py that compares the output of the test with the reference. For the user putting together the test the output and the reference are identical, but for other users on differently configured systems the output may be slightly different. The TOL parameter provides limits on differenes. The two arguments provide tolerance on float output and integer output respectively. Relative differences larger than these user provided values will result in failure. Note that when adding new tests it is worth testing on at least two different compilers, say Intel and GNU, before making a pull request. If the tolerance values are too small failures may result in the Continuous integration. If a large change is being made to the source code that results in the reference values of many tests needing to be changed the script testoutput/update_output.py may be useful.

FV3JEDI uses a test tiering. Tier 1 tests take at most a few seconds and are run on each commit to a pull request. Higher tier tests can take longer and require more memory. These kinds of tests are designed to be run nightly or weekly and test more expensive aspects of the code. Tests are tier 1 by default. If the test is expected to take a long time to run it can be added to the second tier of tests. This is achieved with an additional parameter as follows:

# Add an application test
fv3jedi_add_test( NAME    test_name
                  EXE     fv3jedi_application.x
                  TIER    2 )

See Controlling the testing for how to run tests of different tiers.

If a test depends on another test first having run a dependency parameter can be added:

# Add an application test
fv3jedi_add_test( NAME            test_name
                  EXE             fv3jedi_application.x
                  TEST_DEPENDS    other_test_name )

Timing test

FV3-JEDI includes a timing test that runs as part of the Continuous integration. In this test the time each test takes to run is compared with some predetermined values. If a change is made that dramatically increases the time any of the tests take to run it will result in a failure. Reference timings for each test are located at e.g. test/testoutput/CTestCostData.txt.awsintel.test and the tests for which the run times are checked are at test/testinput/test_time.yaml.