Adding a New Test

So, you’ve developed a new feature in some JEDI repository and now you want to test it. You have come to the right place.

This document is a step-by-step guide on how to add a test to JEDI. Here we focus on unit tests but in the final section we include some guidance on how you might proceed to Add an Application test.

The first thing to emphasize is that there are many levels of sophistication on how you might wish to proceed. The simplest case is to use the existing unit test infrastructure. If you go this route, then you may just have to create an appropriate configuration file and then register your test with CMake and CTest (Steps 7-8). Or, you may wish to add a new test to an existing test suite, to create a new test suite, or even to establish the proper directory structure.

Wherever you are on this spectrum of possibilities, we hope this document will be useful. Just be aware that you are under no obligation to follow all the steps. If the infrastructure for that step is already in place, then feel free to proceed to the next step.

In any case, it is imperative that you first read this document that describes how tests are organized, implemented and executed within JEDI. In particular, please read our vision on how we would like the development of the JEDI testing framework to proceed.

Step 1: Create a File for your Test Application

The goal of the JEDI Testing Framework is to have the test directory mirror the source directory such that each of the main C++ classes defined in the source directory has a corresponding test.

So let’s say that there is a file in some JEDI repository called src/mydir/MyClass.h that defines a C++ class called MyClass. And, we want to define a unit test suite to test the various methods and constructors in MyClass. The first thing we would want to do is to create a file called test/mydir/MyClass.h that will contain the test application.

If you’re working in a well-established JEDI directory then this file may already exist. If that’s the case, then you can probably move on to Step 3. On the other hand, if you’re adding a new model to JEDI, it’s possible that the directory won’t exist yet, let alone the file. So, create the directory and the file as needed.

Before proceeding we should emphasize again that there is another option. Often the existing test applications defined within oops/src/test/interface will be sufficient to test your new feature. If this is the case, then you can skip ahead to Step 6. There you will see an example where this is done for FV3.

Or, even better - sometimes the test infrastructure will already exist within the JEDI repository you are working with. Often you can reuse an existing test and just specify a different configuration that activates your new feature. In that case all you have to do is to create an appropriate configuration file and you can proceed directly to Step 7!

If you’re still in this section, read on for some tips on how to write the application file test/mydir/MyClass.h.

What MyClass.h Should Contain

As a prelude to the steps that follow, we note that the main purpose of test/mydir/MyClass.h is to define test::MyClass as a sub-class of oops::Test. As a sub-class of oops::Test, test::MyClass will also be a sub-class of oops::Application. That is, it will be an Application object that can be executed by passing it to an oops::Run object. In addition to declaring and defining test::MyClass, our file might also define a MyClassFixture class to help with accessing the configuration file (Step 2). Necessary components of test/mydir/MyClass.h include one or more functions that define the unit tests as well as a register_tests() method within test::MyClass that adds these tests to the master test suite.

Since we’ll be building off of oops::Test and the eckit unit test suite, one necessary item in the header of our MyClass.h file is:

#define ECKIT_TESTING_SELF_REGISTER_CASES 0

And the contents of the file should be encapsulated within the test namespace, to distinguish it from the corresponding class of the same name in the src directory.

What MyClass.h Should not Contain

As described elsewhere, the unit tests are initialized and executed by means of the execute() method in oops::Test. As a sub-class of oops::Test, the test::MyClass object will have access to this method and it is best not to re-define it. Doing so may disrupt how eckit executes the tests.

Step 2: Define A Test Fixture

In JEDI, test fixtures are generally used to create objects as directed by the relevant sections of the configuration file, for use with the unit tests. As an example, consider this code segment in oops/src/test/interface/Increment.h:

template <typename> class IncrementFixture : private boost::noncopyable {
typedef oops::Geometry<MODEL>       Geometry_;

public:
 static const Geometry_       & resol()   {return *getInstance().resol_;}
 static const oops::Variables & ctlvars() {return *getInstance().ctlvars_;}
 static const util::DateTime  & time()    {return *getInstance().time_;}

private:
 static IncrementFixture<MODEL>& getInstance() {
   static IncrementFixture<MODEL> theIncrementFixture;
   return theIncrementFixture;
 }

 IncrementFixture<MODEL>() {
//  Setup a geometry
   const eckit::LocalConfiguration resolConfig(TestEnvironment::config(), "Geometry");
   resol_.reset(new Geometry_(resolConfig));

   const eckit::LocalConfiguration varConfig(TestEnvironment::config(), "Variables");
   ctlvars_.reset(new oops::Variables(varConfig));

   time_.reset(new util::DateTime(TestEnvironment::config().getString("TestDate")));
 }

 ~IncrementFixture<MODEL>() {}

 boost::scoped_ptr<Geometry_>       resol_;
 boost::scoped_ptr<oops::Variables> ctlvars_;
 boost::scoped_ptr<util::DateTime>  time_;
 };

Note that this, like other oops test objects, is a class template, with a different instance for each model. This may not be necessary for your test if it is model-specific. The main point here is that the resol(), ctlvars(), and time() methods of test::IncrementFixture<MODEL> access the “Geometry”, “Variables”, and “TestDate” sections of the configuration file and use this information to create objects of type oops::Geometry<MODEL>, oops::Variables, and util:DateTime. These methods are then used repeatedly by the various unit tests that are included in test::Increment. The TestEnvironment::config() calls in the code above provide a way to pass global configuration data to the test fixtures.

So, proceeding with our example, it would be advisable to begin by defining a test::MyClassFixture class in test/mydir/MyClass.h to facilitate the creation of useful objects as specified in the configuration file. For many more examples see the various files in oops/src/test/interface.

Step 3: Define Your Unit Tests

Now the next step would be to define the unit tests themselves as functions within test/mydir/MyClass.h. As a guide you can use the illustrative example in Anatomy of a Unit Test or the many examples to be found in oops/src/test/interface. The possibilities are endless, but just remember two things:

  1. Include one or more calls to eckit check functions

  2. Use your test fixture to create objects based on the information in the configuration file

Step 4: Register your Unit Tests with eckit

In order for eckit to run your tests, you have to register each individual test. This is achieved by means of the register_tests() method of test::MyClass and as this test::Increment example (from (oops/src/test/interface/Increment.h) demonstrates, there is little else needed to define the class:

template <typename MODEL> class Increment : public oops::Test {
 public:
  Increment() {}
  virtual ~Increment() {}
 private:
  std::string testid() const {return "test::Increment<" + MODEL::name() + ">";}

  void register_tests() const {
    std::vector<eckit::testing::Test>& ts = eckit::testing::specification();

    ts.emplace_back(CASE("interface/Increment/testIncrementConstructor")
      { testIncrementConstructor<MODEL>(); });
    ts.emplace_back(CASE("interface/Increment/testIncrementCopyConstructor")
      { testIncrementCopyConstructor<MODEL>(); });
    ts.emplace_back(CASE("interface/Increment/testIncrementTriangle")
      { testIncrementTriangle<MODEL>(); });
    ts.emplace_back(CASE("interface/Increment/testIncrementOpPlusEq")
      { testIncrementOpPlusEq<MODEL>(); });
    ts.emplace_back(CASE("interface/Increment/testIncrementDotProduct")
      { testIncrementDotProduct<MODEL>(); });
    ts.emplace_back(CASE("interface/Increment/testIncrementAxpy")
      { testIncrementAxpy<MODEL>(); });
    ts.emplace_back(CASE("interface/Increment/testIncrementInterpAD")
      { testIncrementInterpAD<MODEL>(); });
    }
  };

So, we would proceed by defining test::MyClass in a similar way. Just specify the test object (here ts) and add each of your test functions one by one using emplace_back as shown.

Then no more action is required for test/mydir/MyClass.h; Our test::MyClass::register_tests() method will be executed automatically when we pass test::MyClass as an application to oops::Run (see Initialization and Execution of Unit Tests).

Step 6: Create an Executable

Executables for each test are generally located in the test/executables directory of each JEDI repository, though sometimes this directory is called test/mains. This is not to be confused with the mains directory (branching off the top level of the repository) which is typically reserved for the production-level programs.

As described in Tests as Applications, there is not much for the executable file to do. It only really has three tasks:

  1. Create an oops::Run object

  2. Create an oops::Application object (in our example, this would be test::MyClass)

  3. Pass the Application object to the execute() method of the Run object

So, to proceed with our example, we might go to the test/executables directory of our repository (create it if it’s not there already) and create a file called TestMyClass.cc with the following contents:

#include "oops/runs/Run.h"
#include "../mydir/MyClass.h"

int main(int argc,  char ** argv) {
  oops::Run run(argc, argv);
  test::MyClass tests;
  run.execute(tests);
  return 0;
};

That’s it. Note that the include paths for a given repository are specified in the CMakeLists.txt file in the top level of the repository. All existing JEDI repositories will already have access to oops/src by means of these lines, or something similar:

ecbuild_use_package( PROJECT oops VERSION 0.2.1 REQUIRED )
include_directories( ${OOPS_INCLUDE_DIRS} )

So, the first include statement in the TestMyClass.cc example above should have no problem finding oops/src/oops/runs/Run.h, where the oops::Run class is defined.

It is likely that the src directory of the working repository is also in the include path. So, in the above example we specified the relative path of our MyClass.h file in the test directory so the compiler does not confuse it with the file of the same name in the src directory.

In some situations it might be beneficial to define a modified Run object that does some additional model-specific set up. Here is an example from fv3-jedi/test/executables/TestModel.cc

#include "FV3JEDITraits.h"
#include "RunFV3JEDI.h"
#include "test/interface/Model.h"

int main(int argc,  char ** argv) {
  fv3jedi::RunFV3JEDI run(argc, argv);
  test::Model<fv3jedi::FV3JEDITraits> tests;
  run.execute(tests);
  return 0;
};

However, fv3jedi::RunFV3JEDI is a sub-class of oops::Run and it uses the execute() method of its parent. So, the execution of the test is essentially the same as the previous example.

Also, it is worth noting that the application used here is the fv3jedi::FV3JEDITraits instance of test::Model<MODEL>, which is already defined in oops/src/test/interface/Model.h. So, in this case there would be no need to create a new test application as described in Steps 1-5.

Step 7: Create a Configuration File

Along with the executable, the configuration file is the way to tell JEDI what you want it to do. We reserve a detailed description of how to work with JEDI configuration files for another document.

Here we’ll just say that the proper place to put it is in the test/testinput directory of the JEDI repository that you are working with. Or, if your tests are located in test/mydir, another option would be to put the associated input files in test/mydir/testinput. If there are already some files there, you can use them as a template for creating your own. Or, you can look for testinput files from other repositories that test similar functionality.

Let’s call our configuration file test/testinput/myclass.yaml. To proceed, we would create the file and then edit it to activate the code features that we wish to test.

As mentioned way back in Step 1, some tests do not require new infrastructure. Some new tests only require a different configuration file to activate a different feature of the code. If this is the case for you, then you can just duplicate an existing configuration file and modify it accordingly, skipping Steps 1-6.

Step 8: Register all files with CMake and CTest

In steps 1-7 above we have created or modified three files, namely the source code for our tests, test/mydir/MyClass.h, the executable test/executables/TestMyClass.cc, and the configuration file test/testinput/myclass.yaml. In order for CMake to compile and run these files, we have to let CMake know they exist.

We achieve this by editing the file test/CMakeLists.txt. This is where the tests are managed from the perspective of CMake and CTest.

We’ll start with the configuration file because every new test you add is likely to have a new configuration file. Edit the CMakeLists.txt file and look for a list of input files like this one from oops/qg/test/CMakeLists.txt:

list( APPEND qg_test_input
  testinput/3dvar.yaml
  testinput/3dfgat.yaml
  testinput/4densvar.yaml
  testinput/4dvar.alpha.yaml
  [...]
  testinput/test_op_obs.yaml
  testinput/analytic_init.yaml
  testinput/analytic_init_fc.yaml
  compare.sh
)

You would add your input file, test/testinput/myclass.yaml to this list (note that the path is relative to the path of the CMakeLists.txt file itself). If you search on qg_test_input in the file, you can see that list is later used to create a soft link for the input files in the build directory, where the tests will be run.

Finally, at long last, you can register your test with CTest. We can do this with a call to ecbuild_add_test() in the test/CMakeLists.txt file. Here is an example from oops/qg/test/CMakeLists.txt:

ecbuild_add_test( TARGET  test_qg_state
               SOURCES executables/TestState.cc
               ARGS    "testinput/interfaces.yaml"
               LIBS    qg )

The TARGET option defines the name of the test. The use of TARGET, as opposed to COMMAND, tells CMake to compile the executable before running it. This requires that we specify the executable with the SOURCES argument, as shown.

The configuration file is specified using the ARGS argument to ecbuild_add_test(). This will be implemented as a command-line argument to the executable as described in Manual Execution. The LIBS argument specifies the relevant source code through a previous call to ecbuild_add_library().

So, our example would look something like this:

ecbuild_add_test( TARGET  test_myrepo_myclass
               SOURCES executables/TestMyClass.cc
               ARGS    "../testinput/myclass.yaml"
               LIBS    myrepo )

Note that this is sufficient to inform CMake of the existence of our executable so it need not appear in any other list of files (such as test_qg_input above or similar lists of source files used to create the ecbuild libraries). Furthermore, since the executable includes our test application file test/mydir/MyClass.h, it will be compiled as well, as part of the compilation of the executable. So, we’re done! Good luck with debugging!

There are many other useful arguments for ecbuild_add_test(). As usual, the best source for information is the file that defines the macro itself, cmake/ecbuild_add_test.cmake in ECMWF’s ecbuild repository. And, as usual, we recommend that you peruse the other JEDI repositories for relevant examples. If you want to add input data files and/or Fortran namelists to your test configurations, have a look at how this is done in fv3-jedi/test/CMakeLists.txt.

Adding an Application Test

The steps above are specific to Unit Tests. You could in principle follow much of the same procedure to create an Application test but since these are usually used to test existing Applications, steps 1-5 would usually not be necessary.

You would have to design your application to produce a text output file as described in Application Testing and you would have to provide a reference output file to compare against. These reference output files would be have to be added to the CMakeLists.txt file in much the same way as the input configuration files (Step 8) in order to ensure that they will be visible from the build directory; see oops/qg/test/CMakeLists.txt for an example.

You would add your test to the appropriate CMakeLists.txt file with ecbuild_add_test() as described in Step 8 but the argument list would be somewhat different as illustrated here:

ecbuild_add_test( TARGET test_qg_truth
               TYPE SCRIPT
               COMMAND "compare.sh"
               ARGS "${CMAKE_BINARY_DIR}/bin/qg_forecast.x testinput/truth.yaml"
                    testoutput/truth.test
               DEPENDS qg_forecast.x )

Here we include a TYPE SCRIPT argument and we specify command.sh as the command to be executed. The ARGS argument now includes the two files to be compared, namely the output of our application ${CMAKE_BINARY_DIR}/bin/qg_forecast.x testinput/truth.yaml (set off by quotes) and our reference file, testoutput/truth.test. We include the executable application in the DEPENDS argument to make sure that CMake knows it needs to compile this application before running the test.

However, before you add an Application test we must warn you again that the compare.sh script may run into problems if you run your application on multiple MPI threads. We are currently working on a more robust framework for Application testing.