Building and Testing MPAS-JEDI

This section describes how to build MPAS-JEDI using CMake, then confirm that your build is working properly with CTest. Usage of CMake and CTest are described in the JEDI CMake, CTest, and ecbuild documentation.

MPAS-BUNDLE

In order to build MPAS-JEDI and its dependencies, it is recommended to use MPAS-BUNDLE, available at https://github.com/JCSDA/mpas-bundle. Within MPAS-BUNDLE, the file named CMakeLists.txt controls the dependency chain of components that are either essential (e.g., OOPS, IODA, UFO, SABER, and CRTM) or optional (e.g., RTTOV) to the procedure that eventually generates MPAS-JEDI executables. MPAS-BUNDLE is built using ecbuild. Full details on how to build any JEDI bundle are provided elsewhere, and it is recommended to familiarize yourself with those instructions before continuing here.

Building and testing MPAS-BUNDLE on Cheyenne

Most development and testing of MPAS-JEDI has been performed on NCAR’s Cheyenne HPC system. Custom scripts for creating the required build environment on Cheyenne are provided in MPAS-BUNDLE. After cloning MPAS-BUNDLE from Github, you can find these scripts in mpas-bundle/env-setup. Before executing the ecbuild command, source the script appropriate for your choice of compiler, MPI implementation, and shell (e.g., gnu-openmpi-cheyenne.sh). The commands in the environment script are consistent with the instructions for Cheyenne under Using spack-stack modules to build and run JEDI.

After building MPAS-BUNDLE, it is recommended to run the ctests. Passing this suite of tests confirms that your build is working as expected.

Starting from a project directory such as $HOME/jedi, the entire build and test workflow on Cheyenne would look like:

git clone https://github.com/JCSDA/mpas-bundle.git     # this creates the 'mpas-bundle' directory
source mpas-bundle/env-setup/<desired environment script>
mkdir ./<build-directory>
cd ./<build-directory>
ecbuild  ../mpas-bundle
make update
make -j4
cd mpas-jedi
ctest
Notes about building on Cheyenne:

  • The gnu-openmpi environment has been more extensively tested than the intel-impi environment on Cheyenne

  • The <build-directory> cannot be the directory named mpas-bundle, where the repository is cloned, because doing so will create conflict between the source code directory and the CMake-generated build sub-directories

  • Users can expect the above build and test procedure to take approximately 45 minutes. For some speedup, it is recommended to execute the make step in a job script with -j16 instead of -j4, which will use 16 processors instead of 4 in the parallel build. Much of the total time spent is during the ecbuild step, which downloads the code for the first time.

Building MPAS-BUNDLE in Singularity

MPAS-BUNDLE can also be built and tested in the JEDI development Singularity container. Detailed instructions are provided at that link. If you do not plan to or are unable to install Singularity natively, you may be interested to learn how to launch a Singularity container in a Vagrant Virtual Machine. When working in the Singularity container, the main difference from the instructions provided above for Cheyenne is that the environment is already set up properly within the container. Thus there is no need to source an environment setup file.

Built executables

After completing the MPAS-BUNDLE build, users have access to many executables under <build-directory>/bin, many of which are generated when building the projects on which MPAS-JEDI is dependent (OOPS, UFO, SABER). The executables that are relevant to MPAS are as follows, grouped separately for MPAS-A and MPAS-JEDI.

MPAS-A

  • mpas_atmosphere: can be used interchangeably with the atmosphere_model executable that would normally be built using the non-JEDI (standalone) MPAS-Model build mechanism for the atmosphere core. Its purpose is to integrate the model forward in time from an initial time to a final time with periodic IO of model fields of importance.

  • mpas_init_atmosphere: can be used interchangeably with the init_atmosphere_model executable that would normally be built using the non-JEDI (standalone) MPAS-Model build mechanism for the init_atmosphere core. Its purpose is to generate cold-start initial condition and surface input files.

MPAS-JEDI

Each of these executables are model-specific implementations of generic applications that are derived from the oops::Application class, i.e., oops/src/oops/runs/Application.h. Descriptions of the generic applications are located under the OOPS Applications documentation. Here we give short synopses of a few specific MPAS-JEDI implementations.

  • Applications with one initial state

    • mpasjedi_convertstate.x (oops::ConvertState)

    • mpasjedi_dirac.x (oops::Dirac)

    • mpasjedi_forecast.x (oops::Forecast): essentially does the same as the mpas_atmosphere executable, but through the JEDI generic framework via the MPAS-JEDI interface. There is more overhead than when running the non-JEDI exectuable, and this requires a YAML file in addition to the standard namelist.atmosphere used to configure mpas_atmosphere.

    • mpasjedi_gen_ens_pert_B.x (oops::GenEnsPertB)

    • mpasjedi_hofx.x (oops::HofX4D)

    • mpasjedi_hofx3d.x (oops::HofX3D)

    • mpasjedi_parameters.x (saber::EstimateParams): used to estimate static background error covariance and localization matrices

    • mpasjedi_staticbinit.x (oops::StaticBInit)

    • mpasjedi_variational.x (oops::Variational): carries out many different flavors of variational data assimilation (3DVar, 3DEnVar, 3DFGAT, 4DEnVar) with a variety of incremental minimization algorithms

  • Applications with multiple initial states

    • mpasjedi_eda.x (oops::EnsembleApplication<oops::Variational>)

    • mpasjedi_enshofx.x (oops::EnsembleApplication<oops::HofX4D>)

    • mpasjedi_rtpp.x (oops::RTPP): standalone application that carries out Relaxation to Prior Perturbation, as introduced by Zhang et al. (2004). The intended purpose is to inflate the analysis ensemble spread after running the EDA application.

Most of the MPAS-JEDI executables are exercised in ctests. As users learn how to use MPAS-JEDI for larger-scale applications, it is useful to consider the ctests as examples and templates. For more information on the individual ctests, see the documentation for their yaml configuration files.

Controlling the testing

In addition to the basic ctest command shown in Building and testing MPAS-BUNDLE on Cheyenne, which runs all of the available tests for MPAS-JEDI, ctest has basic flags and arguments available for selecting a subset of tests. ctest also automatically provides some logging functionality that is useful for reviewing passing and failing test cases. Both of those aspects of ctest are described in more detail within the JEDI Developer Tools and JEDI Testing documentations.

References

Zhang, F., C. Snyder, and J. Sun (2004): Impacts of initial estimate and observation availability on convective-scale data assimilation with an ensemble Kalman filter. Mon. Wea. Rev., 132, 1238–1253