Building and compiling JEDI

As described in detail CMake, CTest, and ecbuild, the procedure for building and compiling JEDI rests heavily on the software tools CMake and ecbuild, which make your life much easier. A typical workflow proceeds in the following steps, which are described in more detail in the sections that follow:

  1. Configure System: set git settings and load spack-stack environment (if not previously set)

  2. Clone the desired JEDI bundle This includes the creation and activation of a python3 virtual environment (venv) based on the spack-stack python installation.

  3. Optionally edit the CMakeLists.txt file in the bundle to choose the code branches you want to work with

  4. cd to the build directory and run ecbuild to generate the Makefiles and other infrastructure

  5. Run make update to pull the latest, up-to-date code from GitHub (optional) and make to compile the code

  6. Run ctest to verify that the bundle is working correctly

In terms of the actual commands you would enter, these steps will look something like this:

cd <src-directory>
git clone https://github.com/JCSDA/jedi-bundle.git
cd <build-directory>
# See build step 3 for possible ways of detecting the correct
# Python3 interpreter in the ecbuild command
ecbuild <src-directory>/jedi-bundle
make update
make -j4
ctest

In this document, we describe Steps 0 through 4, including the various options you have available to you at each step. For a description of Step 5, see our page on JEDI Testing.

You will probably only need to do Steps 0 and 1 once. However, if you are a developer who is making changes to one or more JEDI repositories, you will likely find it useful to execute Steps 2 through 5 multiple times, with progressively increasing frequency. For example, if you are working with a single repository, you may only need to do Step 2 once in order to tell ecbuild to compile your local branch. And, you’ll only need to run ecbuild (Step 3) occasionally, when you make changes that affect the directory tree or compilation (for example, adding a file that was not there previously or substantially altering the dependencies). By comparison, you will likely execute Steps 4 and 5 frequently as you proceed to make changes and test them.

Step 0: System Configuration

Before jumping into building JEDI, there are a few system settings to configure. First, load the spack environment on your system (see Using spack-stack modules to build and run JEDI). Supported HPC platforms have a year’s worth of the latest spack-stack releases pre-built and installed. If you will be using a publicly released version of JEDI, make sure to use the corresponding spack-stack version (see the table below):

Release Date

JEDI-bundle Version

Spack-Stack Version

develop

develop

version 1.8.0

April 2024

skylab-v8.0.0

version 1.7.0

Jan 2024

skylab-v7.0.0

version 1.6.0

Sept 2023

skylab-v6.0.0

version 1.5.1

For building the development version of the JEDI-bundle, use the latest spack stack release. If you are working on a local machine or a non-supported platform, you will have to build the spack-stack.

Next, there are a few one-time settings to configure that will (1) let you avoid the need to enter your GitHub password many times during the JEDI build process, which will make the automated cloning of git repositories much faster, (2) set up git lfs (large file storage) for handling large data files, and (3) – IF the CPU on your machine has fewer than 6 threads – configure Open MPI settings to allow up to 6 MPI tasks to run.

All JEDI repositories are stored and distributed by means of GitHub. If you have used git before, then you probably already have a .gitconfig configuration file in your home directory. If you have not already done so at some point in the past, you can create a git configuration file by specifying your GitHub username and email as follows:

git config --global user.name <username-for-github>
git config --global user.email <email-used-for-github>

This is a recommended action for any user of GitHub since it governs how you access GitHub with git. However, there is another action that you may not have set up previously but that will be immensely useful to all JEDI users and developers: tell GitHub to cache your GitHub credentials:

git config --global credential.helper 'cache --timeout=3600'

This tells GitHub to keep your GitHub login information for an hour, i.e. 3600 seconds (feel free to increase this time if you wish). If you don’t do this, you may regret it - you’ll have to enter your GitHub password repeatedly throughout the build process as ecbuild proceeds to clone multiple GitHub repositories.

The statement above should be sufficient for most systems. However, on some systems (particularly HPC systems with stringent security protocols), it may be necessary to explicitly give git permission to store your GitHub password unencrypted on disk as follows:

git config --global --add credential.helper 'store'

As for all your files, your password will still be protected by the security protocols necessary to simply access the system as a whole and your own filesystem in particular. So, this should still be pretty secure on HPC systems but you might want to use it with caution in less secure environments such as laptops or desktops. For other alternatives, see the documentation on git credentials.

Before building the jedi code, you should also make sure that git is configured to interpret files that are stored on Git-LFS:

git lfs install --skip-repo

This only needs to be done once, and it is required even if you are running in a container.

Another thing to keep in mind is that many JEDI tests likely require more MPI tasks to run than the number of processor cores on your system. For example, many laptops have two or four processor cores but the minimum number of MPI tasks needed to run many fv3-jedi ctests within the jedi-bundle is 6. That’s no problem - you just have to tell Open MPI that it is ok to run more than one MPI task on each core. To do this, run these commands:

mkdir -p ~/.openmpi
echo "rmaps_base_oversubscribe = 1" > ~/.openmpi/mca-params.conf

Step 1: Clone the Desired JEDI Bundle

JEDI applications are organized into high-level bundles that conveniently gather together all the git repositories necessary for JEDI applications to run. Sometimes a bundle is associated with a particular model, such as FV3 or MPAS, but for most cases using the jedi-bundle is highly recommended.

To start your JEDI adventure, first choose a place – and create a directory – as a home for your bundle (or bundles–plural–if you’re ambitious!). This directory will be referred to as JEDI_ROOT throughout the JEDI documentation. You may call this directory what ever you wish, but jedi is a good choice! Once you create this directory, export it as an environment variable for convenience:

mkdir <path-to-root>/jedi
export $JEDI_ROOT=<path-to-root>/jedi

It is essential to create (one time configuration) and activate a python virtual environment (venv) based on the spack-stack python3 installation before starting the steps to build JEDI. After the spack-stack environment is loaded, there will be a python3 installation located in the path contained in the environment variable python_ROOT. To create the python venv, do the following:

# Create the python venv - first time only
cd $JEDI_ROOT
$python_ROOT/bin/python3 -m venv --system-site-packages venv

# Always activate
source venv/bin/activate

Next, navigate into your JEDI_ROOT and clone the GitHub repository that contains the bundle you want. Those wishing to use JEDI should clone the latest release branch from the publicly available bundle at https://github.com/JCSDA:

cd $JEDI_ROOT
git clone -b 8.0.0 https://github.com/JCSDA/jedi-bundle.git

Alternatively, developers with access to the internal repositories should instead clone the development branch. For the internal repositories, clone from https://github.com/jcsda-internal:

cd $JEDI_ROOT
git clone https://github.com/jcsda-internal/jedi-bundle.git

Step 2: Choose your Repos

As executed above in Step 1, cloning a bundle will create a directory <JEDI_ROOT>/<your-bundle>. This checkout of the bundle will be referred to as the JEDI_SRC (source). Export this as an evironment variable like you did for the JEDI_ROOT. For the jedi-bundle:

export JEDI_SRC=$JEDI_ROOT/jedi-bundle

Navigate (cd) into this source directory and have a look (modify this as needed if you used a different path or a different bundle). There’s not much there. There is a README file that you might want to consult for specific information on how to work with this bundle. But in this Step we’ll focus on the CMakeLists.txt file. This contains a list of repositories that the application needs to run. In the case of jedi-bundle one of the more important sections of the CMakeLists.txt looks like this:

ecbuild_bundle( PROJECT gsw      GIT "https://github.com/jcsda-internal/GSW-Fortran.git" BRANCH develop UPDATE )

ecbuild_bundle( PROJECT oops     GIT "https://github.com/jcsda-internal/oops.git"        BRANCH develop UPDATE )
ecbuild_bundle( PROJECT vader    GIT "https://github.com/jcsda-internal/vader.git"       BRANCH develop UPDATE )
ecbuild_bundle( PROJECT saber    GIT "https://github.com/jcsda-internal/saber.git"       BRANCH develop UPDATE )

ecbuild_bundle( PROJECT crtm     GIT "https://github.com/jcsda/CRTMv3.git"               BRANCH develop UPDATE )

option(ENABLE_IODA_DATA "Obtain ioda test data from ioda-data repository (vs tarball)" ON)
ecbuild_bundle( PROJECT ioda-data GIT "https://github.com/jcsda-internal/ioda-data.git"  BRANCH develop UPDATE )
ecbuild_bundle( PROJECT ioda     GIT "https://github.com/jcsda-internal/ioda.git"        BRANCH develop UPDATE )

option(ENABLE_UFO_DATA "Obtain ufo test data from ufo-data repository (vs tarball)" ON)
ecbuild_bundle( PROJECT ufo-data GIT "https://github.com/jcsda-internal/ufo-data.git"    BRANCH develop UPDATE )
ecbuild_bundle( PROJECT ufo      GIT "https://github.com/jcsda-internal/ufo.git"         BRANCH develop UPDATE )

The lines above tell ecbuild which specific branches to retrieve from each GitHub repository. Modify these accordingly if you wish to use different branches. When you then run ecbuild as described in Step 3 below, it will first check to see if these repositories already exist on your system, within the directory of the bundle you are building. If not, it will clone them from GitHub. Then ecbuild will proceed to checkout the branch specified by the BRANCH argument, fetching it from GitHub if necessary.

If the specified branch of the repository already exists on your system, then ecbuild will not fetch it from GitHub. If you want to make sure that you are using the latest and greatest version of the branch, then there are two things you need to do.

First, you need to include the (optional) UPDATE argument in the ecbuild_bundle() call as shown in each of the lines above. Second, you need to explicitly initiate the update by running make update as described in Step 4.

This will tell ecbuild to do a fresh pull of each of the branches that include the UPDATE argument. Note that make update will not work if there is no Makefile in the build directory. So, this command will only work after you have already run ecbuild at least once.

If you are a developer, you will, by definition, be modifying the code. And, if you are a legitimate JEDI Master, you will be following the Git flow (Application) workflow. So, you will have created a feature (or bugfix) branch on your local computer where you are implementing your changes.

For illustration, let’s say we created a feature branch of ufo called feature/newstuff, which exists on your local system. Now we want to tell ecbuild to use this branch to compile the bundle instead of some other remote branch on GitHub. To achieve this, we would change the appropriate line in the CMakeLists.txt file to point to the correct branch and we would remove the UPDATE argument:

ecbuild_bundle( PROJECT ufo GIT "<JEDI_ROOT>/jedi-bundle/ufo" BRANCH feature/newstuff )

This may be all you need to know about ecbuild_bundle() but other options are available. For example, if you would like to fetch a particular release of a remote GitHub repository you can do this:

ecbuild_bundle( PROJECT eckit GIT "https://github.com/ECMWF/eckit.git" TAG 0.18.5 )

For further information see the cmake/ecbuild_bundle.cmake file in ECMWF’s ecbuild repository.

Step 3: Run ecbuild (from the build directory)

After you have chosen which repositories to build, the next step is to create a build directory and export it as JEDI_BUILD for convenience:

cd $JEDI_ROOT
mkdir build
export JEDI_BUILD=$JEDI_ROOT/build

Then, from that build directory, run ecbuild, specifying the path to the directory that contains the source code for the bundle you wish to build:

cd $JEDI_ROOT/build
ecbuild $JEDI_SRC

Here we have used $JEDI_SRC as our source directory and $JEDI_ROOT/build as our build directory. Feel free to change this as you wish, but just make sure that your source and build directories are different. This command should work for most bundles, and in particular when working on a preconfigured HPC or AWS instance. The ecbuild command may take several minutes to run.

In case cmake is picking up the wrong python3 interpreter, an optional argument to the ecbuild command can be used to specify the correct python3 interpreter during the build process. When using the modules provided by spack-stack, the argument -DPython3_EXECUTABLE=${python_ROOT}/bin/python3 will guarantee that the spack-stack python3 interpreter is getting used. A similar method can be used to point to another python3 installation.

Warning

Some bundles may require you to run a build script prior to or in lieu of running ecbuild, particularly if you are running on an HPC system. Check the README file in the top directory of the bundle repository to see if this is necessary, particularly if you encounter problems running ecbuild, cmake, or ctest.

As described in CMake, CTest, and ecbuild, ecbuild is a sophisticated interface to CMake. So, if there are any CMake options or arguments you wish to invoke, you can pass them to ecbuild and it will kindly pass them on to CMake. The general calling syntax is:

ecbuild [ecbuild-options] [--] [cmake-options] <src-directory>

Where src-directory is the path to the source code of the bundle you wish to build (in this case, your JEDI_SRC). The most useful ecbuild option is debug:

ecbuild --build=debug $JEDI_SRC

This will invoke the debug flags on the C++ and Fortran compilers and it will also generate other output that may help you track down errors when you run applications and/or tests. You can also specify which compilers you want and you can even add compiler options. For example:

ecbuild -- -DCMAKE_CXX_COMPILER=/usr/bin/g++ -DCMAKE_CXX_FLAGS="-Wfloat-equal -Wcast-align" $JEDI_SRC

If you are working on an HPC system, then we recommend that your first check to see if there are Using spack-stack modules to build and run JEDI installed on your system. If your system is listed on this modules documentation page then you can simply load the modules as described there and you will have access to ecbuild, eckit, and many other third-party libraries. Also, be sure to check out the Skylab HPC users guide page for more information on HPCs.

If your system is not one that is supported by the spack-stack maintainers, then refer to the spack-stack instructions on how to generate a site config and install the environment yourself.

Step 4: Run make (from the build directory)

After running ecbuild, the next step is to make sure the code is up to date. You can do this by running make update from the build directory as described in Step 2:

make update

Warning

Running make update will initiate a git pull operation for each of the repositories that include the GIT and UPDATE arguments in the call to ecbuild_bundle() in CMakeLists.txt. So, if you have modified these repositories on your local system, there may be merge conflicts that you have to resolve before proceeding.

Now, at long last, you are ready to compile the code. From the build directory, just type

make -j4

The -j4 flag tells make to use four parallel processes. Since many desktops, laptops, and of course HPC systems come with 4 or more compute cores, this can greatly speed up the compile time. Feel free to increase this number if appropriate for your hardware.

The most useful option you’re likely to want for make other than -j is the verbose option, which will tell you the actual commands that are being executed in glorious detail:

make VERBOSE=1 -j4

As usual, to see a list of other options, enter make --help.

Again, the compile can take some time (10 minutes or more) so be patient. Then, when it finishes, the next step is to run the test following the instructions in JEDI Testing.

If the parallel compile fails, the true error may not be in the last line of the output because all processes are writing output simultaneously and some may still continue while another fails. So, in that case, it can be useful to re-run make with only a single process. Omitting the -j option is the same as including -j1:

make VERBOSE=1