Using spack-stack modules to build and run JEDI¶

The instructions in this section are specific to the use of spack-stack environment modules (lmod/lua or tcl/tk) for building and running JEDI applications. For general information on using spack-stack to build and run software, see the spack-stack documentation.

One of the big advantages of spack-stack is that it automatically generates modules for all compiled packages and Python packages and works in exactly the same way on HPCs, on the cloud, and on a personal computer. Environment modules are available on basically all HPC systems and any modern macOS or Linux distribution, and are an easy and effective way to manage software libraries. There are two main flavors, the older tcl/tk modules and the newer lmod/lua modules, with the latter being superior and therefore preferred, if available. The two implementations share similar commands, such as:

module list # list modules you currently have loaded
module spider <string> # list all modules that contain <string>
module avail # list modules that are compatible with the modules you already have loaded
module load <package1> <package2> <...> # load specified packages
module unload <package1> <package2> <...> # unload specified packages
module purge # unload all modules

lmod/lua modules provide other convenient commands such as

module swap <packageA> <packageB> # swap one module for another

and handle module dependencies, conflicts, loading and unloading better than tcl/tk modules.

For further information (and more commands) you can refer to the Lmod and Environment Modules documentation.

We currently offer spack-stack modules for JEDI on several HPC systems, as described below. Consult the appropriate section for instructions on how to access the JEDI modules on each system.

These modules provide all of the software libraries necessary to build and run JEDI. It is highly recommended that users start with a clean environment, i.e. that they do not load modules or modify search paths etc. in the automatically source .bashrc, .bash_profile etc. scripts. After loading the appropriate modules, users can proceed to compile and run the JEDI bundle of their choice.

General Instructions¶

This section outlines the general steps to set up spack-stack modules for the desired HPC. Please see Skylab HPC users guide for more information on how to run jobs on the different HPCs.

Load HPC specific modules from spack-stack pre-configured sites documentation.

Load appropriate modules for JEDI and the Skylab v8 release.

module load jedi-fv3-env
module load ewok-env
module load soca-env
module load sp

Your environment is now set up, users can proceed to compile and run the JEDI bundle of their choice.

JEDI and Skylab environment set up using jedi-tools¶

Setup scripts are available in the jedi-tools github repository. These scripts will allow you to correctly set up your JEDI environment in order to build JEDI and also run Skylab experiments. The systems (and compilers) that are currently supported are localhost (gnu, intel, clang), aws-pcluster (gnu, intel), derecho (intel, gnu), discover (intel, gnu), hercules (intel, gnu), orion (intel, gnu), and s4 (intel).

On certain HPCs, like Hercules and Discover, you will need to load the git-lfs module first. You can verify by running module show git-lfs to see if the module is available. This needs to be done before you check out any JEDI repository. Note, some HPCs have git lfs installed via the OS and do not provide a git-lfs module. If that is the case, this step can be skipped.
```
module show git-lfs
module load git-lfs
```
Create your JEDI_ROOT directory and clone the jedi-tools github repository. In JEDI_ROOT you will clone the JEDI code and all the files needed to build, test, and run JEDI and SkyLab.
```
mkdir $JEDI_ROOT
cd $JEDI_ROOT
git clone https://github.com/JCSDA-internal/jedi-tools
```
Edit the header of jedi-tools/buildscripts/setup.sh to fill in your JEDI_ROOT location, HOST, and COMPILER. Then source the updated setup.sh script. Note, there is more information needed to be filled out if you are running on localhost.
```
cd jedi-tools
vi buildscripts/setup.sh
source buildscripts/setup.sh
```
At this point you should have all of the modules needed and loaded in your HPC environment to continue with building JEDI. Verify the modules by running module list and proceed to Building and compiling JEDI.

For HPC specific build and testing help, see Skylab HPC users guide