Installing the jedicluster app

After you have gained access to the JCSDA AWS resources, the next step is to install and configure the AWS Command Line interface (CLI). This will allow you to launch either a single compute node or a multi-node cluster from your computer. After you have created a compute instance or a cluster, you can then log into it and proceed to build and run JEDI.

The easiest way to install the AWS CLI is through a package installer. For example, you can use Homebrew on a Mac:

brew install awscli

or the apt installer on a Debian-based linux OS such as Ubuntu:

sudo apt-get install awscli

Or, since the AWS CLI is a python package, you can also install it with pip or conda, for example:

pip3 install -U awscli --user

For further details see the AWS documentation.

The next step is to configure the AWS CLI to use your AWS login credentials. When you were granted access to JCSDA AWS resources, a JEDI master should have given you an AWS secret access key and associated ID in addition to your username and password. Have this secret access key and ID handy before running this command to configure your AWS CLI:

aws configure

Enter your secret access key ID and the access key itself at the prompts. When prompted for your default region, enter us-east-1. This is where most of the JEDI AMIs are currently housed. At other prompts, including the default output format, you can just type enter to select the default (None).

Now you are ready to install the jedicluster application. This is a python tool developed by JCSDA in order to facilitate the creation of an AWS node or cluster that is ready to run JEDI. In particular, the compute nodes that you create with jedicluster will already be provisioned with all the software applications and libraries you need to build and run JEDI, including the gnu compiler suite, openmpi, netcdf, git-lfs, and much more.

To install jedicluster on your computer, navigate to an appropriate directory (of your choice) and enter these commands:

git clone https://github.com/JCSDA/jedi-tools
cd jedi-tools/AWS/python
make install

This builds the jedicluster executable and installs it somewhere in your home directory. On most Mac and linux systems you can find it in ~/.local/bin but if it’s not there you might try ~/Library/Python/<version>/bin. In any case, just make sure that directory is included in your PATH environment variable. If not, you can add it as follows (bash syntax; you may wish to add this to your ~/.bashrc file so it is available automatically whenever you start a new shell):

Note

The make install command uses the python tool easy_install-<version> where <version> is the python version. If you get a message that that is not found, then you can install it with pip: pip3 install –user setuptools. If that still doesn’t work then you can try editing the Makefile to remove the python version, since some python versions vary on how they name it. That is, replace easy_install-${PYTHON_VERSION} with easy_install.

export PATH=$HOME/.local/bin:$PATH

Once the jedicluster app is installed on your system, you may use it as much as you wish; in principle you only need to follow this procedure once. However, occasionally the jedi stack is updated with new packages or new versions of old packages. These compiled packages are stored in what is called an Amazon Machine Image (AMI), from which the EC2 instances and clusters are created. So, if there is a change in the AMIs, then you’ll have to pull the latest version of jedi-tools and re-install:

cd <path>/jedi-tools/AWS/python
git pull
make install

Changes in the jedi-stack will generally be announced on the GitHub JEDI team discussion board. However, even if you follow that page, it is good practice to occasionally update your jedicluster app to ensure that it is compatible with the latest JEDI code, particularly if you run into build problems.