Vagrant (for Mac and Windows)¶
In order to set up the JEDI environment with Singularity, you’ll need to first set up a Linux operating system. We will often focus on Ubuntu for illustration purposes but if you prefer other varieties of Linux, these are also available from the Virtualbox provider that we describe below.
So, if you’re using a Mac or Windows computer, you will want to set up a local, self-contained Linux environment within your broader operating system. This is commonly called a virtual machine (VM). Once you have a linux VM up and running, you will be able to install Singularity and triumphantly enter the corresponding JEDI Container.
This can all be achieved using an application called Vagrant, which is developed and distributed by a company called Hashicorp. Though we will sometimes refer to Vagrant as the virtual machine provider, the actual Linux operating system is ultimately provided by Oracle’s VirtualBox software package. Vagrant is essentially a tool will allow you to build, configure, and manage a VirtualBox operating system. In particular, we will use Vagrant and VirtualBox to create an Ubuntu virtual machine on your Mac workstation or laptop. We’ll then populate that Linux VM with specific JEDI software tools (compilers, ecbuild, etc.) using Singularity.
From this brief introduction, it is clear that you do not need to worry about Vagrant or VirtualBox if your working machine (whether it is a workstation, laptop, or HPC system) is already running Linux and/or is already running Singularity. By working machine we mean whatever machine you plan to compile and run JEDI on.
You do need Vagrant and VirtualBox (or something equivalent) if you wish to run JEDI from a Mac or Windows machine. Though you can use Vagrant for both platforms, we focus here on Mac OS X.
We refer Windows users to the Vagrant download page where you can download a binary implementation for windows and install it using the Windows package manager. After installing Vagrant, you may wish to return to this document for tips on how to configure it for JEDI (skipping Step A of the next section).
Installing and Configuring Vagrant¶
A: Install Vagrant and VirtualBox¶
As with Windows, you can install Vagrant on a Mac by downloading a pre-compiled binary package from the Vagrant Download Page. However, we recommend that you install with Homebrew as described below to give you more flexibility in managing both vagrant and virtualbox.
Before you begin you should install or update Homebrew.
You’ll need a relatively recent version in order to use the
Once you have done this, you can proceed as follows:
brew cask install virtualbox brew cask install vagrant brew cask install vagrant-manager
B: Download JEDI Configuration File¶
Now we need to tell Vagrant what type of virtual machine we want to create and how to
provision it with the software we need. This is done by means of a configuration
file that goes by the default name of
So, to proceed, you should first create a directory where you will place your
Vagrantfile. This is where you will launch your virtual machine. You should also
create a subdirectory called
vagrant_data that we will use.
If you don’t create this directory, you will get an error when vagrant tried to mount it.
You can call the parent directory whatever you wish but if you change the name of
vagrant_data directory then you will also have to change the Vagrantfile.
mkdir $HOME/jedi-vm cd $HOME/jedi-vm mkdir vagrant_data
In what follows, we will refer to this as the home directory of your Vagrant Virtual Machine (VM).
We at JCSDA provide a Vagrantfile that can be used to create a virtual machine that is pre-configured to build and run JEDI, with both Singularity pre-installed.
Or, alternatively, you can retrieve it with
Place this Vagrantfile in the home directory of your Vagrant VM.
If you already have a Vagrant VM installed and you want to install a new one (particularly using a Vagrantfile in the same directory as before), then you may have to fully delete the previous VM first to avoid any conflicts. Instructions on how to do this are provided in the Deleting a Vagrant VM section below.
If you have problems with this JEDI Vagrantfile, there an alternative Vagrantfile
that you can download
that expands the disk storage using the
disksize plugin to Vagrant.
This also comes with Singularity pre-installed. After downloading this file,
it’s easiest to change its name to
Vagrantfile and then run
vagrant up again.
However, before trying this make sure that you either destroy your previous VM
or create the new VM from a different directory and give it a different
name (edit the Vagrantfile and search for jedibox).
C: Launch your Virtual Machine¶
Now you are ready to create your virtual machine by running this command:
The first time you run this command, it will take several minutes. Vagrant is installing Singularity and a few other supporting software packages. Once created, these will be part of your virtual machine and they do not need to be re-installed (unless you explicitly tell vagrant to do so).
So, when this command finishes, you can log into your virtual machine with
Now you are in a linux environment (CentoOS 7). From here you can pull the JEDI container of your choice,
Depending on which Vagrantfile you use, your VM may run either the Ubuntu or the CentOS operating system. However, you shouldn’t need to be too concerned about this because you’ll be working mostly in the Singularity container which runs Ubuntu. So, if you work within the container, you will be in an Ubuntu environment regardless of which OS your vagrant VM is running.
D: Exit Container and Vagrant¶
Normally you will be spending your time working in the Singularity container.
When you’re finished working for the day, it’s important to remember to
exit twice, once to exit the container and once to log out of the
Vagrant virtual machine:
exit # to exit Singularity exit # to exit Vagrant
Now, to temporarily shut down your virtual machine, enter
Note that this is very different than the
vagrant destroy command,
which should be used with caution. As the name of the command suggests, vagrant
destroy will completely destroy the virtual machine along with all the files and data it contains.
So, if you do this, you will have to re-create the virtual machine and re-install
any JEDI bundles that you are working with. And, you will lose any files that
you have been editing. By contrast, vagrant halt will merely shut down the virtual machine,
retaining all your files. This will allow you to gracefully log out of your workstation
or laptop without harming your JEDI environment. For further details see the Vagrant
E: Enable X Forwarding (Optional)¶
If you’d like to use graphical tools such as
emacs from within the
Singularity container, you will need to set up X forwarding. If you’re doing this
on a Mac, you will first need to install XQuartz,
if it’s not already installed.
After XQuartz is up and running, you can create and enter your VM as described
in step C above. Next you will have to set your
DISPLAY environment variable
to use your local machine. This is best done from within the container because
environment variables set outside the container may not be accessible from within.
# inside the container export DISPLAY=10.0.2.2:0.0
You may wish to add the appropriate display definition to an initialization script that you can run every time you enter the singularity container as described here. Then, enter this on your host machine (i.e. your Mac or Windows machine), to grant the VM permission to display
#On your Mac xhost + 127.0.0.1
These are the addresses that Vagrant uses for by default. You may wish to add the appropriate display definition to an initialization script that you can run every time you enter the singularity container as described here.
To test the display, you can start a graphical application. For example:
# inside the container emacs &
If the above procedure did not work, there are several things to try.
First, if you have a Mac, make sure XQuartz is installed. You may need to re-boot your VM for a new installation to take effect.
Next, try running emacs from outside the container to see if the problem is with Vagrant or with the container.
If you used a different Vagrant box than the one specified in the JEDI Vagrantfile (for example, if you used one from Singularityware), if might help to set your DISPLAY variable in the container to this instead:
If the display still does not work, then you may need to explicitly grant Vagrant
access to your display through
xauth as we now describe.
Exit the container and exit vagrant. Then edit your Vagrantfile and add these
two lines (at the bottom, just before the
end in the main
Vagrant.configure("2") do |config| loop will do)
config.ssh.forward_agent = true config.ssh.forward_x11 = true
Then recreate your vagrant VM, log in, and enter the container (for example, for Singularity):
vagrant halt # restart vagrant vagrant up vagrant ssh singularity shell --bind ./vagrant_data -e <singularity-image-file>
Now create an
.Xauthority file and generate an authorization key for your display:
touch ~/.Xauthority xauth generate 10.0.2.2:0.0 . trusted
You can list your new authorization key as follows:
There should be at least one entry, corresponding to the display you entered in
xauth generate command above (you can ignore other entries, if present).
For example, it should look something like this:
10.0.2.2:0 MIT-MAGIC-COOKIE-1 <hex-key>
<hex-key> is a hexadecimal key with about 30-40 digits.
Now, copy this information and paste it onto the end of the
xauth add command as follows:
xauth add 10.0.2.2:0 MIT-MAGIC-COOKIE-1 <hex-key>
If all worked as planned, this should grant permission for vagrant to use your display.
Customizing the Vagrantfile (optional)¶
The JEDI Vagrantfile you downloaded in Step B above is already provisioned with everything you need to run JEDI, by means of the Singularity software containers.
However, it’s useful to point out a few configuration options that some users may wish to customize.
Creating your own Vagrantfile¶
First comes the choice of machine. The JEDI Vagrantfile uses a CentOS 7 operating system but there are a number of other options available, particularly with the well-maintained bento boxes provided by Vagrant. You may wish to maintain multiple virtual machines with different Linux operating systems.
For example, you can create your own Vagrantfile by entering something like this:
vagrant init bento/ubuntu-20.04
When you then run
vagrant up, this will create an Ubuntu 20.04 operating system.
You can then install Singularity manually.
The makers of Singularity also provide their own Vagrant box, with Singularity pre-installed:
vagrant init sylabs/singularity-3.0-ubuntu-bionic64
Allocating Resources for your Virtual Machine¶
The JEDI Vagrantfile comes pre-configured to allocate 16GB of memory and 18 virtual CPUS to the VM. This is the minimum resource allocation to run many tests and applications. Furthermore, if you create your own Vagrantfile, the default resource allocation will likely be insufficient to run JEDI.
You can change these resource allocations by editing the Vagrantfile.
Look for the following section that specifies the provider-specific configuration (variable names may differ).
vb.memory (in MB) and
vb.cpus fields as shown here:
config.vm.provider "virtualbox" do |vb| # [...] # Customize the amount of memory on the VM: vb.memory = "16384" # Customize the number of cores in the VM: vb.cpus = "18" # [...] end
File transfer between your Mac and the VM¶
In Step B above we created a directory called
The JEDI Vagrantfile is configured to use this directory to transfer files between
your host machine (which may be running Mac OS or Windows) and your VM.
Within the VM, this directory is mounted as
To change this, you can edit the Vagrantfile and find the section for a synced folder:
# Share an additional folder to the guest VM. The first argument is # the path on the host to the actual folder. The second argument is # the path on the guest to mount the folder. And the optional third # argument is a set of non-required options. c.vm.synced_folder "vagrant_data", "/home/vagrant/vagrant_data"
The first argument specifies the directory on the host machine, relative to the home directory of your Vagrant VM (i.e. the directory where the Vagrantfile is). The second specifies the path of the directory on the VM. You can change these paths and/or names if you wish but make sure the host directory exists before running vagrant up so it can be properly mounted.
It might also be necessary to create the mount point from within the vagrant VM:
mkdir ~/vagrant_data # from within the VM, if necessary
And, here is another tip: Use an absolute path for your guest directory.
Vagrant will complain if you use a relative path, such as
You will need root permission if you want to branch off of root (for example
the default mounting if you run
On a related note: your default user name when you enter Vagrant will be
and your home directory will be
If you want to change this you can do so by adding a line like this to your Vagrantfile:
config.ssh.username = 'vagabond'
For more information, and more options, see the Vagrant documentation.
Working with Vagrant and the JEDI Container¶
Once you have Vagrant and a container provider all set up as discussed above, your daily workflow may be as follows. You might start by going to the directory where you put your Vagrantfile. Then fire up and log in to your virtual machine.
cd $HOME/jedi-vm vagrant up vagrant ssh
From there you can enter the container and (optionally) run your startup script. For example:
singularity shell -e <singularity-image-file> source startup.sh
Now you’re in the JEDI container and you can do whatever you wish: edit files, build, compile and run JEDI, etc. If you want to use X-forwarding you’ll have to explicitly tell your Mac to accept graphical input from the Vagrant VM as described in Step G above:
#On your Mac xhost + 127.0.0.1
You may be tempted to automate this so you don’t have to enter this command every
time you start up your virtual machine. However, this is more subtle than you might expect.
Since this is the IP address of localhost, placing this command in your
file might cause your terminal application to hang when you first start it up because localhost is not yet defined.
You can avoid this by adding
xhost + to your
.bash_profile but be
careful with this because it may open you up to security vulnerabilities by allowing
clients to connect to your machine from any remote host.
Entering the explicit command above or putting it in a bash script that you execute
manually every time you log in is somewhat inconvenient but much safer.
When you’re done for the day you can exit and shut down the VM:
exit # to exit Singularity exit # to exit Vagrant vagrant halt # to shut down the virtual machine
Deleting a Vagrant VM¶
When you shut down a Vagrant virtual machine (VM) with
vagrant halt, it’s
like shutting down your laptop or workstation. When you restart the VM, you can
pick up where you left off. You’ll see all the files and directories that were there before.
This is usually desirable. However, it does mean that the VM is occupying disk space on your machine even when it is suspended. If you have created multiple VMs, this can add up. So, it is often useful to delete a VM if you are done using it.
To check vagrant’s status at any time enter
This is a useful command to know about. It will tell you all the VMs vagrant knows
about on your computer including the path where the Vagrantfile is located and the state.
vagrant up command will put the VM in a
running state while a
command will put the VM in a poweroff state.
If you want to delete one or more of these VMs, the first step is to save
any files you have on the VM that you want to preserve. This can be done by
moving them to the
~/vagrant_data directory which will still exists on your
local computer after the VM is deleted.
Now, the best way to proceed is to go to the directory where the vagrant file is and enter:
vagrant destroy # enter y at the prompt rm -rf .vagrant
The first command deletes all of the disks used by the virtual machine, with the
exception of the cross-mounted
vagrant_data directory which still exists
on your local computer. The second command resets the vagrant configuration.
This is particularly important if you re-install a new VM where another VM had
been previously. If you skip this step,
vagrant up may give you errors
that complain about mounting the
vagrant_data directory (“…it seems
that you don’t have the privileges to change the firewall…”).
This is a start, but you’re not done. As mentioned at the top of this document, Vagrant is really just an interface to VirtualBox, which provides the Linux OS. The Virtualbox VM that contains the Linux OS still exists and is still using resources on your computer. To see the VirtualBoxes that are currently installed though Vagrant, run
vagrant box list
If you used the JEDI Vagrantfile as described in Step B above, then you’ll see
one or more entries with the name
centos/7. The first step here is to
prune any that are not being used any more with
vagrant box prune
However, even this might not delete the VM you want to delete.
vagrant list to see if it is still there and if it is, you can delete it with
vagrant box remove centos/7
..or ubuntu or singularityware or whatever name is listed for the box you want to delete.
In some cases it might also help to delete the hidden
.vagrant file that
is created by vagrant in the same directory as your Vagrantfile. So, from that directory, enter:
rm -rf .vagrant
Now, this should be sufficient for most situations. Most users can stop here with confidence that they have deleted their unwanted VMs and have freed up the resources on their local computer.
However, it is possible that there might still be VirtualBox VMs on your machine
that Vagrant has lost track of. You might notice this if you try to create a new
vagrant up and it complains that “A VirtualBox machine with the
name jedibox already exists” (or a similar error message).
If this is the case, you can run VirtualBox directly to manage your VMs. This
can be done through the command line with the
vboxmanage --help for information) but we recommend the VirtualBox GUI,
which is more user-friendly.
To access the GUI on a Mac or Windows machine, just go to your Applications folder and double click on the VirtualBox icon. There you will see a complete list of all the VirtualBox VMs installed on your system and you can delete any that you don’t want by selecting the machine menu item and then remove.
Tunneling to Host from Singularity: jupyter-lab Example¶
Tunneling from Singularity to the host can enable several useful ways of interacting
between the host and the container. The benefits are multiple but some of the syntax
for doing it could be described as obscure. A motivating example use case is
jupyter-lab in Singularity and accessing it from the host machine.
This not only allows the user to run jupyter notebooks from the browser, a terminal
jupyterlab can also be used to build and run JEDI repositores. The general
outlines of establishing the tunnel below are followed by a recipe for installing
python virtual environments in the container, including
Tunneling starts in the Vagrantfile, search “forwarded_port” and set the following line as follows (with your choice of port, we use 8111 throughout):
config.vm.network "forwarded_port", guest: 8111, host: 8111
On the host machine, restart Vagrant (if necessary) and enter Vagrant using the special syntax:
vagrant halt # if running vagrant up vagrant ssh -- -L 8111:localhost:8111
Now inside Vagrant, start Singularity thusly:
singularity shell -e jedi-clang-mpich-dev_latest.sif portmap=8111:8111/tcp
The above should establish the tunnel from the host through Vagrant to Singularity.
Next we install a python virtual environment with
jupyter-lab and test the tunnel.
We choose to install our virtual environment(s) in a directory mounted into Vagrant from the host.
For example, the
vagrant_data directory as specified above in the Vagrantfile:
config.vm.synced_folder "./vagrant_data", "/home/vagrant/vagrant_data", mount_options: ["dmode=775,fmode=777"]
The following script is to be sourced inside Singularity, configring the
variable to install the virtual environment in a synced directory. The example
script installs a virtual environment and
jupyter-lab in that resulting environment:
#!/bin/bash # Configure where to install: venv_dir=~/vagrant_data/venvs/my_venv # ---------------------------------------------------- (return 0 2>/dev/null) && sourced=1 || sourced=0 if [[ sourced -eq 0 ]]; then echo "This script must be sourced." return 1 else echo "Setting up virtual env: $venv_dir" fi if [ -d $venv_dir ]; then echo "The environment ($venv_dir) already exists, returning." return 2 fi export PATH=$PATH:/home/vagrant/.local/bin/ python -m pip install --user virtualenv # If subsequent installation troubles arise, # run this line to update wheels in venv and try again: # virtualenv --upgrade-embed-wheels True $venv_dir virtualenv $venv_dir source $venv_dir/bin/activate pip install jupyter jupyterlab return 0
The above script must be sourced in
bash and will produce an error if otherwise
executed. If the script completes successfully, the virtual environment will be activated.
In future Singularity sessions, it can be activated as normal with virtual environments,
$venv_dir specified in the script to locate the
Then we can navigate to the desired root directory and start
cd /the/path/of/choice jupyter-lab --no-browser --port 8111
Jupyter will print output to the terminal, including a url to use to connect from a browser. Copy and paste the URL from jupyter into your host’s browser and go!
We note that in the current containers (March 2021), the following harmless warning is printed in the jupyter-lab session when the browser connects: Could not determine jupyterlab build status without nodejs. Also noteworthy is that testing the tunnel on any machine (Singularity, Vagrant, or the host) can be done via
jupyter-lab will register GETs in the terminal resembling
[I 2021-01-05 22:25:35.249 ServerApp] 302 GET / (::1) 0.62ms