Singularity / Apptainer
squashfuse not found
. To overcome the error, please use the flag --unsquash
What is a container?
Container provide an executable environment comprised of pieces of software to facilitate portability and reproducibility of execution. You may build a container on your personal laptop, copy it to the high-performance cluster (like MOGON 2) and execute it there. The next day you may decide to run the same container at another computational facility. Instead of compiling code for some different cluster environment (different modules, different filesystems, etc.) from scratch, you just take the same container you've created the other day on your laptop.
About Singularity
It is an open-source project initiated at the Lawrence Berkeley National Laboratory. It is focused on:
- Integration over isolation by default. Easily make use of GPUs, high speed networks, parallel filesystems on a cluster.
- Portability and reproducibility of computations. The single file SIF container format is easy to transport and share.
- A simple, effective security model. You are the same user inside a container as outside, and cannot gain additional privilege on the host system by default. Where other container solutions allow the escalation of privileges onto the host system, this is prevented by Singularity.
Using Singularity / Apptainer on MOGON
We provide Singularity as an environment module:
$ module load tools/Singularity
After some stagnation, the Singularity project is being developed further under the name Apptainer
:
$ module load tools/Apptainer
Interactive usage
You can use Singularity in your interactive job. After starting the job, ssh
to the dedicated nodes and change into the folder with your Singularity container. Most of the time, it has extension sif, so your container may be called my-fluffy-cow.sif
. To start the shell inside this container, simply execute
$ singularity shell my-fluffy-cow.sif
Cross your fingers and if everything goes right, your prompt will tell you that you are inside the container now.
Executing Commands within Singularity Containers
You want just to check your container quickly? Or maybe execute a specific command as a part of your data analysis? In this case, you don't need to start a shell inside of the container. You can rather execute a specific command inside of the container. Let's take our example my-fluffy-cow.sif
container:
$ singularity exec my-fluffy-cow.sif pwd
The part of the command following the name of the container will be executed in the container (in this case the working directory of the container will be printed onto the screen). After its successful execution your prompt will return to the cluster shell.
Other use cases are described in the official documentation here.
Batch Usage
Example submit script when using a singularity container within a batch script
#!/bin/bash #SBATCH -A m2_him_exp # Specify allocation to charge against #SBATCH --partition=himster2_exp. # Queue name 'smp' or 'parallel' on Mogon II #SBATCH --time=24:00:00. # Run time (hh:mm:ss) #Load the Singularity module module load tools/Singularity #if image is >250MB, change the TMP dir to prevent a overfull /tmp directory on node SINGULARITY_TMPDIR=/localscratch/${SLURM_JOB_ID}/singularity_tmp/ export SINGULARITY_TMPDIR mkdir -p $SINGULARITY_TMPDIR singularity exec /path/to/image_file.sif ./YOUR_CMD.sh
Converting Docker Images to Singularity Images
Are there any plans to provide Docker support on MOGON?
NO, but you can convert your Docker image into a Singularity image and launch it on MOGON 2.- It is possible to directly download a Docker image from the docker hub and convert it into Singularity image:
$ singularity build my-fluffy-cow.sif docker://namespace/my-fluffy-cow
More details can be found on official Singularity website here.
- If you have already your own image, which is not on the docker hub, you still can convert it. Here are the steps you have to do on your local machine, where you have a Docker image called
my-fluffy-cow
:
- Find out the Docker image ID with
docker images
. In our example it is362df94cb548
:$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE my-fluffy-cow latest 362df94cb548 1 hour ago 1.72GB
- Create a tarball using Docker
save
command:$ docker save 362df94cb548 -o my-fluffy-cow.tar
- Copy the tarball to MOGON 2 via
scp
command (here into the$HOME
folder assuming you have your config setup as suggested here):$ scp my-fluffy-cow.tar mogon2:~/
After this, you'll have to wait couple of moments until the archive is copied. When it is done, login to the MOGON 2 and:
- Load Singularity module with:
$ module load tools/Singularity
- Convert Docker tarball to Singularity image:
$ singularity build my-fluffy-cow.sif docker-archive://my-fluffy-cow.tar