Containers

Using container technologies like Apptainer and Docker on MOGON

What is a container? #

A container provides an executable environment comprised of pieces of software to facilitate portability and reproducibility of execution. You may build a container on your personal laptop, copy it to the high-performance cluster (like MOGON 2) and execute it there. The next day you may decide to run the same container at another computational facility. Instead of compiling code for some different cluster environment (different modules, different filesystems, etc.) from scratch, you just take the same container you’ve created the other day on your laptop.

About Apptainer / Singularity #

It focuses on:

• Security - create a new container including any application(s) or use an existing OCI/Docker container and secure it cryptographically to guarantee its contents.

• Portability - archive, distribute, share containers with others via container registeries, object stores, HTTP services, or shared storage.

Using Apptainer / Singularity on MOGON #

We provide Apptainer as an environment module:

module load tools/AppTainer

Interactive usage #

You can use Apptainer in your interactive job. After starting the job, ssh to the dedicated nodes and change into the folder with your Apptainer container. Most of the time, it has the extension .sif, so your container may be called my-fluffy-cow.sif. To start the shell inside this container, simply execute

apptainer shell my-fluffy-cow.sif

Cross your fingers and if everything goes right, your prompt will tell you that you are inside the container now.

Executing Commands within Apptainer Containers #

You want just to check your container quickly? Or maybe execute a specific command as a part of your data analysis? In this case, you don’t need to start a shell inside of the container. You can rather execute a specific command inside of the container. Let’s take our example my-fluffy-cow.sif container:

apptainer exec my-fluffy-cow.sif pwd

The part of the command following the name of the container will be executed in the container (in this case the working directory of the container will be printed onto the screen). After its successful execution, your prompt will return to the cluster shell.

Other use cases are described in the official documentation here.

Batch Usage #

Here is an example submit script using an Apptainer container within a batch script:

#!/bin/bash
#SBATCH -A m2_him_exp                # Specify allocation to charge against
#SBATCH --partition=himster2_exp.    # Queue name 'smp' or 'parallel' on Mogon II
#SBATCH --time=24:00:00.             # Run time (hh:mm:ss)

#Load the Apptainer module
module load tools/AppTainer

#if image is >250MB, change the TMP dir to prevent a overfull /tmp directory on node
APPTAINER_TMPDIR=/localscratch/${SLURM_JOB_ID}/apptainer_tmp/
export APPTAINER_TMPDIR
mkdir -p $APPTAINER_TMPDIR

apptainer exec /path/to/image_file.sif ./YOUR_CMD.sh

Converting Docker Images to Apptainer Images #

• It is possible to directly download a Docker image from the docker hub and convert it into an Apptainer image:

  apptainer build my-fluffy-cow.sif docker://namespace/my-fluffy-cow
  

More details can be found on official Apptainer website here.

• If you have already your own image, which is not on the docker hub, you still can convert it. Here are the steps you have to do on your local machine, where you have a Docker image called my-fluffy-cow:

1. Find out the Docker image ID with docker images. In our example it is 362df94cb548:
   

docker images
  REPOSITORY                  TAG       IMAGE ID       CREATED        SIZE
  my-fluffy-cow               latest    362df94cb548   1 hour ago     1.72GB

2. Create a tarball using Docker save command:

docker save 362df94cb548 -o my-fluffy-cow.tar

3. Copy the tarball to MOGON 2 via scp command (here into the $HOME folder assuming you have your config setup as suggested here):

scp my-fluffy-cow.tar mogon2:~/

After this, you’ll have to wait a couple of moments until the archive is copied. When it is done, login to the MOGON II and:

1. Load Apptainer module with:

module load tools/AppTainer

2. Convert Docker tarball to Apptainer image:

apptainer build my-fluffy-cow.sif docker-archive://my-fluffy-cow.tar