start:software:containers:singularity

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
start:software:containers:singularity [2021/01/11 23:13]
ntretyak [Use cases]
start:software:containers:singularity [2021/09/14 15:26]
jrutte02 [Using Singularity on Mogon]
Line 3: Line 3:
 ===== What is a container? ===== ===== What is a container? =====
  
-Container is an executable environment that package up pieces of software in a way it is portable and reproducible. You can build a container on your personal laptop, copy it to the high-performance cluster (like MOGON 2) and execute it there. The next day you may decide to run the same container at another computational facility. Instead of compiling code for some different cluster environment (different modules, different filesystems, etc.) from scratch, you just take the same container you've created the other day on your laptop. That is cool, isn't it? 8-) +Container provide an executable environment comprised of pieces of software to facilitate portability and reproducibility of execution. You may build a container on your personal laptop, copy it to the high-performance cluster (like MOGON 2) and execute it there. The next day you may decide to run the same container at another computational facility. Instead of compiling code for some different cluster environment (different modules, different filesystems, etc.) from scratch, you just take the same container you've created the other day on your laptop. 
-===== Why Singularity? =====+
  
-Citing the official documentation of Singularity<callout type="default">Singularity was created to run complex applications on HPC clusters in a simple, portable, and reproducible way.</callout>+===== About Singularity ===== 
 + 
 +<callout type="default">[[https://sylabs.io/guides/3.5/user-guide/introduction.html|Singularity was created to run complex applications on HPC clusters in a simple, portable, and reproducible way.]]</callout>
  
 It is an open-source project initiated at the Lawrence Berkeley National Laboratory. It is focused on: It is an open-source project initiated at the Lawrence Berkeley National Laboratory. It is focused on:
Line 14: Line 15:
   * Portability and reproducibility of computations. The single file SIF container format is easy to transport and share.   * Portability and reproducibility of computations. The single file SIF container format is easy to transport and share.
  
-  * A simple, effective security model. You are the same user inside a container as outside, and cannot gain additional privilege on the host system by default.+  * A simple, effective security model. You are the same user inside a container as outside, and cannot gain additional privilege on the host system by default. Where other container solutions allow the escalation of privileges onto the host system, this is prevented by Singularity.  
  
-===== Basic usage =====+===== Using Singularity on MOGON ===== 
 + 
 +We provide Singularity as an [[start:working_on_mogon:workflow_organization:setting_up_environment_modules|environment module]]:
  
 <code bash> <code bash>
-$ module load tools/Singularity/3.6.4-Go-1.14.1+$ module load tools/Singularity
 </code> </code>
-===== Use cases ===== 
  
-=== Interactive usage === +==== Interactive usage ===
-You can use Singularity in your [[https://mogonwiki.zdv.uni-mainz.de/dokuwiki/start:working_on_mogon:slurm_submit#interactive_jobs|interactive job]]. After starting the job, ssh to the dedicated nodes and change into the folder with your Singularity container. Most of the time, it has extension //sif//, so your container may be called ''my-fluffy-cow.sif''. To start the shell inside this container, simply execute + 
 +You can use Singularity in your [[https://mogonwiki.zdv.uni-mainz.de/dokuwiki/start:working_on_mogon:slurm_submit#interactive_jobs|interactive job]]. After starting the job, ''ssh'' to the dedicated nodes and change into the folder with your Singularity container. Most of the time, it has extension //sif//, so your container may be called ''my-fluffy-cow.sif''. To start the shell inside this container, simply execute 
 <code bash> <code bash>
 $ singularity shell my-fluffy-cow.sif $ singularity shell my-fluffy-cow.sif
Line 30: Line 33:
 Cross your fingers and if everything goes right, your prompt will tell you that you are inside the container now.  Cross your fingers and if everything goes right, your prompt will tell you that you are inside the container now. 
  
-=== Usage as a "black box" ===+=== Executing Commands within Singularity Containers  ===
  
-You want just to check your container quick? Or maybe execute a specific command as a part of your data analysis? In this case, you don't need to start a shell inside of the container. You can rather execute a specific command inside of the container. Let's take our beloved ''my-fluffy-cow.sif'' container+You want just to check your container quickly? Or maybe execute a specific command as a part of your data analysis? In this case, you don't need to start a shell inside of the container. You can rather execute a specific command inside of the container. Let's take our example ''my-fluffy-cow.sif'' container:
  
 <code bash> <code bash>
Line 42: Line 45:
  
 Other use cases are described in the official documentation [[https://sylabs.io/guides/3.7/user-guide/introduction.html#use-cases|here]]. Other use cases are described in the official documentation [[https://sylabs.io/guides/3.7/user-guide/introduction.html#use-cases|here]].
-===== Are there any plans to provide Docker as well? =====+===== Converting Docker Images to Singularity Images ===== 
 + 
 +<callout type="info" title="Are there any plans to provide Docker support on Mogon?" icon="true"> 
 +NO, but you can convert your Docker image into a Singularity image and launch it on MOGON 2. 
 +</callout> 
 + 
 + 
 +  * It is possible to directly download a Docker image from the docker hub and convert it into Singularity image: <code bash>$ singularity build my-fluffy-cow.sif docker://namespace/my-fluffy-cow</code> More details can be found on official Singularity website [[https://singularity.lbl.gov/docs-build-container|here]]. 
 + 
 +  * If you have already your own image, which is not on the docker hub, you still can convert it. Here are the steps you have to do **on your local machine**, where you have a Docker image called ''my-fluffy-cow'': 
 + 
 +  - Find out the Docker image ID with ''docker images''. In our example it is ''362df94cb548'':<code bash>$ docker images 
 +REPOSITORY                  TAG       IMAGE ID       CREATED        SIZE 
 +my-fluffy-cow               latest    362df94cb548   1 hour ago     1.72GB 
 +</code> 
 +  - Create a tarball using Docker ''save'' command:<code bash>$ docker save 362df94cb548 -o my-fluffy-cow.tar</code> 
 +  - Copy the tarball to MOGON 2 via ''scp'' command (here into the ''$HOME'' folder assuming you have your config setup as suggested [[https://mogonwiki.zdv.uni-mainz.de/dokuwiki/start:mogon_cluster:access_from_outside_unix#using_the_ssh_config_file|here]]):<code bash>$ scp my-fluffy-cow.tar mogon2:~/</code> 
 + 
 +After this, you'll have to wait couple of moments until the archive is copied. When it is done, login to the MOGON 2 and:
  
-**NO**+  - Load Singularity module with: <code bash>$ module load tools/Singularity</code> 
 +  - Convert Docker tarball to Singularity image: <code bash>$ singularity build my-fluffy-cow.sif docker-archive://my-fluffy-cow.tar</code>
  • start/software/containers/singularity.txt
  • Last modified: 2021/09/14 15:27
  • by jrutte02