top of page
Feb 5, 2023

Intro into Singularity


Singularity is a container platform that allows users to create and manage lightweight, portable, and reproducible containers that are compatible with Linux and macOS operating systems. A container is a self-contained execution environment that packages up code and all its dependencies so that it can run on any machine.


One of the key features of Singularity is the "build" command, which can be used to create containers from various sources. These sources include the Container Library, Docker Hub, Singularity Hub, and local directories or files. The build command can create containers in two formats: the compressed read-only Singularity Image File (SIF) format, which is suitable for production, and a writable (ch)root directory called a sandbox, which is useful for interactive development.


To create a container using the build command, you need to provide a target that defines the method to be used. If the target is a URI beginning with "library://", the build command will download the container from the Container Library. If the target is a URI beginning with "docker://", the build command will download the layers from Docker Hub and assemble them into a Singularity container. If the target is a URI beginning with "shub://", the build command will download the container from Singularity Hub. If the target is a path to a local container or directory, the build command will use that container or directory as the source. If the target is a path to a Singularity definition file, the build command will use the definition file to create the container.


To download an existing container from Docker Hub, use the following command:

sudo singularity build lolcow.sif docker://godlovedc/lolcow

To create a writable sandbox directory, use the following command:

sudo singularity build --sandbox lolcow/ docker://godlovedc/lolcow

Singularity definition file

A Singularity definition file is a plain text file that contains instructions on how to build a container. It is used by the Singularity build command to create a container from scratch or customize an existing container. The definition file is written in a simple, easy-to-read syntax that includes various sections and directives.


The first line of a Singularity definition file must be the BootStrap directive, which specifies the method to be used to build the container. The supported values for the BootStrap directive are docker, library, and shub. For example, to build a container using the Docker Hub as the source, the BootStrap directive would be docker.


Here is an example of a Singularity definition file that uses the Docker Hub as the source:

BootStrap: docker
From: ubuntu:18.04

%post
    apt-get update
    apt-get install -y python3 python3-pip
    pip3 install numpy scipy

%runscript
    python3 my_script.py

The From directive specifies the base image to be used for the container. In this case, the base image is ubuntu:18.04.


The %post section contains commands that will be run during the build process. In this example, the commands update the package manager and install Python 3 and the NumPy and SciPy libraries.


The %runscript section contains the default command that will be run when the container is invoked. In this case, the command runs a Python script called my_script.py.


In addition to the BootStrap, From, %post, and %runscript sections, a Singularity definition file can also include the %files, %environment, and %labels sections to further customize the container.


The %files section allows you to specify files and directories to be included in the container. For example, you can use the following directive to include a file called my_script.py in the container:

%files
    my_script.py

You can also use wildcards to include multiple files or directories. For example, to include all files in the scripts directory, you can use the following directive:

%files
    scripts/*

The %environment section allows you to specify environment variables that will be set in the container. For example, you can use the following directive to set the PYTHONPATH environment variable:

%environment
    PYTHONPATH=/usr/local/lib/python3.6/site-packages

The %labels section allows you to specify metadata for the container, such as the container's author and version. Labels are specified in the form of key-value pairs. For example, you can use the following directive to specify the author and version of the container:

%labels
    Author Gerben Voshol
    Version 1.0

In addition to these sections and directives, you can also include pre- and post-execution scripts in your definition file. Pre-execution scripts are scripts that are run before the container is invoked, while post-execution scripts are run after the container is finished. These scripts are useful for performing tasks such as setting up or cleaning up the environment. To include a pre-execution script, use the %pre section, and to include a post-execution script, use the %post section.


Here is an example of a Singularity definition file that includes all of these sections and directives:

BootStrap: docker
From: ubuntu:18.04

%environment
    PYTHONPATH=/usr/local/lib/python3.6/site-packages

%files
    my_script.py

%post
    apt-get update
    apt-get install -y python3 python3-pip
    pip3 install numpy scipy

%runscript
    python3 my_script.py

%labels
    Author John Doe
    Version 1.0

For a complete list of supported sections and directives, refer to the Singularity documentation.


Running Singularity containers

Once you have created a container using the Singularity build command, you can use the run, exec, and shell commands to run commands within the container.


The run command allows you to execute a command within the container and display the output. For example, to run the default %runscript command within a container called lolcow.sif, you can use the following command:

singularity run lolcow.sif

If the %runscript directive is not defined, you are also able to run other commands such as ls within the container using the following command:

 singularity run container.sif ls

The exec command allows you to execute a command within the container and replace the current process with the command. For example, to start a new shell within the container, you can use the following command:

singularity exec lolcow.sif /bin/bash

The shell command allows you to enter the container and run commands interactively. For example, to enter the container in read-only mode, you can use the following command:

singularity shell lolcow.sif

To enter a writable sandbox directory, use the --writable option:

sudo singularity shell --writable lolcow/

It is important to note that, when using the shell command, you are entering the container as the current user. If the current user does not have permission to access certain files or directories within the container, you may need to use the sudo command to gain superuser privileges.


You can also use the --bind option with the run, exec, and shell commands. For example, to run a command within the container that accesses a file in the mounted host directory, you can use the following command:

singularity run --bind /data:/mnt/data lolcow.sif python my_script.py /mnt/data/input_file.txt

This mounts a host directory called /data into the container at the /mnt/data location.


It is important to note that the --bind option does not work with writable sandbox containers. If you need to access host directories or files within a writable sandbox container, you can use the --writable option along with the --bind option.


Working with Singularity images requires lots of storage space. By default Singularity will use ~/.singularity as a cache directory which can cause you to go over your /home quota. You can use the --cache option to specify a custom cache directory. For example, to use a cache directory located at /home/user/singularity_cache, you can use the following command:

singularity --cache /home/user/singularity_cache build lolcow.sif lolcow.def

Alternatively, you can also consider adding these environment variables to your ~/.bashrc file:

export SINGULARITY_CACHEDIR=/scratch/gpfs/$USER/SINGULARITY_CACHE
export SINGULARITY_TMPDIR=/tmp

In summary, the run, exec, and shell commands are useful for running commands within a Singularity container. The run command allows you to execute a command and display the output, the exec command allows you to execute a command and replace the current process with the command, and the shell command allows you to enter the container and run commands interactively.


Overall, Singularity is a powerful tool for creating and running containers on HPC systems. It supports a wide range of container formats, and offers a variety of options for customizing and executing containers.


Links to other useful resources





Comments

bottom of page