Table of Contents


    Singularity 3.x Usage on HPC Clusters

    Singularity is available on HPC clusters at Iowa State and allows users to create their own environment and let the users develop and customize their workflow without the need for admin intervention.

    As always, email if you have questions regarding the use of Singularity.

    Using docker and singularity images from existing container libraries

    List of useful container libraries

    1. Docker Based Container Libraries

    Docker Hub:

    Nvidia GPU-Accelerated Containers (NGC):

    Quay (Bioinformatics): or

    2. Singularity Container Library

    Singularity Library:

    Using Singularity

    Visit any one of the above listed libraries and search for the container image you need.

    After gaining interactive access to one of the compute nodes and loading the module, we first pull (download) the pre-built image

    module load singularity # loads the latest singularity module v3.1.1 as of 09-23-2019
    singularity pull docker://gcc                      # pulls an image from docker hub

    This pulls the latest GCC container (gcc v 9.1.0 as of 08/19/2019) and saves the image in your current working directory.

    If you prefer to pull a specific GCC version, look at the available tags for the specific container and append the tag version to the end of the container name. For example, if you need to pull the GCC v 5.3.0

    singularity pull docker://gcc:5.3.0

    Note: You can pull the images to a directory of your choosing (assuming you have write permission) by setting the variables SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR. For instance,


    Note on home directory and Singularity
    While pulling the containers, pay attention to the home directory as the cached image blobs will be saved in ${HOME}/.singularity
    Since the home directory has a limited amount of space, this can fill up quite easily. Users can change where the files will be cached by setting SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR environment variables.

    To use the executables within the container

    $ singularity exec gcc.img gcc --version  
    gcc (GCC) 9.1.0
      Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions. 
      There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

    You can use this to compile your code or run your scripts using the container’s executables.

    $ singularity exec gcc.img gcc hello_world.c -o hello_world  
    $ singularity exec gcc.img ./hello_world  
    Hello, World!

    /home/${USER}, /work, /ptmp and ${TMPDIR} are accessible via the container image.

    $ pwd
    $ ls
    catkin_ws       cuda       env_before     luarocks
    $ singularity exec gcc.img pwd
    $ singularity exec gcc.img ls
      catkin_ws       cuda         env_before          luarocks 

    Interactive access

    To gain interactive access to the container

    $ singularity shell gcc.img
    Singularity gcc.img:~> gcc --version
      gcc (GCC) 9.1.0
      Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions. 
      There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 

    To access GPUs within a container, use “–nv” option

    singualrity exec --nv <options>
    singularity shell --nv <options> 

    Slurm batch jobs with containers

    Here is a example slurm batch script that downloads a container and uses it to run a program.

      #SBATCH -N1
    #SBATCH -n20
    #SBATCH -t120
    cd $TMPDIR
    gunzip protein.fa.gz
    SINGULARITY_TMPDIR=$TMPDIR SINGULARITY_CACHEDIR=$TMPDIR singularity build clustalo.sif docker://
    singularity exec clustalo.sif clustalo -i protein.fa -o result.fasta --threads=${SLURM_NPROCS} -v

    Singularity and MPI


    Singularity supports running MPI applications via a container. In order to use singularity containers with MPI, the MPI application must be made available on both the host and within the container.

    This requires the host and container MPI versions to be compatible. One way to make sure the versions match is to use the MPI executables and libraries available on the host and bind it to the containers.

    Below is a simple MPI batch script that uses OpenMPI

    #SBATCH -N1
    #SBATCH -n4
    #SBATCH -t20
    #SBATCH -e slurm-%j.err
      #SBATCH -o slurm-%j.out
    cd $TMPDIR
      SINGULARITY_TMPDIR=$TMPDIR SINGULARITY_CACHEDIR=$TMPDIR singularity build centos-openmpi.sif docker://centos
      export SINGULARITYENV_PREPEND_PATH=/opt/rit/spack-app/linux-rhel7-x86_64/gcc-4.8.5/openmpi-3.1.3-dfskvlwuv5kwrmgx47ct57aorxdpak2c/bin
    export SINGULARITYENV_LD_LIBRARY_PATH=/opt/rit/spack-app/linux-rhel7-x86_64/gcc-4.8.5/openmpi-3.1.3-dfskvlwuv5kwrmgx47ct57aorxdpak2c/lib
    module load openmpi
    mpicc mpi_hello_world.c -o hello_world
    mpirun -np ${SLURM_NPROCS} singularity exec --bind /opt/rit/spack-app/linux-rhel7-x86_64/gcc-4.8.5/openmpi-3.1.3-dfskvlwuv5kwrmgx47ct57aorxdpak2c centos-openmpi.sif ./hello_world 

    In the above example script - openmpi is made available with the container using

    export SINGULARITYENV_PREPEND_PATH=/opt/rit/spack-app/linux-rhel7-x86_64/gcc-4.8.5/openmpi-3.1.3-dfskvlwuv5kwrmgx47ct57aorxdpak2c/bin
    export SINGULARITYENV_LD_LIBRARY_PATH=/opt/rit/spack-app/linux-rhel7-x86_64/gcc-4.8.5/openmpi-3.1.3-dfskvlwuv5kwrmgx47ct57aorxdpak2c/lib 

    and then bind mounted to the container. (By default we don’t bind mount the software directories)

    In most cases, the containers we use don’t include the Infiniband drivers, OpenMPI fails through and then falls back to the ethernet interface.

    If users are building their own containers which make use of MPI be sure to include the OFED IB driver stack.

    Singularity and Nvidia Container Library (Optional)

    Nvidia provides a container library that works with both Docker and Singularity.


    NCG Container library is now public. No account and API Key required to access the library.

    Users must have an account created in NGC and create an API key (save your key in a secure location!!!). Instructions to get started are here:

    Once you have an account created, export the required environment variables and download the containers for the library.

    Do not set the environment variables SINGULARITY_DOCKER_USERNAME and SINGULARITY_DOCKER_PASSWORD in your .bashrc file as it prevents pulling containers from libraries other than NGC!topic/singularity/9q1aTycZ6CA

    Below is an example script that downloads namd container and then runs an example on GPU node.

    #SBATCH -N1
    #SBATCH -n36
    #SBATCH -t10
      #SBATCH -pgpu
    cd $TMPDIR
    export SINGULARITY_DOCKER_USERNAME='$oauthtoken'
    tar xf stmv.tar.gz
    curl -s > stmv/stmv_nve_cuda.namd  # constant energy
    curl -s > stmv/stmv_npt_cuda.namd  # constant pressure
    nvidia-cuda-mps-server # loads nvidia modules to work with singularity GPU containers
    singularity exec --nv --bind $PWD:/host_pwd namd_2.13-singlenode.sif namd2 +ppn ${SLURM_NPROCS} +setcpuaffinity +idlepoll +devices 0,1 /host_pwd/stmv/stmv_nve_cuda.namd # stmv constant energy benchmark
    singularity exec --nv --bind $PWD:/host_pwd namd_2.13-singlenode.sif namd2 +ppn ${SLURM_NPROCS} +setcpuaffinity +idlepoll +devices 0,1 /host_pwd/stmv/stmv_npt_cuda.namd # stmv constant pressure benchmark

    Building Containers

    Since docker images are compatible with Singularity, users can write recipes for their containers either as a dockerfile or singularity definition file.

    Be aware that regular users do not have privileges to build container images from recipes on the cluster. They either need privileged access to a machine with docker or singularity installed or use docker hub/singularity library to push their recipes and build container images - we recommend the latter.


    Below are the links to get started on building docker containers -

    Dockerfile reference -

    Upload your dockerfiles to docker hub -


    Create an account at Sylabs to build a container using a definition.

    Once logged in we are presented with a text box to start writing the definition.

    Detailed documentation for a Singularity definition file is here. But below is a simple example to get you started.

    Bootstrap: docker
    From: continuumio/miniconda3
    maintainer "Name" <email address>
    apt-get update && apt-get install -y git
    # Conda install stringtie
      conda install -c bioconda stringtie 

    The header includes the Bootstrap and the label of the container.

    Most of the definition files use docker as their bootstrap as docker library is more robust and well maintained.

    This is using an existing docker image for miniconda (Python 3) available from

    Post section is any modifications or additions the user can make to the original container - in this case we are adding stringtie package to the container.

    Things to keep in mind

    • By default - work, home, ${TMPDIR} and ptmp are bind mounted to the container

    • User outside the container = User inside the container (This implies that the permissions within the container are the same as on the bare-metal compute node)

    • All the networking stack is available from within the container - if the container has Infiniband stack installed it will make use of the network

    • Having

        unset XDG_RUNTIME_DIR  

      in your slurm script is useful when you have jupyter notebook in the container. Also removes some annoying warnings from your logs

    • In the examples above, everything was done within ${TMPDIR} which will be deleted at the end of the job. Make sure you copy the output to your project directory to retain your work

    • Make sure you issue


      when using GPU nodes as this loads all the required modules to make them work with Singularity



    Workshop Materials

    The following materials from the workshop on october 14, 2021 are available for download: