Getting started with Apptainer

Apptainer is an open-source container platform designed for high-performance computing (HPC). It lets you run complex software stacks, including any Docker image, without installing anything on the cluster.

Key properties on our cluster that are applied automatically:

• You are the same user inside and outside the container: same UID, same permissions, no privilege escalation possible.
• Containers are single, immutable SIF files that are easy to copy, share, and archive for reproducibility.
• Your home directory is mounted automatically, and your shell environment variables are passed into the container by default.
• Apptainer pulls Docker/OCI images directly from Docker Hub, NVIDIA NGC, and other registries.

Apptainer is installed on all nodes at /opt/apptainer/bin/apptainer (version 1.4.5). Learn more at the official Apptainer Documentation website.

When to use Apptainer (vs. EESSI)

EESSI is the primary software source on this cluster. Use Apptainer when EESSI does not cover your needs:

Situation	Recommendation
The software exists as an EESSI module	Use EESSI (module load ...)
You need a tool distributed as a Docker image (e.g. from Docker Hub or NVIDIA NGC)	Apptainer
You need an exact, frozen software environment for reproducibility	Apptainer (archive the SIF file)
Your software has complex system-level dependencies	Apptainer
You need MPI or compilers inside a container	Apptainer + EESSI combined (see MPI section)

Quick start

Pull an image from a registry and run a command in it:

# Download an image (converted to a single .sif file)
apptainer pull ubuntu.sif docker://ubuntu:24.04

# Run a command inside the container
apptainer exec ubuntu.sif cat /etc/os-release

# Or open an interactive shell inside the container
apptainer shell ubuntu.sif

# To remove the image
rm ubuntu.sif

Three things to know: inside the container you are the same user as on the host, your home directory is visible, and the host environment is inherited. If you want a clean environment inside the container, add --cleanenv.

How Apptainer works with SLURM

A container is an ordinary user process. sbatch and srun launch apptainer exactly like any other program, which means:

• Resource limits apply automatically. SLURM enforces CPU, memory, and GPU limits through cgroups, and the container lives inside them. For example, a job with --cpus-per-task=2 sees 2 CPUs inside the container.
• SLURM environment variables are visible inside the container. $SLURM_JOB_ID, $SLURM_NTASKS, $CUDA_VISIBLE_DEVICES, and so on are inherited, because Apptainer passes the host environment through.
• There is no root. Unlike Docker, nothing runs outside your job. When your job ends, everything gets removed.

An example batch job:

#!/bin/bash
#SBATCH --job-name=container_job
#SBATCH --output=stdout-%x-%j.out
#SBATCH --error=stderr-%x-%j.err
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --partition=research
#SBATCH --account=research

srun apptainer exec ubuntu.sif ./my_program

Submit with sbatch job.slurm. As well, please pull the image once on the login node before submitting.

GPU jobs

Add the --nv flag to apptainer exec. It binds the host NVIDIA driver into the container, so the container's CUDA runtime can use the GPUs SLURM allocated. Without --nv, the container has no GPU access at all.

An example with PyTorch from Docker Hub:

#!/bin/bash
#SBATCH --job-name=pytorch_container
#SBATCH --output=stdout-%x-%j.out
#SBATCH --error=stderr-%x-%j.err
#SBATCH --time=00:15:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --gpus=1
#SBATCH --partition=research
#SBATCH --account=research

srun apptainer exec --nv pytorch.sif python my_training.py

with the image pulled beforehand on the login node:

apptainer pull pytorch.sif docker://pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime

Multi-node MPI jobs in containers

The rule for MPI with containers: srun starts one container per MPI rank, and the MPI library inside the container coordinates with SLURM on the host via PMIx. mpirun inside a container cannot launch multi-node jobs.

On this cluster the simplest approach needs no MPI installed in the image at all: bind /cvmfs into the container and use EESSI's OpenMPI. In the terminology of the official Apptainer MPI documentation this is the bind model, launched with srun as described under Alternative Launchers. Example:

#!/bin/bash
#SBATCH --job-name=mpi_container
#SBATCH --output=stdout-%x-%j.out
#SBATCH --error=stderr-%x-%j.err
#SBATCH --time=00:10:00
#SBATCH --nodes=2
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --partition=research
#SBATCH --account=research

# Compile once, inside the container, with EESSI's toolchain
apptainer exec --bind /cvmfs ubuntu.sif bash -c '
    source /cvmfs/software.eessi.io/versions/2023.06/init/bash > /dev/null
    module load buildenv/default-foss-2023b
    mpicc -O2 hello_mpi.c -o hello_mpi
'

# Flags required on this cluster for OpenMPI (inherited into the containers)
export PMIX_MCA_psec=native
export OMPI_MCA_pml=ob1
export OMPI_MCA_btl=self,vader,tcp

# srun launches one container per rank; --mpi=pmix connects them
srun --mpi=pmix apptainer exec --bind /cvmfs ubuntu.sif bash -c '
    source /cvmfs/software.eessi.io/versions/2023.06/init/bash > /dev/null
    module load buildenv/default-foss-2023b
    ./hello_mpi
'

Notes:

• --mpi=pmix is required, exactly as for non-container MPI jobs on this cluster.
• If your image already contains its own MPI (e.g. an NGC image), the same srun --mpi=pmix apptainer exec ... pattern applies, provided the MPI inside the image was built with PMIx support (OpenMPI 4 and newer usually is). --bind /cvmfs is then not needed.

Building your own images

Most users never need to build anything: pulling public images covers the common cases. If you do need a custom image, unprivileged builds work directly on the login node:

apptainer build myimage.sif mydef.def

where mydef.def is an Apptainer definition file. You can also build on your own machine (with Apptainer or Docker) and copy the resulting image to the cluster with scp.

Troubleshooting

Symptom	Cause / fix
"nvidia-smi": executable file not found in $PATH	You forgot --nv, or the image has no NVIDIA userland tools. Add --nv.
ls: cannot access '/cvmfs' inside a container	EESSI is not bound in by default. Add --bind /cvmfs.
could not open image ...sif: no such file or directory	The SIF path is relative to the job's working directory. Use an absolute path or submit from the directory containing the image.
Host Python/tools leaking into the container	The environment is inherited by default. Add --cleanenv.
MPI job hangs	Check that you used srun --mpi=pmix (not mpirun) and exported the three MCA variables shown above.

Command Reference

The full CLI reference for the Apptainer version installed on this cluster can be found at Apptainer Documentation.