Schematic representation of slurm with job dependencies and job array. Made with ChtGPT

Mastering Slurm Job Submissions: From Basics to Advanced Use Cases

Introduction

In the first article, I went through the structure of an HPC cluster and the role of Slurm as a workload manager for high-performance computing (HPC) clusters. It allows researchers and scientists to work with large datasets and complex simulations efficiently. In this blog post, I’ll guide you through the intricacies of Slurm job submissions, starting with fundamental concepts and progressing to advanced use cases.

Basic Concepts

Job: A set of operations that you want to run on the cluster.

Scheduler: A piece of software (like Slurm) that:

• Manages and allocates resources for your jobs (CPU, memory, disk space, GPUs).
• Schedules the jobs based on resource availability.
• Sets up the environment.

Job Submission Script: A shell script that outlines the resources your job requires and the operations it will perform.

Simple Job Submission

To submit a job, you will use the sbatch command (see the full list of options here).

Let’s begin with a basic scenario and move toward more advanced use cases. The first example is a single, simple task. You’ll need to create a job submission script, typically a Bash script, with the following components:

1. Shebang Line: Specifies the interpreter (usually /bin/bash)
2. Comments: Enhance readability and clarity.
3. SBATCH Directives: Specify job parameters using #SBATCH directives. These lines are comments to the shell but interpreted by Slurm.

Here we are naming our job my_python_job. If not supplied, the default is slurm-%j. Naming jobs is good practice as it makes monitoring and debugging easier. We're also specifying 4 CPUs, 8 GB of RAM, and a 1-hour time limit. After that time, Slurm will terminate the job if it hasn't finished.

Following the job parameters, add the actual commands to run. If your task is computationally demanding or parallelized, you may use srun

A full script might look like this:

#!/bin/bash
# Submission script for demonstrating
# Slurm usage.

#SBATCH --job-name=my_python_job
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=01:00:00
#SBATCH --output=my_job_output.txt
#SBATCH --error=my_job_error.txt

echo "Job started at $(date)"
srun myprogram input
echo "Job ended at $(date)"

srunis useful when parallel execution is involved. For simpler jobs, direct Bash commands are sufficient.

Remember that the script runs on a compute node, not your terminal. The --output and --error options let you save stdout and stderr to separate files. If not specified, both are mixed and saved to slurm-%j.out.

To submit the script:

sbatch my_job_script.sh

You can also override #SBATCH options directly via the command line:

sbatch -–time=02:00:00 my_job_script.sh

This will override the time limit defined in the script.

Advanced Usages

Let’s look into more advanced Slurm usage, especially suited for bioinformatics tasks.

Job Dependencies

When you need to execute tasks in a sequence — where one job must finish before the next starts — Slurm’s dependency management helps.

You can chain jobs using the --dependency option.

sbatch job1.sh

Then:

sbatch --dependency=afterok:<JOB_ID> job2.sh

Key types of dependencies:

• afterok: Run only if the previous job finished successfully.
• afterany: Run after the job completes, regardless of success or failure.
• afternotok: Run only if the previous job failed.

Example: Molecular Dynamics Checkpointing

#!/bin/bash
#SBATCH --job-name=md_simulation
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --time=24:00:00
#SBATCH --output=md_%j.out
#SBATCH --error=md_%j.err

TOTAL_STEPS=1000000
STEPS_PER_JOB=200000
CHECKPOINT_FILE="md.cpt"
CURRENT_STEP=0
if [ -f "$CHECKPOINT_FILE" ]; then
  CURRENT_STEP=$(grep "step =" "$CHECKPOINT_FILE" | tail -1 | awk '{print $3}')
  echo "Resuming from step $CURRENT_STEP"
else
  echo "No checkpoint found, starting from step 0"
fi
STEPS_REMAINING=$(( TOTAL_STEPS - CURRENT_STEP ))
STEPS_TO_RUN=$(( STEPS_REMAINING < STEPS_PER_JOB ? STEPS_REMAINING : STEPS_PER_JOB ))
echo "Running $STEPS_TO_RUN steps..."
gmx_mpi mdrun -s md.tpr -deffnm md -cpi "$CHECKPOINT_FILE" -nsteps "$STEPS_TO_RUN"
echo "Done running $STEPS_TO_RUN steps."

chain jobs:

#!/bin/bash

TOTAL_STEPS=1000000
STEPS_PER_JOB=200000
NUM_JOBS=$(( (TOTAL_STEPS + STEPS_PER_JOB - 1) / STEPS_PER_JOB ))
SCRIPT_PATH="/path/to/run_md_job.sh"
JOB_ID=$(sbatch --parsable "$SCRIPT_PATH")
for ((i = 2; i <= NUM_JOBS; i++)); do
  JOB_ID=$(sbatch --parsable --dependency=afterok:$JOB_ID "$SCRIPT_PATH")
done
echo "Submitted $NUM_JOBS jobs in chain."

• Use checkpoint files if supported by your simulation tool.
• Be aware of cluster-specific job limits and queue policies.
• Use -signal to gracefully exit before timeout, allowing checkpoint saves.

Job Arrays

Job arrays are useful when you need to run the same task on multiple inputs — like applying an analysis to many files.

Example: Simple Task ID Print

Let’s start with a simple script that prints the id of each element of the array:

#!/bin/bash
#SBATCH --job-name=simple_array
#SBATCH --array=1-5
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:05:00

echo "This is task $SLURM_ARRAY_TASK_ID"

and submit it:

sbatch job_array_simple.sh

Example: Analyzing Multiple FASTA Files

Important Note: You cannot dynamically set --array size inside #SBATCH. Instead, determine it beforehand in a wrapper script:

NUM_FILES=$(ls *.fasta | wc -l)
sbatch --array=0-$(($NUM_FILES - 1)) job_array_fasta.sh

Script (job_array_fasta.sh):

#!/bin/bash
#SBATCH --job-name=fasta_analysis
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --time=01:00:00

FASTA_FILES=(*.fasta)
FASTA_FILE=${FASTA_FILES[$SLURM_ARRAY_TASK_ID]}
my_analysis_tool "$FASTA_FILE" > output_${SLURM_ARRAY_TASK_ID}.txt

Additional Tips

• Use -array=1-1000%50 to run 50 tasks concurrently (throttling).
• Use SLURM_ARRAY_TASK_ID to manage input/output dynamically.
• Combine arrays with dependencies for even more complex workflows.

Use Cases

Genomics

• Variant Calling (e.g., GATK, Freebayes)
• RNA-Seq Differential Expression (e.g., DESeq2, edgeR)
• Whole Genome Alignments (e.g., BWA)

Protein Structure

• Homology Modeling
• Docking Simulations
• MD Simulations with multiple conditions

Phylogenetics

• Tree building (e.g., RAxML)
• Ancestral sequence reconstruction

Conclusion

In this article, we explored the different Slurm submission strategies available for building bioinformatics workflows. From simple job scripts to chained executions and job arrays, each tool serves a purpose — whether to parallelize tasks or manage long-running simulations. By thoughtfully combining these options, you can optimize both resource usage and runtime.

However, avoid unnecessary complexity. Often, a simple loop or Bash script can achieve the same result more transparently. Focus on clarity, reproducibility, and adapting your workflow to your real computational needs.