Snakemake Tutorial
Bioinformatics workflows with snakemake and conda

Scaling up and using snakemake on HPC clusters#

Note: To make it clear where you should execute a certain command, the prompt is prefixed with the location, i.e.

The actual command comes only after this prefix.

Attention: Some of the commands in this part of the tutorial are specific to the University of Luxembourg’s Iris cluster. Please adapt them accordingly if you use a different HPC cluster.

Table of contents#

  1. Setup
  2. Cluster configuration for snakemake
    1. Adjust mapping step to run on multiple threads
    2. Configure job parameters with cluster.yaml
    3. Run snakemake with cluster configuration
  3. (Optional) Immediately submit all jobs
  4. (Optional) Revert the changes to your environment
  5. Useful stuff

Setup#

This part starts from the Snakefile created in the main tutorial. Please follow the steps there, first.

If you completed the main tutorial on your local machine, then you need to follow these steps to set up your HPC account:

If not already done in the setup steps, connect to your cluster access node, start a job, activate the tutorial environment and go to your working folder:

(laptop)$> ssh iris-cluster
(access)$> si
(node)$> conda activate bioinfo_tutorial
(node)$> cd $SCRATCH/bioinfo_tutorial

Cluster configuration for snakemake#

Until now the workflow just runs on a single CPU on a single machine, which is not very efficient when we have much more resources available. To speed up the computation you should check in the documentation of the software you use how it can scale. For bioinformatics tools the most common option is multithreading.

In this workflow only bowtie2 has the option to run on multiple threads.

Adjust mapping step to run on multiple threads#

We add the thread directive to the snakemake rule for the mapping step, to tell snakemake that this step can use multiple threads.

Snakemake can alter the number of cores available based on command line options. Therefore it is useful to propagate it via the built in variable threads rather than hardcoding it into the shell command. In particular, it should be noted that the specified threads have to be seen as a maximum. When Snakemake is executed with fewer cores, the number of threads will be adjusted, i.e. threads = min(threads, cores) with cores being the number of cores specified at the command line (option --cores).

— Snakemake manual - Threads

So the value for threads should be the maximum that is reasonable for the respective software. For many software the speed-up plateaus at a certain number of threads or even starts to decrease again. For a regular bowtie2 run 16 is a good maximum, but for this tutorial we will only go up to 4 because we have a small dataset.

In the mapping rule in your Snakefile add the following line after the conda directive:

  threads: 4

We also need to add the option -p {threads} to the bowtie2 command-line call, to make it actually use those threads:

    bowtie2 -p {threads} -x {params.idx} -U {input} 2> {log} | \

such that the complete mapping rule now is the following:

rule mapping:
  input: "chip-seq/{sample}.fastq.gz"
  output: "bowtie2/{sample}.bam"
  params:
    idx = "reference/Mus_musculus.GRCm38.dna_sm.chromosome.12"
  log: "logs/bowtie2_{sample}.log"
  benchmark: "benchmarks/mapping/{sample}.tsv"
  conda: "envs/bowtie2.yaml"
  threads: 4
  shell:
    """
    bowtie2 -p {threads} -x {params.idx} -U {input} 2> {log} | \
    samtools sort - > {output}
    samtools index {output}
    """

If we want to rerun the workflow to compare different options, we need to delete the output files, otherwise snakemake will not run the steps again. Fortunately snakemake has a dedicated option for this:

(node)$> snakemake all --delete-all-output -j 1

Quit your current job and start a new one with more cores to test the multithreading:

(node)$> exit
(access)$> srun --cpu-bind=none -p interactive -t 0-0:15:0 -N 1 -c 6 --ntasks-per-node=1 --pty bash -i
(node)$> conda activate bioinfo_tutorial
(node)$> cd $SCRATCH/bioinfo_tutorial

Now we also need to tell snakemake that it has multiple cores available and can run steps multithreaded or run multiple tasks in parallel. This is done with the -j option followed by the number of available cores (e.g. the number of cores you have reserved if you run it interactively).

(node)$> snakemake -j 4 -pr --use-conda bowtie2/INPUT-TC1-ST2-D0.12.bam

You should see in the output that the command-line call of bowtie2 now shows -p 4.

Check again the benchmark report:

(node)$> cat benchmarks/mapping/INPUT-TC1-ST2-D0.12.tsv
s      h:m:s   max_rss max_vms max_uss max_pss io_in io_out mean_load
6.7687 0:00:06 295.01  1728.68 291.64  291.79  0.00  16.00  0.00

Notice that the runtime has decreased, but I/O has increased.

Exercise: Try several options for -j up to the number of cores you reserved (6) and check the bowtie2 command and the values in the benchmark. Don’t forget the clean-up between the tries.

Configure job parameters with cluster.yaml#

Instead of reserving an interactive job and running snakemake inside that job, we want to use snakemake’s cluster functionality to make it submit jobs to Slurm. For this we create a configuration file named cluster.yaml to define the values for the different sbatch options.

Options under the __default__ header apply to all rules, but it’s possible to override them selectively with rule-specific options.

Create the file cluster.yaml in the same directory as the Snakefile with the following content:

__default__:
  time: "0-00:01:00"
  partition: "batch"
  nodes: 1
  ntasks: 1
  ncpus: 1
  job-name: "{rule}"
  output: "slurm-%j-%x.out"
  error: "slurm-%j-%x.err"
mapping:
  ncpus: 4

The only settings you may need to change per rule are:

as long as the software you use only does multithreading or doesn’t scale at all. With multithreading you cannot run across multiple nodes, so nodes needs to be 1 , and snakemake always just runs one task per job, so ntasks also stays at 1.

Attention: Be aware that ncpus should match the threads directive in the respective rule. If ncpus is less than threads snakemake will reserve only ncpus cores, but run the rule on the number of threads specified with threads . When running on the Iris cluster, the value for both should not exceed 28, the number of cores on a regular node.

Run snakemake with cluster configuration#

Make sure you quit your job and run the following from the access node.

Now we need to map the variables defined in cluster.yaml to the command-line parameters of sbatch. Check the documentation on the HPC website for details about the parameters.

The meaning of the option -j changes when running in cluster mode to denote the maximum number of simultaneous jobs and the option -c specifies the the number of total cores used over all jobs. For bigger workflows than this example, you should look into increasing these parameters while staying within the limits of the slurm configuration and not flooding the cluster with your jobs.

(node)$> exit
(access)$> cd $SCRATCH/bioinfo_tutorial
(access)$> conda activate bioinfo_tutorial
(access)$> snakemake all --delete-all-output -j 1

(access)$> SLURM_ARGS="-p {cluster.partition} -N {cluster.nodes} -n {cluster.ntasks} -c {cluster.ncpus} -t {cluster.time} -J {cluster.job-name} -o {cluster.output} -e {cluster.error}"

(access)$> snakemake -c 100 -j 10 -pr --use-conda --cluster-config cluster.yaml --cluster "sbatch $SLURM_ARGS"

Let’s have a look at the jobs that were submitted:

# only job allocations
(access)$> sacct -X --name="mapping","peak_calling","bigwig" --format JobID%15,JobName%15,AllocCPUS,Submit,Start,End,Elapsed

# including all steps
(access)$> sacct --name="mapping","peak_calling","bigwig" --format JobID%15,JobName%15,NTasks,AllocCPUS,Submit,Start,End,Elapsed,MaxVMSize

Check the submit and end time to see which jobs were running at the same time and when snakemake waited for jobs to finish.

After this step you should see a bunch of slurm-XXXXXX-<rule name>.out and slurm-XXXXXX-<rule name>.err files in your working directory, which contain the (error) logs of the different snakemake rules.

(Optional) Immediately submit all jobs#

Snakemake has an option to immediately submit all jobs to the cluster and tell the scheduler about the dependencies so they run in the right order. It submits the jobs one-by-one, collecting the job ID of each from the Slurm output, and then forwards those job IDs as dependencies to the follow-up jobs.

Unfortunately snakemake doesn’t parse the job submission message from Slurm cleanly, so the dependency lists look like 'Submitted', 'batch', 'job', '374519', 'Submitted', 'batch', 'job', '374520' instead of being just a list of the job IDs. Therefore, we need a wrapper script to get the dependencies right.

Create a python script called immediate_submit.py with the following content:

#!/usr/bin/env python3
import os
import sys

from snakemake.utils import read_job_properties

# last command-line argument is the job script
jobscript = sys.argv[-1]

# all other command-line arguments are the dependencies
dependencies = set(sys.argv[1:-1])

# parse the job script for the job properties that are encoded by snakemake within
job_properties = read_job_properties(jobscript)

# collect all command-line options in an array
cmdline = ["sbatch"]

# set all the slurm submit options as before
slurm_args = " -p {partition} -N {nodes} -n {ntasks} -c {ncpus} -t {time} -J {job-name} -o {output} -e {error} ".format(**job_properties["cluster"])

cmdline.append(slurm_args)

if dependencies:
    cmdline.append("--dependency")
    # only keep numbers in dependencies list
    dependencies = [ x for x in dependencies if x.isdigit() ]
    cmdline.append("afterok:" + ",".join(dependencies))

cmdline.append(jobscript)

os.system(" ".join(cmdline))

Besides the dependencies this script now also takes care of all the other Slurm options, so you don’t need to define SLURM_ARGS anymore in the shell.

Make the script executable:

(access)$> chmod +x immediate_submit.py

Run snakemake with the following command and replace <your_username> with your ULHPC user login:

(access)$> snakemake all --delete-all-output -j 1
(access)$> snakemake --cluster-config cluster.yaml -c 100 -j 10 -pr --use-conda --immediate-submit --notemp --cluster "/scratch/users/<your_username>/bioinfo_tutorial/immediate_submit.py {dependencies}"

With squeue -u <your_username> you can check the status of the submitted jobs and see when they all have finished.

(Optional) Revert the changes to your environment#

Unset PYTHONNOUSERSITE#

Remove PYTHONNOUSERSITE=True, so python finds the packages in your $HOME/.local again:

Remove conda#

If you want to stop conda from always being active:

(access)$> conda init --reverse

In case you want to get rid of conda completely, you can now also delete the directory where you installed it (default is $HOME/miniconda3).

Useful stuff#

Page outline
Last modification
2025-02-21