Skip to content

Run on clusters and in pipelines

DiMA is a single binary with no runtime dependencies, so it runs on login and compute nodes as it does on a laptop. This guide covers what changes in batch jobs and pipelines.

Use the Linux archive for your architecture; the x86_64-unknown-linux-gnu build runs on systems as old as CentOS 7 (glibc 2.17), and the static musl builds run anywhere. Put dima in a directory on your PATH, for example ~/.local/bin. See Install DiMA. No root access is needed.

By default DiMA uses the CPUs your job was granted on the node, not all the CPUs of the node, so it does not overload shared nodes:

SchedulerRequestDiMA uses
Slurm#SBATCH --cpus-per-task=88 (SLURM_CPUS_ON_NODE in the batch script, SLURM_CPUS_PER_TASK inside srun)
PBS Pro#PBS -l select=1:ncpus=88 (NCPUS)
Torque#PBS -l nodes=1:ppn=88 (PBS_NUM_PPN)
SGE#$ -pe smp 88 (NSLOTS; on several hosts, this host’s slots in PE_HOSTFILE)
LSF#BSUB -n 8 with -R "span[hosts=1]"8 (LSB_DJOB_NUMPROC)

--threads N or RAYON_NUM_THREADS=N override this. OMP_NUM_THREADS is ignored (DiMA does not use OpenMP). With -v, DiMA prints the thread count and the variable it came from. The exact rules are in Environment variables. The thread count never changes the results, and memory use does not grow with it. If the node limits processes and threads below what you asked for (ulimit -u, a container’s pids limit), DiMA runs with the worker threads it can start and warns (threads_refused); the run report records how many started.

#!/bin/bash
#SBATCH --job-name=dima
#SBATCH --cpus-per-task=8
#SBATCH --mem=8G
#SBATCH --time=02:00:00
set -euo pipefail
dima analyze -i alignment.fasta.zst -o results.json.zst --tsv-dir tables \
--hcs-output conserved.tsv --run-report run-report.json -v

DiMA parallelises within one alignment, so run alignments one after another with all threads, or several at once with fewer threads each (-t):

Terminal window
for f in alignments/*.fasta.zst; do
name=$(basename "$f" .fasta.zst)
dima analyze -i "$f" -o "results/$name.json.zst" --run-report "results/$name.run.json" -q
done

In a Slurm array job, give each task its own alignment and let DiMA read the task’s CPUs.

Messages go to standard error and data to standard output or files, so 2> dima.log captures the messages alone. The redrawn progress line appears only on terminals; in a batch log, -v (or --progress plain) prints one line per phase and the progress at most every 10 seconds and at least once a minute, with the elapsed time, and -q limits the output to errors. Warnings are printed once per kind of problem, with counts and examples, and are also stored in the results (run.diagnostics).

StatusMeaning
0success
2invalid options (fix the command)
3a file could not be read or written
4invalid input data
1internal error (please report it)
143, 130, 129interrupted by SIGTERM, SIGINT, SIGHUP

When a scheduler stops a job — at its time limit Slurm sends SIGTERM and, after a grace period, SIGKILL — DiMA removes its unfinished temporary files within a few seconds and ends by the same signal, so the job ends with the conventional status and no partial output is left. Re-run the job to produce the results. See Exit codes.

--run-report FILE writes a small JSON file describing the run: version and build, the equivalent command, status and exit code, start and end time, threads and their source, time per phase, CPU time, peak memory, and the size and SHA-256 checksum of the input and every output. It is written even when the run fails or is interrupted, so it is useful for accounting and debugging jobs. It contains no user or host names, no directories and no environment values. See Make results reproducible.

Under nohup (or any parent that ignores SIGHUP), DiMA keeps ignoring SIGHUP, so closing the terminal does not stop it:

Terminal window
nohup dima analyze -i big.fasta.zst -o big.json.zst -v > dima.log 2>&1 &

tmux and screen work as usual.

DiMA needs nothing but the binary, so it fits in any image, including minimal ones (use the static musl build for Alpine or scratch-like images). Two details:

  • When dima is process 1 of the container (docker run image dima … without --init), a SIGTERM from docker stop still ends it promptly, with status 143, after removing its temporary files.
  • Mount the input and output directories; outputs are written next to their destination before being renamed into place, so the output directory must be writable.

A Bioconda package, and with it BioContainers images for Docker and Apptainer/Singularity (Grüning, Dale et al. 2018; da Veiga Leprevost et al. 2017; Kurtzer et al. 2017), is planned; until the recipe is accepted, use the release archives.

In Snakemake or Nextflow rules, give DiMA the task’s CPUs explicitly (-t {threads} or -t ${task.cpus}) — workflow managers do not always run tasks under a scheduler allocation that DiMA can read — and use -q or -v as you prefer for the task log. Outputs are complete or absent, never partial, so a failed task never leaves a file a later rule could mistake for a result.

  • da Veiga Leprevost F, Grüning BA, Alves Aflitos S, Röst HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Vera Alvarez R, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017;33(16):2580-2582. doi:10.1093/bioinformatics/btx192 ↩︎
  • Grüning B, Dale R, Sjödin A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, Köster J, Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nature Methods. 2018;15(7):475-476. doi:10.1038/s41592-018-0046-7 ↩︎
  • Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLOS ONE. 2017;12(5):e0177459. doi:10.1371/journal.pone.0177459 ↩︎