This is the documentation of DiMA . The newest release is .
This documentation of the development version includes changes that are not released yet. The newest release is .
Run on clusters and in pipelines
DiMA is a single binary with no runtime dependencies, so it runs on login and compute nodes as it does on a laptop. This guide covers what changes in batch jobs and pipelines.
Install on a cluster
Section titled “Install on a cluster”Use the Linux archive for your architecture; the x86_64-unknown-linux-gnu build runs on systems
as old as CentOS 7 (glibc 2.17), and the static musl builds run anywhere. Put dima in a
directory on your PATH, for example ~/.local/bin. See Install DiMA.
No root access is needed.
Threads
Section titled “Threads”By default DiMA uses the CPUs your job was granted on the node, not all the CPUs of the node, so it does not overload shared nodes:
| Scheduler | Request | DiMA uses |
|---|---|---|
| Slurm | #SBATCH --cpus-per-task=8 | 8 (SLURM_CPUS_ON_NODE in the batch script, SLURM_CPUS_PER_TASK inside srun) |
| PBS Pro | #PBS -l select=1:ncpus=8 | 8 (NCPUS) |
| Torque | #PBS -l nodes=1:ppn=8 | 8 (PBS_NUM_PPN) |
| SGE | #$ -pe smp 8 | 8 (NSLOTS; on several hosts, this host’s slots in PE_HOSTFILE) |
| LSF | #BSUB -n 8 with -R "span[hosts=1]" | 8 (LSB_DJOB_NUMPROC) |
--threads N or RAYON_NUM_THREADS=N override this. OMP_NUM_THREADS is ignored (DiMA does not
use OpenMP). With -v, DiMA prints the thread count and the variable it came from. The exact
rules are in Environment variables. The thread count never
changes the results, and memory use does not grow with it. If the node limits processes and
threads below what you asked for (ulimit -u, a container’s pids limit), DiMA runs with the
worker threads it can start and warns (threads_refused); the run report records how many
started.
A Slurm job
Section titled “A Slurm job”#!/bin/bash#SBATCH --job-name=dima#SBATCH --cpus-per-task=8#SBATCH --mem=8G#SBATCH --time=02:00:00set -euo pipefail
dima analyze -i alignment.fasta.zst -o results.json.zst --tsv-dir tables \ --hcs-output conserved.tsv --run-report run-report.json -vMany alignments
Section titled “Many alignments”DiMA parallelises within one alignment, so run alignments one after another with all threads, or
several at once with fewer threads each (-t):
for f in alignments/*.fasta.zst; do name=$(basename "$f" .fasta.zst) dima analyze -i "$f" -o "results/$name.json.zst" --run-report "results/$name.run.json" -qdoneIn a Slurm array job, give each task its own alignment and let DiMA read the task’s CPUs.
Messages go to standard error and data to standard output or files, so 2> dima.log captures
the messages alone. The redrawn progress line appears only on terminals; in a batch log, -v
(or --progress plain) prints one line per phase and the progress at most every 10 seconds and
at least once a minute, with the elapsed time, and -q limits the output to errors. Warnings are
printed once per kind of problem, with counts and examples, and are also stored in the results
(run.diagnostics).
Exit codes and interruptions
Section titled “Exit codes and interruptions”| Status | Meaning |
|---|---|
| 0 | success |
| 2 | invalid options (fix the command) |
| 3 | a file could not be read or written |
| 4 | invalid input data |
| 1 | internal error (please report it) |
| 143, 130, 129 | interrupted by SIGTERM, SIGINT, SIGHUP |
When a scheduler stops a job — at its time limit Slurm sends SIGTERM and, after a grace period, SIGKILL — DiMA removes its unfinished temporary files within a few seconds and ends by the same signal, so the job ends with the conventional status and no partial output is left. Re-run the job to produce the results. See Exit codes.
Run reports
Section titled “Run reports”--run-report FILE writes a small JSON file describing the run: version and build, the
equivalent command, status and exit code, start and end time, threads and their source, time per
phase, CPU time, peak memory, and the size and SHA-256 checksum of the input and every output.
It is written even when the run fails or is interrupted, so it is useful for accounting and
debugging jobs. It contains no user or host names, no directories and no environment values. See
Make results reproducible.
nohup, screen and background jobs
Section titled “nohup, screen and background jobs”Under nohup (or any parent that ignores SIGHUP), DiMA keeps ignoring SIGHUP, so closing the
terminal does not stop it:
nohup dima analyze -i big.fasta.zst -o big.json.zst -v > dima.log 2>&1 &tmux and screen work as usual.
Containers
Section titled “Containers”DiMA needs nothing but the binary, so it fits in any image, including minimal ones (use the
static musl build for Alpine or scratch-like images). Two details:
- When
dimais process 1 of the container (docker run image dima …without--init), a SIGTERM fromdocker stopstill ends it promptly, with status 143, after removing its temporary files. - Mount the input and output directories; outputs are written next to their destination before being renamed into place, so the output directory must be writable.
A Bioconda package, and with it BioContainers images for Docker and Apptainer/Singularity (Grüning, Dale et al. 2018; da Veiga Leprevost et al. 2017; Kurtzer et al. 2017), is planned; until the recipe is accepted, use the release archives.
Workflow managers
Section titled “Workflow managers”In Snakemake or Nextflow rules, give DiMA the task’s CPUs explicitly (-t {threads} or
-t ${task.cpus}) — workflow managers do not always run tasks under a scheduler allocation that
DiMA can read — and use -q or -v as you prefer for the task log. Outputs are complete or
absent, never partial, so a failed task never leaves a file a later rule could mistake for a
result.
References
Section titled “References”- da Veiga Leprevost F, Grüning BA, Alves Aflitos S, Röst HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Vera Alvarez R, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017;33(16):2580-2582. doi:10.1093/bioinformatics/btx192 ↩︎
- Grüning B, Dale R, Sjödin A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, Köster J, Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nature Methods. 2018;15(7):475-476. doi:10.1038/s41592-018-0046-7 ↩︎
- Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLOS ONE. 2017;12(5):e0177459. doi:10.1371/journal.pone.0177459 ↩︎