Skip to content

Benchmarks

DiMA’s benchmarks are measured, not estimated, by one script, so that anyone can repeat them on their own hardware (Weber et al. 2019; Mangul, Martin et al. 2019). BENCHMARK.md in the repository records the last run, with its date, mode and machine:

Terminal window
scripts/bench/run.sh # full run: ≥ 10 repetitions per measurement
scripts/bench/run.sh --quick # fewer repetitions and smaller data, marked as such

The script measures wall time with hyperfine and peak memory with GNU time on the datasets committed to the repository, on synthetic scale-ups (millions of sequences, a long genome, a highly diverse alignment), on every input compression format, and across thread counts; it can also run DiMA v5.0.9 (--v509) for comparison. It writes BENCHMARK.md at the root of the repository with the hardware, operating system, software versions and the exact commands.

For your own runs, dima analyze -v prints the time spent in each phase and the peak memory, and --run-report records them in a JSON file (see Make results reproducible).

  • Mangul S, Martin LS, Hill BL, Lam AKM, Distler MG, Zelikovsky A, Eskin E, Flint J. Systematic benchmarking of omics computational tools. Nature Communications. 2019;10(1):1393. doi:10.1038/s41467-019-09406-4 ↩︎
  • Weber LM, Saelens W, Cannoodt R, Soneson C, Hapfelmeier A, Gardner PP, Boulesteix AL, Saeys Y, Robinson MD. Essential guidelines for computational method benchmarking. Genome Biology. 2019;20(1):125. doi:10.1186/s13059-019-1738-8 ↩︎