This is the documentation of DiMA . The newest release is .
This documentation of the development version includes changes that are not released yet. The newest release is .
Changelog
All notable changes to this project are documented in this file.
The format is based on Keep a Changelog 1.1.0, and this project adheres to Semantic Versioning 2.0.0. The “Methodology” group lists every change that can alter a result, with the sources behind it.
[1.0.0] - 2026-10-07
First release of the Rust implementation of DiMA (Tharanga et al. 2025, Briefings in Bioinformatics 26(1):bbae607, doi:10.1093/bib/bbae607). It replaces every earlier version of this repository; results of those versions are not comparable (see “Fixed”).
Methodology
- Entropy correction (
--estimator dima, the default) follows the paper’s Fig. 2 exactly: subsample sizes of R% of the sequences, with R from max(50, ⌈100·T/N⌉) rounded up to a multiple of 5 in steps of 5, plus 100%; sizes ⌈R·N/100⌉; sampling with replacement; the entropy regressed on 1/n; the intercept is the estimate, with the paper’s fallbacks (raw entropy when N ≤ T or with fewer than 5 points). It is computed as the exact expectation of the random intercept, so results are reproducible bit for bit, with the random intercept’s mean and no resampling noise (Rao 1945; Blackwell 1947, doi:10.1214/aoms/1177730497): the limit of averaging many random subsets, as DiMA’s predecessors did (Khan et al. 2017, doi:10.1186/s12920-017-0301-2). The expected intercept is never negative, so the paper’s negative-intercept rule never applies; DiMA v5.0.9 applied it to each random run, which raised its average at near-conserved positions (MERS-CoV spike protein eligible mean 0.126 bits here, about 0.142 averaged over v5.0.9 runs; HIV-1 pol unchanged). Every reported entropy is checked to be a finite number of at least 0 bits; a violation is an internal error (exit 1), never replaced by another value. On the paper’s HIV-1 pol dataset the eligible mean is 5.3504 bits (paper: ~5.35). - Alternative estimators:
chao-wang-jost(Chao, Wang & Jost 2013, doi:10.1111/2041-210X.12108),miller-madow(Miller 1955) andraw(plug-in). Each position records the estimator or fallback used. The second term of Chao–Wang–Jost is evaluated exactly in a time independent of the number of sequences (hypergeometric series or Gauss’s continued fraction: Hankin 2015, doi:10.32614/RJ-2015-022; Van Vleck 1901, doi:10.2307/1967627; Henrici & Pflüger 1966, doi:10.1007/BF02166031; Jones & Thron 1974, doi:10.1090/S0025-5718-1974-0373265-5; Alzer 1997, doi:10.1090/S0025-5718-97-00807-7), where summing it over every sequence would cost about 10% of a run and lose up to 8 digits. - Sample coverage per position (Chao & Jost 2012, Eq. 4a,
doi:10.1890/11-1952.1), with an optional
--min-coverageflag. - Positions without support (NS) have no entropy (
null/NA), never 0 bits; summaries cover positions with support above T only, with the paper’s averaging also reported asall_positions_summary. - Lowest entropy is reported next to the mean and the highest, each with its position and the number of tied positions (first position wins).
- Historically conserved sequences follow the DiMA lineage (Hu et al. 2013,
doi:10.1371/journal.pone.0059994; Abd Raman
et al. 2020, doi:10.7717/peerj.7954): positions whose
Index incidence reaches the threshold (above 50% and at most 100%, compared exactly) are
joined when their k-mers overlap or abut and agree; regions that include positions with
support ≤ T are flagged, and each region reports the exact 95% Clopper–Pearson lower bound
of its conservation (Clopper & Pearson 1934,
doi:10.1093/biomet/26.4.404).
--hcs-eligible-onlybuilds regions from positions with support above T only. - Alphabet: detected with Easel’s rules as in HMMER (Eddy 2011,
doi:10.1371/journal.pcbi.1002195) or set
with
--alphabet; lower case is folded to upper case; T and U are one symbol; gaps (- . _), missing data (~ ?), stops (* $ ^), frameshifts (#) and ambiguity codes (Cornish-Bowden 1985, doi:10.1093/nar/13.9.3021; Los Alamos HIV Sequence Database symbols) exclude the k-mers that contain them; protein U and O are residues. - Default k follows the alphabet: 9 for proteins, 27 for nucleotides. Any k from 1 to the alignment length is accepted.
- Header metadata: values are split on
|, cleaned (invisible characters removed, white space trimmed, Unicode NFC; Unicode Standard Annex #15) and empty values get the fill value.
Added
dima analyze,view,hcs,citeandcompletionssubcommands;--run-reportwrites a separate JSON record of how a run went (versions, command, times, threads, resources, checksums) without identifying details.- Outputs: complete JSON with provenance, JSONL (one position per line), four tidy TSV tables
(positions, variants, metadata, conserved sequences) and conserved-sequence JSON, each
optionally compressed (
.gz,.zst); JSON Schemas (Draft 2020-12) for every JSON output. - Typed diagnostics in the results and on standard error, each with a count, examples and a
hint;
--validation strict|permissive|report. - Input: plain, gzip/BGZF, bzip2 and zstd files (xz and other formats are refused with a hint on how to convert them), concatenated streams, standard input, pipes and FIFOs; the SHA-256 of the stored input is recorded.
dima citeprints the references in text, BibTeX, RIS or CFF;--from results.jsonprints a methods paragraph and the references of that run.CITATION.cffand.zenodo.jsondescribe the software.- Interruptions: SIGINT, SIGTERM and SIGHUP (and SIGUSR1, SIGUSR2, SIGXCPU and SIGALRM, which batch systems send before a limit) stop the run, remove unfinished outputs and end it by the same signal; outputs are written to temporary files and replace their destinations only when complete, keeping the permissions of a replaced file.
- Batch-scheduler-aware default thread count (Slurm, PBS, SGE, LSF). When the system refuses
some worker threads (a process, thread or container limit), the analysis continues with those
it started and warns (
threads_refused); the run report records both numbers. - Messages,
--help(apart from authors’ names) and the progress line are plain ASCII, so they read the same in any locale, pager, Windows console or screen reader;--progress auto|plain|off(orDIMA_PROGRESS) chooses between a line redrawn on terminals, appended lines for screen readers and logs, and nothing; colour follows--color,NO_COLOR,FORCE_COLORandCLICOLOR_FORCE. - Numerical accuracy is measured and documented up to 17.5 million sequences per position: rounding error reaches at most 5e-11 relative, at least 10^10 times below the sampling error (Darriba et al. 2018, doi:10.1093/molbev/msy014). A fix for the last digits was prototyped and is deferred, because it would change stored digits without changing any scientific result.
- Desktop application on the same library.
Changed
- New command-line names (
-s/--support-threshold,-a/--alphabet,--estimator,--query-name,--hcs-min-incidence, …); the support threshold defaults to 100 as in the paper. - Exit codes: 0 success, 1 internal error, 2 usage, 3 input/output or system error (such as a refused thread), 4 invalid input data, 130/143/129 interrupted.
- Results are identical across operating systems, thread counts and batch sizes, and for wrapped, lower-case, CRLF or compressed copies of the same alignment.
- Lines before the first record that are blank or hold only white space are skipped (DiMA
v5.0.9 stopped at any line before the first header); other text before it, including an
indented
>, is an error with its line and column. - Memory grows with the diversity of the alignment, not with the number of sequences times its length.
Removed
- The binary
.dimaformat, memory-mapping and disk-backed matrix options (--low-memory,--force-ram,--temp-dir),--allow-lowercase,--report-invalid,--no-headerand the record-length and sequence-count limits.
Fixed
- Line-wrapped FASTA files were read with line breaks inside k-mers.
- Lower-case residues were rejected, and malformed records were skipped silently.
- Concatenated bzip2 streams were truncated after the first stream.
- The equal-length check compared k-mer counts instead of alignment columns.
- k was limited to 14 (protein) or 27 (nucleotide).
- Several options had no effect; the desktop app and the command line disagreed on defaults and alphabets.