This is the documentation of DiMA . The newest release is .
This documentation of the development version includes changes that are not released yet. The newest release is .
Reading alignments
DiMA’s results depend only on the alignment’s residues and headers, never on how the file
happens to be formatted. A line-wrapped, lower-case, CRLF or compressed copy of an alignment gives
the same positions, variants, summaries and tables as the plain original; only the run record
of how the file was read (its name, size, checksum, compression and characters) differs. This page
explains the rules that make this so.
FASTA records
Section titled “FASTA records”The input is FASTA (Pearson & Lipman 1988): each record starts with a > line, the header, followed
by sequence lines up to the next > or the end of the file.
- The header is the whole line after
>(with a final carriage return removed). - Sequence lines are joined; line breaks never enter a k-mer. Space, tab, carriage return,
vertical tab and form feed inside sequence lines are ignored. White space at the end of a line
is dropped silently; white space between residues is ignored too but counted and reported
(
interior_whitespace), because it often means the file was reformatted by an editor. - A carriage return directly before a line feed is part of the line ending, so Windows (CRLF)
files read like Unix (LF) files. One UTF-8 byte-order mark at the start is skipped, and so are
lines before the first record that are blank or hold only white space (space, tab, carriage
return, vertical tab, form feed), as MAFFT, Clustal Omega, EMBOSS and HMMER read them; the
first other line must start with
>in its first column. - Every record must have the same number of alignment columns (characters other than white space) as the first one; an empty sequence is an error. DiMA needs an existing multiple sequence alignment, made for example with MAFFT or Clustal Omega (Katoh & Standley 2013; Sievers et al. 2011).
Problems with the file’s structure always stop the run, in every validation mode, with as much
of their location as applies — the record, line, alignment column and byte: no > before the
first sequence, a FASTQ file (@ headers), a NUL byte, a carriage return inside a header
(classic Mac OS line endings), an empty sequence, unequal lengths, UTF-16 or UTF-32 text.
Records are never skipped. Records, lines and columns are counted from 1; the byte is an offset
from 0 in the decompressed text, so it can be found with cmp, dd or a hex editor after
decompressing the file.
Compression and sources
Section titled “Compression and sources”The format is recognised from the first bytes, not from the file name: plain text, gzip
(including BGZF, the block-gzip format of SAMtools and HTSlib (Li et al. 2009; Bonfield et al. 2021)), bzip2
and Zstandard. Files made of several concatenated streams (joined with cat, or written by
pbzip2) are read to the end, and Zstandard frames written with --long windows up to 2 GiB are
accepted. xz, Unix compress (.Z), LZ4, legacy LZMA, zip and 7z files are refused with a
message that names the format and says how to convert it; an xz file, for example, can be
decompressed with xz -d or recompressed once with Zstandard, which keeps it as small
(large alignments).
The input can be a file, standard input (-i -, or no -i with a pipe), a named pipe or a shell
process substitution such as <(zcat a.fa.gz). Pipes and standard input are read in a single
pass, without seeking, so they work exactly like files.
While reading, DiMA computes the SHA-256 checksum of the stored bytes (compressed, if the file
is compressed) and records it with the file name and size in run.input, so a result can always
be matched to the file it came from.
The alphabet
Section titled “The alphabet”The alphabet — protein, DNA or RNA — decides which letters are residues. --alphabet auto (the
default) detects it with the rules of esl_abc_GuessAlphabet() from Easel, the sequence library
of HMMER (Eddy 2011), ported rule by rule. They are applied to the letter counts (case
insensitive) of the first batch of records — at most 10,000 records or 64 MiB, and at least the
first 1,000 records (or all of them, if there are fewer); with
: undetermined (too little evidence); and every letter is N: DNA; - any of E, F, I, J, L, O, P, Q, Z: protein;
- at least 98% A, C, G, T or N, and all of A, C, G and T present: DNA;
- at least 98% A, C, G, U or N, and all of A, C, G and U present: RNA;
- at least 98% amino-acid letters (or N, T, X), more D, H, K, M, R, S, V, W, Y than A, C, G, and at least 15 different letters: protein;
- otherwise undetermined.
Easel tuned these rules to make no false calls; when they do not decide, DiMA stops and asks for
--alphabet protein, dna or rna instead of guessing. (As Easel’s code is written, rule 6
cannot fire, because without rule 3’s letters at most 14 of its letters can occur; the port keeps
this.) A sequence set with substantial amounts of both T and U is undetermined too. When an
explicit --alphabet contradicts what the rules would detect, the run continues with the
explicit choice and reports alphabet_contradiction. The decision, its method (detected or
user) and the letters examined are recorded in run.alphabet.
Symbols
Section titled “Symbols”Letters are folded to upper case first (soft-masked or lower-case alignments are common) and the number folded is reported. T and U are one symbol, spelled as the resolved alphabet spells it. Every other byte belongs to one class:
| Class | Protein | DNA / RNA | Effect |
|---|---|---|---|
| residue | A C D E F G H I K L M N P Q R S T V W Y U O | A C G T / A C G U | counted |
| gap | - . _ | - . _ | the k-mer is invalid |
| missing | ~ ? | ~ ? | the k-mer is invalid |
| ambiguity | X B J Z | R Y K M S W B D H V N X I | the k-mer is invalid |
| stop | * $ ^ | * $ ^ | the k-mer is invalid |
| frameshift | # | # | the k-mer is invalid |
| white space | space, tab, CR, VT, FF | same | ignored |
| unknown | anything else | anything else | depends on the validation mode |
The sources: the nucleotide ambiguity codes are the IUPAC-IUB recommendations
(Cornish-Bowden 1985), with X and I (inosine, used in probes (Martin et al. 1985; Ohtsuka et al. 1985))
treated as ambiguous too; U (selenocysteine) and O (pyrrolysine) are genetically encoded amino
acids (Böck et al. 1991; Srinivasan et al. 2002); * marks a stop codon in translated sequences and is a
non-residue in Easel, and $ and ^ (stop), # (frameshift) and ? (no consensus) are the
symbols of the Los Alamos HIV Sequence Database alignments (Los Alamos HIV Sequence Database 2009; Los Alamos HIV Sequence Database 2024), which
keep sequences with inactivating frameshifts (Kuiken et al. 2003); . and _ as gaps and ~ as
missing data follow Easel (Eddy 2011); and spaces, tabs and carriage returns inside sequence
lines are ignored as Easel’s FASTA reader ignores them (Eddy 2011), with vertical tabs and form
feeds (the rest of the POSIX space class) ignored too. The complete table, byte by
byte, is in Symbol tables.
Treating stops and frameshifts as non-residues matters: on the HIV-1 pol protein alignment,
which contains * and #, it moves 115 positions from LS to NS, and the mean
entropy over eligible positions becomes 1.9072 bits (paper: about 1.91), against 1.9125 when they
are counted as residues, as DiMA v5.0.9 did (both computed with the independent reference
implementation in scripts/reference/, the second with v5.0.9’s symbol rules substituted).
Unknown characters and validation modes
Section titled “Unknown characters and validation modes”Characters of no class above (digits, other punctuation, letters outside the alphabet) are
handled by --validation:
strict(default): the run stops at the first one, with its record, line and column;permissive: the k-mers that contain them are excluded, and one aggregated warning (unknown_characters_excluded) gives their count and first occurrences;report: as permissive, and the table of every character found is printed.
Every character found is always counted by class in run.validation.characters. See
Validate input characters.
Header metadata
Section titled “Header metadata”With --header-format, each header is split on | into named fields, as in the paper
(Tharanga et al. 2025). The whole header line is split before any cleaning, so cleaning can never
create or remove a delimiter. Each value is then cleaned in a fixed order:
- invisible characters are removed: Unicode format characters (General Category Cf, such as zero-width spaces and bidirectional marks (UAX #44)) and control characters except tab;
- Unicode white space is trimmed from both ends;
- the value is normalised to Unicode NFC (UAX #15; Phillips 2021), so that the same text typed with precomposed or combining characters counts as one value (compatibility forms are not folded: that would change visible text);
- an empty value becomes the fill value (
--header-fillna, defaultUnknown).
Nothing else is changed: case, punctuation and the spaces inside a value are kept. Every change is counted in
run.validation.header_cleaning, together with the Unicode version used. Headers that are not
valid UTF-8 stop a strict run with the record and byte (convert the file, for example with
iconv); permissive runs replace the invalid bytes with U+FFFD and count them. What happens to
headers with the wrong number of fields, and the warnings about identifier-like fields,
missing-value tokens and near-duplicate spellings, is described in
Stratify by header metadata.
The query name
Section titled “The query name”The results carry the name of the analysed protein or gene, query_name. Unless
--query-name sets it, it is the input file name without its compression and FASTA extensions,
stripped repeatedly and case-insensitively, as MultiQC derives sample names (Ewels et al. 2016):
spike.fasta.gz gives spike. The recognised extensions are .gz .bgz .bz2 .xz .zst and
.fasta .fas .fa .fna .ffn .faa .frn .afa .aln .mfa. Standard input gives Unknown Protein,
Unknown DNA or Unknown RNA. Where the name came from is recorded in run.parameters.
References
Section titled “References”- Böck A, Forchhammer K, Heider J, Leinfelder W, Sawers G, Veprek B, Zinoni F. Selenocysteine: the 21st amino acid. Molecular Microbiology. 1991;5(3):515-520. doi:10.1111/j.1365-2958.1991.tb00722.x ↩︎
- Bonfield JK, Marshall J, Danecek P, Li H, Ohan V, Whitwham A, Keane T, Davies RM. HTSlib: C library for reading/writing high-throughput sequencing data. GigaScience. 2021;10(2):giab007. doi:10.1093/gigascience/giab007 ↩︎
- Cornish-Bowden A. Nomenclature for incompletely specified bases in nucleic acid sequences: recommendations 1984. Nucleic Acids Research. 1985;13(9):3021-3030. doi:10.1093/nar/13.9.3021 ↩︎
- Eddy SR. Accelerated Profile HMM Searches. PLoS Computational Biology. 2011;7(10):e1002195. doi:10.1371/journal.pcbi.1002195 ↩︎ 1 2 3
- Ewels P, Magnusson M, Lundin S, Käller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016;32(19):3047-3048. doi:10.1093/bioinformatics/btw354 ↩︎
- Katoh K, Standley DM. MAFFT Multiple Sequence Alignment Software Version 7: Improvements in Performance and Usability. Molecular Biology and Evolution. 2013;30(4):772-780. doi:10.1093/molbev/mst010 ↩︎
- Kuiken C, Korber B, Shafer RW. HIV sequence databases. AIDS Reviews. 2003;5(1):52-61. https://europepmc.org/article/MED/12875108 ↩︎
- Li H, Handsaker B, Wysoker A, Fennell T, Ruan J, Homer N, Marth G, Abecasis G, Durbin R, 1000 Genome Project Data Processing Subgroup. The Sequence Alignment/Map format and SAMtools. Bioinformatics. 2009;25(16):2078-2079. doi:10.1093/bioinformatics/btp352 ↩︎
- Los Alamos HIV Sequence Database. Codes and Symbols in Sequence Alignments. Los Alamos National Laboratory; 2009. https://www.hiv.lanl.gov/content/sequence/HelpDocs/IUPAC.html ↩︎
- Los Alamos HIV Sequence Database. Gene Cutter Help and Explanation. Los Alamos National Laboratory; 2024. https://www.hiv.lanl.gov/content/sequence/GENE_CUTTER/cutter-help.html ↩︎
- Martin FH, Castro MM, Aboul-ela F, Tinoco I. Base pairing involving deoxyinosine: implications for probe design. Nucleic Acids Research. 1985;13(24):8927-8938. doi:10.1093/nar/13.24.8927 ↩︎
- Ohtsuka E, Matsuki S, Ikehara M, Takahashi Y, Matsubara K. An alternative approach to deoxyoligonucleotides as hybridization probes by insertion of deoxyinosine at ambiguous codon positions. Journal of Biological Chemistry. 1985;260(5):2605-2608. doi:10.1016/S0021-9258(18)89400-5 ↩︎
- Pearson WR, Lipman DJ. Improved tools for biological sequence comparison. Proceedings of the National Academy of Sciences. 1988;85(8):2444-2448. doi:10.1073/pnas.85.8.2444 ↩︎
- Phillips A. Character Model for the World Wide Web: String Matching. W3C Working Group Note; 2021. https://www.w3.org/TR/2021/NOTE-charmod-norm-20210811/ ↩︎
- Sievers F, Wilm A, Dineen D, Gibson TJ, Karplus K, Li W, Lopez R, McWilliam H, Remmert M, Söding J, Thompson JD, Higgins DG. Fast, scalable generation of high-quality protein multiple sequence alignments using Clustal Omega. Molecular Systems Biology. 2011;7(1):MSB201175. doi:10.1038/msb.2011.75 ↩︎
- Srinivasan G, James CM, Krzycki JA. Pyrrolysine Encoded by UAG in Archaea: Charging of a UAG-Decoding Specialized tRNA. Science. 2002;296(5572):1459-1462. doi:10.1126/science.1069588 ↩︎
- Tharanga S, Ünlü ES, Hu Y, Sjaugi MF, Çelik MA, Hekimoğlu H, Miotto O, Öncel MM, Khan AM. DiMA: sequence diversity dynamics analyser for viruses. Briefings in Bioinformatics. 2025;26(1):bbae607. doi:10.1093/bib/bbae607 ↩︎
- UAX #15. Whistler K. Unicode Standard Annex #15: Unicode Normalization Forms. The Unicode Consortium; 2025. https://www.unicode.org/reports/tr15/tr15-57.html ↩︎
- UAX #44. Unicode Consortium. Unicode Standard Annex #44: Unicode Character Database. The Unicode Consortium; 2025. https://www.unicode.org/reports/tr44/tr44-36.html ↩︎