Skip to content

Analyse large alignments

DiMA reads an alignment once, as a stream, and never holds all of it in memory: alignments of millions of sequences can be analysed on an ordinary workstation. This guide explains what determines the time and memory a run needs.

DiMA decompresses gzip, bzip2 and Zstandard itself, so there is no need to store large alignments uncompressed. Compressors differ widely in how fast they decompress sequence data (Kryukov et al. 2020), so the format affects the speed of reading:

  • Zstandard (.zst) decompresses fastest and compresses alignments well, especially with a long matching window; DiMA reads windows up to 2 GiB (--long=31):

    Terminal window
    zstd -T0 -19 --long=31 alignment.fasta -o alignment.fasta.zst
  • gzip (including BGZF) is the most widely supported and reasonably fast to decompress.

  • bzip2 compresses well but decompresses slowly.

  • xz is not read: decompress it first, or recompress it with Zstandard once, which keeps it as small:

    Terminal window
    xz -dc alignment.fasta.xz | zstd -T0 -19 --long=31 -o alignment.fasta.zst

For the paper’s HIV-1 pol DNA alignment (9.2 MB), gzip -9 gives 741 KB, bzip2 -9 508 KB, zstd -19 --long=31 423 KB (and xz -9 462 KB). For alignments of many closely related genomes the difference grows: Zstandard’s long window finds the repeats between sequences that lie far apart, while gzip only looks back 32 KB.

With -v, DiMA reports the time spent in each phase and says so when reading the input limited the run; compressing with Zstandard is then the remedy.

DiMA’s memory does not grow with the number of sequences times their length. It holds:

  • at most two batches of input, each up to 10,000 records or 64 MiB — except the first, which holds at least 1,000 records (or all of them) to detect the alphabet, so with very long sequences it can be larger;
  • for each position, its distinct k-mers — about k + 25 bytes each — and, with metadata, about 19 bytes for each combination of k-mer and header value;
  • the results until they are written.

Memory therefore grows with the diversity of the alignment: conserved alignments need little even with millions of sequences, highly diverse ones more. The number of threads does not change it. -v prints the peak memory, and --run-report records it.

Header metadata multiplies the size of the results: every variant stores one count per value of every counted field. Count only the fields you need, and never an identifier such as an accession, which adds one entry per sequence to every variant. DiMA warns when a counted field looks like an identifier and prints the option that leaves it out; see Stratify by header metadata.

The results list every variant of every position. Write them compressed and convert later as needed:

Terminal window
dima analyze -i big.fasta.zst -o big.json.zst
dima view -i big.json.zst --tsv-dir tables # tables only when you need them

JSON Lines (-o big.jsonl.zst) holds one position per line, so other tools can process it position by position without loading the whole file.

Each output is written to a temporary file next to its destination and then renamed, so the destination’s file system needs room for the new file while an old one still exists.

DiMA splits the positions among threads; the counting scales with the number of CPUs until reading the input becomes the bottleneck. See Run on clusters and in pipelines for the default on shared machines.

  1. Compress the alignment with Zstandard.
  2. Count metadata only for the fields you need.
  3. Write compressed results (.json.zst).
  4. Run with -v once to see where the time goes and how much memory the run needs, or keep a --run-report.
  • Kryukov K, Ueda MT, Nakagawa S, Imanishi T. Sequence Compression Benchmark (SCB) database—A comprehensive evaluation of reference-free compressors for FASTA-formatted sequences. GigaScience. 2020;9(7):giaa072. doi:10.1093/gigascience/giaa072 ↩︎