Expand description
Shannon entropy of the k-mer distribution at a position, and its bias-corrected estimators.
All estimators work on the counts of counts of a position (CountProfile): how many
distinct k-mers were seen once, twice, …. That is all they need, it is tiny, and it makes the
cost independent of the k-mer length.
plug_in: the observed (maximum-likelihood) entropy H = −Σ pᵢ log₂ pᵢ (Shannon 1948), biased downwards by about (D − 1)/(2N ln 2) bits (Miller 1955; Paninski 2003).dima_corrected: the paper’s correction (Tharanga et al. 2025, Fig. 2) — entropies of subsamples of R % of the sequences are regressed on 1/n and the intercept (n → ∞) is the estimate. The paper draws one random with-replacement subsample per R; because the OLS intercept is linear in the responses, its expectation is the intercept fitted to the expected subsample entropies, which are computed here exactly (no random numbers). It is the conditional expectation of the random intercept given the data: the same mean and no larger variance (Rao 1945; Blackwell 1947), the limit of averaging many random subsets as DiMA’s predecessors did (Khan et al. 2008, 2017; Miotto et al. 2008), and identical inputs always give identical outputs. The expected intercept is never negative (seedima_corrected), so the paper’s negative-intercept rule never applies to it; DiMA v5.0.9 applies it to each random run instead, which raises its long-run average at near-conserved positions.
Every reported entropy is checked to be a finite number of at least 0 bits (estimate):
the estimators guarantee it, so a violation is a bug and stops the run as an internal error
instead of being replaced by another value (Wilson et al. 2014, Practice 5a; Hunter-Zinck et
al. 2021, Rule 5).
miller_madow: H + (D − 1)/(2N ln 2) (Miller 1955).chao_wang_jost: Chao et al. (2013), the least biased estimator in their comparison.sample_coverage: estimated sample coverage (Chao & Jost 2012), the probability that the next sequence carries an already-seen k-mer.
Structs§
- Count
Profile - Counts of counts of the k-mers at one position.
- Estimate
- The result of estimating the entropy of one position.
Constants§
- RAREFACTION_
MIN_ POINTS - Minimum number of rarefaction points for the regression: DiMA v5.0.9’s (
calculate_entropy); the paper states none. - RAREFACTION_
START - Lowest subsample percentage of the rarefaction grid (Tharanga et al. 2025, Fig. 2).
- RAREFACTION_
STEP - Step of the rarefaction grid, in per cent: Fig. 2 ends “…, 90, 95, 100%”; DiMA v5.0.9 steps by 5 from the start.
Functions§
- chao_
wang_ jost - Chao–Wang–Jost (2013) entropy estimate in bits;
Nonewhen N < 2. - dima_
corrected - The
dimaestimate in bits: the exact expectation of the paper’s procedure (Fig. 2). - estimate
- Estimates the entropy of a position with the requested estimator, applying the documented
fallbacks (support ≤ T for
dima; N = 1 for Miller–Madow and Chao–Wang–Jost). - expected_
subsample_ entropy - Expected plug-in entropy (bits) of a with-replacement subsample of
sizesequences drawn from the position’s empirical k-mer distribution:E[H(Multinomial(size, p̂))]. - miller_
madow - Miller–Madow estimate in bits, H + (D − 1)/(2N ln 2) (Miller 1955);
Nonewhen N < 2. - plug_in
- Plug-in Shannon entropy in bits, H = −Σ (c/N) log₂(c/N) (Shannon 1948);
Nonewhen N = 0. - rarefaction_
grid - The paper’s rarefaction grid for support
nand thresholdt(Fig. 2), with the step and start rounding of DiMA v5.0.9: R % from max(50, ⌈100·T/N⌉ rounded up to a multiple of 5) to 95 in steps of 5 (the 100% point is added separately). Empty when ⌈100·T/N⌉ > 95. - sample_
coverage - Estimated sample coverage Ĉ = 1 − (f₁/N)·A’ (Chao & Jost 2012, Eq. 4a), where
A’ = (N−1)f₁/((N−1)f₁ + 2f₂), or (N−1)(f₁−1)/((N−1)(f₁−1) + 2) when f₂ = 0; 1 when f₁ = 0.
Nonewhen N < 2. - subsample_
size - Subsample size ⌈R·N/100⌉ for R % of
nsequences (Fig. 2, “Upperbound”).