Skip to content

JSON outputs

DiMA writes three kinds of JSON document. Each has a JSON Schema (Draft 2020-12) whose $id is its published address:

DocumentWritten bySchema
Resultsdima analyze (-o file.json, or standard output), dima view -O jsonresults.schema.json
Conserved sequencesdima analyze --hcs-output file.json, dima hcshcs.schema.json
Run reportdima analyze --run-report file.json, the desktop apprun-report.schema.json

Conventions shared by all three:

  • Versioned. schema_version is "1.0.0" for the results and conserved sequences and "1.1.0" for the run report (SemVer). A new minor version may add keys, diagnostic codes or values of an enumerated field (an estimator, method or fallback name); a new major version may change the meaning of existing ones. dima view, dima hcs, dima cite and the desktop app refuse results of another major version. Results of a newer minor version are read, leaving out keys this version does not know (dima view warns when it writes them back); a value it does not know stops the reading with a message naming the newer schema, because guessing its meaning could mislead.
  • Undefined is null. A value that cannot be computed (an entropy without support, a 0 / 0 ratio, a summary without eligible positions) is null, never 0 or NaN. Every number is finite.
  • Positions are 1-based alignment columns.
  • Entropies are in bits; incidences are percentages (0–100); coverages and lower bounds are proportions (0–1).
  • Numbers are written in their shortest exact form; the JSON is compact unless --pretty is given or it is written to a terminal, and always ends with a newline. Files named .json.gz or .json.zst are compressed; DiMA reads them back by content.
  • Every key is always present (no optional keys), so a reader can rely on the shape.

The results of dima analyze (JSONL output holds the same results[] entries, one per line, without the summaries and run).

KeyTypeMeaning
schema_versionstring"1.0.0"
query_namestringName of the analysed protein or gene (see the query name)
sequence_countintegerSequences in the alignment
support_thresholdinteger ≥ 1The support threshold T
kmer_lengthinteger ≥ 1 k
low_support_countintegerPositions with support ≤ T (NS + LS + ELS)
average_entropynumber or nullMean entropy over eligible positions (support > T )
lowest_entropyextremum or nullLowest entropy among eligible positions
highest_entropyextremum or nullHighest entropy among eligible positions
position_countintegerPositions, L − k + 1
eligible_position_countintegerPositions with support > T
support_label_countsobject{"NS": …, "LS": …, "ELS": …, "eligible": …}
all_positions_summaryobjectAll-positions summary (DiMA v5.0.9’s averaging)
coverage_summaryobjectCoverage summary
runobjectProvenance of the run
resultsarrayOne position per k-mer position, in order
KeyTypeMeaning
positioninteger ≥ 1First position with this value
entropynumber ≥ 0The value, in bits
tied_position_countinteger ≥ 1Positions with exactly this value
KeyTypeMeaning
rulestringThe averaging rule, in words
position_countintegerPositions averaged (all of them)
average_entropynumber or nullMean with NS positions as 0 bits and LS/ELS positions with their reported entropy (the observed entropy with the dima and raw estimators)
lowest_entropy, highest_entropyextremum or nullExtrema under the same rule (first position, 1-based)
KeyTypeMeaning
evaluatedintegerPositions with a sample coverage (support ≥ 2)
min, meannumber or nullLowest and mean sample coverage
min_coveragenumber or null--min-coverage, if given
below_min_countinteger or nullPositions below --min-coverage, if given

One entry of results[] (and one line of JSONL output).

KeyTypeMeaning
positioninteger ≥ 1Alignment column of the k-mer’s first residue
low_support"NS", "LS", "ELS" or nullSupport 0, below T , equal to T ; null when eligible
supportintegerSequences with a valid k-mer here ( N )
distinct_variants_countintegerDistinct non-Index variants
entropynumber or nullEntropy in bits by entropy_method; null without support
observed_entropynumber or nullPlug-in (observed) entropy; null without support
entropy_methodstringdima, chao_wang_jost, miller_madow, plug_in or not_estimable
entropy_fallbackstring or nullWhy the requested estimator was not used: support_at_or_below_threshold, too_few_points, degenerate_regression (estimator dima), single_observation (Miller–Madow, Chao–Wang–Jost)
distinct_variants_incidencenumber or nullNon-Index variants per 100 non-Index sequences; null when there are none ( 0 / 0 )
total_variants_incidencenumber or nullNon-Index sequences per 100 supporting sequences; null without support
sample_coveragenumber or nullEstimated sample coverage; null when support < 2
below_min_coverageboolean or nullCoverage below --min-coverage; null without the option or without coverage
diversity_motifsarrayThe variants, most frequent first; empty without support

The formulas are in Entropy and its correction and Diversity motifs.

KeyTypeMeaning
sequencestringThe k-mer, upper case
countinteger ≥ 1Sequences carrying it
incidencenumbercount / support × 100
motif_short"I", "Ma", "Mi", "U"Motif class
motif_long"Index", "Major", "Minor", "Unique"The same, spelled out
metadataobject or nullWith --header-format: for each stratified field (in header-format order), an object mapping each value (sorted) to the number of this variant’s sequences with that value; null without --header-format

Example (MERS-CoV spike position 25, --header-format 'Accession|Host|Geolocation|Year' --metadata-fields 'Host|Geolocation|Year'):

{
"sequence": "SFKSACIEV",
"count": 14,
"incidence": 8.235294117647058,
"motif_short": "Ma",
"motif_long": "Major",
"metadata": {
"Host": { "Camelus": 2, "Camelus dromedarius": 9, "Homo sapiens": 3 },
"Geolocation": { "Oman": 2, "United Arab Emirates": 12 },
"Year": { "2014": 3, "2015": 11 }
}
}

The deterministic provenance of the run: everything needed to interpret and reproduce the results, and nothing that varies between runs (no times, host names, paths or thread counts — those are in the run report).

KeyMeaning
tool{"name": "dima", "version": …}: the DiMA version that wrote the file
parametersEvery analysis parameter as {"value": …, "origin": …}, where origin is default, user (given on the command line or in the app) or derived (from the data, such as the alphabet and the default k): alphabet, kmer_length, support_threshold, estimator, validation, min_coverage, query_name, header_format, metadata_fields, header_fill
alphabetThe alphabet decision: resolved (protein, dna, rna), method (detected or user), records_examined, residues_examined, t_u_folded (T or U letters folded to the alphabet’s spelling) and contradiction (why the data disagree with an explicit --alphabet, or null)
alignment_lengthAlignment columns L
inputThe input: name (file name without directories, - for standard input), size_bytes (stored size), checksum ({"type": "sha-256", "value": …} of the stored, possibly compressed, bytes) and compression (none, gzip, bzip2, zstd)
validationWhat reading found: mode, records, lowercase_folded, whitespace (trailing and interior line counts), characters (every byte found in sequence lines: char — null unless the byte is a visible ASCII character, so also for the space —, byte, class and count), unknown_total, header_utf8_replacements and header_cleaning (null without --header-format; otherwise unicode_version, trimmed, invisible_removed by code point, nfc_changed, fill_uses; they count the values of stratified fields)
metadataNull without --header-format; otherwise fields (all field names), stratified (the stratified ones), probe_records, mismatches (count and the first 20 headers with the wrong number of fields: record, line, header, expected, found) and fill_uses (stratified values replaced by the fill value: empty values, and every stratified field of a header with the wrong number of fields)
entropyestimator, log_base (2) and rarefaction: for the dima estimator, the constants of the paper’s procedure (grid paper_fig2: Fig. 2’s grid with the step and start rounding of DiMA v5.0.9; size_rounding ceil, start_rounding ceil_to_5, sampling expected_with_replacement, min_points 5, DiMA v5.0.9’s minimum); null for other estimators
diagnosticsThe diagnostics of the run: code, severity (info or warning), message, count, up to 5 examples and a hint (or null)
referencesThe references of the methods the run used, for dima cite --from: key, citation (full text), doi and url (either may be null). The first two are always the DiMA paper and this software version

Written by --hcs-output and dima hcs. See Historically conserved sequences.

KeyMeaning
schema_version"1.0.0"
tool{"name": "dima", "version": …}: the version that computed the regions
sourceThe results the regions were computed from: query_name, kmer_length, support_threshold, sequence_count, input_checksum (as in run.input), tool_version and results_schema_version
parametersthreshold_percent (above 50, at most 100), merge_rule (overlap_or_adjacent), eligibility (all_positions, or support_gt_threshold with --hcs-eligible-only) and confidence ({"method": "clopper_pearson", "level": 0.95})
region_count, flagged_region_countRegions, and regions that include positions with support ≤ T
regionsThe regions, in column order
diagnosticshcs_low_support_regions when regions are flagged (same shape as in the results)
referencesThe references of the method
KeyTypeMeaning
idinteger ≥ 1Region number
start_column, end_column, lengthintegersAlignment columns covered
sequencestringThe conserved sequence
min_incidencenumberLowest Index incidence among the members (%)
min_supportinteger ≥ 2Lowest support among the members
conservation_lower_bound_95numberLowest exact 95% Clopper–Pearson lower bound among the members (proportion)
low_support_positionsarray of integersEvery position in the region’s span with support ≤ T , member or not
flaggedbooleanlow_support_positions is not empty
split_from_previousbooleanThe region starts where a k-mer overlapping the previous region disagreed with it
membersarrayThe qualifying positions: position, kmer, count, support, incidence and conservation_lower_bound_95

Written by --run-report PATH (and the desktop app’s run report export), also when the run fails or is interrupted. It describes how the run went and is kept separate from the results, which stay byte-identical from run to run. It never contains host or user names, directories, the raw command line or environment values.

KeyMeaning
schema_version"1.1.0" (1.1.0 added threads.started)
toolname (dima or dima-gui), version and target (the Rust target triple of the binary, e.g. x86_64-unknown-linux-gnu)
commandThe equivalent command, rebuilt from the parsed options: long option names, file names without directories
status, exit_code, errorcompleted, failed or cancelled; the exit code; the error report with directories removed (null unless failed)
start_time, end_timeUTC, RFC 3339 with milliseconds
threadscount requested, started (the same, or fewer when the system refused some; null if the run stopped before starting them) and source (--threads, RAYON_NUM_THREADS, available_parallelism, the desktop app’s settings, or the name of the scheduler variable that limited the count); null if the run stopped earlier
platformos, arch, logical_cpus
resourceswall_seconds, cpu_user_seconds, cpu_system_seconds and peak_rss_bytes (null where not measured, as in the desktop app), waited_for_input_seconds (time the counting threads waited for the reader; null when the analysis failed or was cancelled)
phasesWall time of each phase reached: opening, counting, finalising, writing (for a run that failed or was cancelled, the phases it had reached, the last one until it stopped)
workrecords read and positions analysed
inputs, outputsEach file’s role (alignment, results, results_jsonl, positions_table, variants_table, metadata_table, hcs_table, hcs), name, size_bytes and sha256 of its stored bytes