Reading findings

report.findings is a list of dictionaries. Each entry describes a key, dependency, association, or data-quality issue. Start with claim for a readable summary, then inspect the columns and measurements behind it.

The example adds a concentrated sales channel and mixed-format invoice numbers to an orders table.

Categories¶

The category describes the result. To identify the method that produced it, read evidence[*].method.detector.

Views is opt-in. Its cluster findings do not identify temporal change points; see the views chapter.

Count findings by category to see what this report contains:

Methods and scores¶

The following result predicts region_id from region_name. Both the boosted model (gbm) and the decision table found this relationship, so the finding contains evidence from both.

The boosted model reports accuracy on a holdout. The decision table reports cross-validated accuracy. Inspect method.detector and validation.method before comparing scores: the models and evaluated rows differ.

Both methods report perfect accuracy in this fixture. That result is expected because each region name maps to exactly one region code. The functional-dependency findings check the mapping directly and report its violation count.

Conditional rules also include discovery and evaluation results. In the next example, PRISM finds that double_quantity <= 2 implies quantity = 1. The rule evaluator then counts matches on its diagnostic rows:

The rule holds in 38 of 38 matching diagnostic rows. Its baseline rate is 11.80%, so the reported lift is approximately 8.47. These are the rule_evaluator metrics. PRISM’s construction score describes the search that produced the rule and should not be used as a replacement for those counts.

Finding identifiers¶

IDs are derived from the category, type, subject, and specification. Use them to match the same finding across reports generated with the same schema. They do not encode rank or detector order.

Ranking¶

interestingness is a priority score. It is the mean of two scores: the detector’s own score, preserved on the finding as detector_interestingness, and a ledger evidence score that combines evidence quality, effect strength, specificity, and simplicity:

• Quality: full-data direct comparisons score 1.0; holdouts, cross-validation, and diagnostic partitions score 0.8. Other evidence uses its basis: measured 0.85, estimated 0.6, inferred 0.4.

• Strength: the metric depends on the category. Examples include the proportion of rows satisfying a functional dependency, predictive performance, and a rule’s improvement over its baseline.

• Specificity: a specified form, such as an affine formula, scores 1.0; an unspecified form scores 0.5.

• Simplicity: fewer inputs and conditions score higher, using $1/(1 + (\text{inputs} + \text{conditions})/8)$.

The chapter’s exact dependency region_id → region_name scores 0.7448. Its ledger score is 0.9889 — quality, strength, and specificity each score 1.0, while its one input gives simplicity 0.889 — and its detector score is 0.5007, because the mapping was verified over only 3 distinct inputs in 1500 rows. A score near one is not a probability of correctness.

A finding can also be demoted for triviality: when a statement restates the file’s own structure, its selection score is multiplied by a factor (0.6 per reason) and its evidence records triviality (the reasons) and triviality_factor. Demoted findings are never removed, only ranked lower. The reasons in use: an FD whose inputs contain an exact unique key (implied_by_unique_key); a float column presented as a key candidate (continuous_measure_key_candidate); a format mix that differs only in letter capitalization (capitalization_only_format_mix); and an FD from a long-format frame’s measure column to its attribute column (long_format_encoding), which holds because the levels occupy disjoint value ranges.

Report size¶

The selector sorts candidates and applies these limits:

Python exposes max_findings and max_rules. Rust’s SelectionPolicy exposes all four. report.data["selection"] shows how many candidates and distinct findings were considered and how many were included.

Related findings¶

report.groups connects findings that share columns. For example, quantity and double_quantity appear together with the findings about their relationship. Before removing a column as redundant, check whether the other columns determine it and whether the mapping has exceptions.

Filter findings¶

Select the categories relevant to your task:

To inspect a particular column, filter on subject.columns:

To select findings with measurements over the full table, inspect each evidence record’s basis and scope.sampling:

For a model result, also inspect validation.method. Cross-validation, a holdout, and a diagnostic partition use different row sets and can participate in model or finding selection. The evidence chapter walks through an example.

Acting on a finding¶

interpretation suggests possible actions. For example, a duplicate-row finding may suggest checking for repeated imports. Confirm the intended meaning of the rows before deleting them. Similarly, verify a column’s intended key or code semantics before changing a join or model.

For use on new data, test predictive results on observations that were not used to fit or select the model. A diagnostic-partition rate describes the recorded rows and may reuse discovery data.

If an expected finding is absent, inspect coverage. It lists skipped passes and search limits; an absent finding alone does not establish that the columns are independent.

Select detectors¶

onceover.available_detectors() lists detector IDs, output categories, and required detectors. Pass IDs in detectors=[...] to select which findings are requested:

Some selected detectors need another detector’s output. Decision tables, for example, use target profiles from gbm. The required pass runs to supply that data, but contributes findings only if it was also selected. Coverage identifies these required passes and the passes that were not enabled.

Findings as a DataFrame¶

report.findings_frame() returns the ledger as a Polars DataFrame: one row per finding with id, category, type, columns, inputs, target, interestingness, claim, n_evidence, and the primary evidence record’s basis, in report order. Sorting and filtering the frame is the programmatic counterpart to reading the Markdown report, which prints each finding’s evidence directly under its claim.

Long-format frames: the pivot finding¶

One finding describes the frame rather than a relation inside it. When a table is in long format — one row per (index, attribute) pair, with a single measure column — the pivot detector reports the triple and the pivot that widens it: a column_roles finding of type long_format, form pivot, whose definition names index, attribute, and measure.

Recognition is verified, not inferred from names. The frame must have exactly one continuous measure column (several continuous columns are read as an already-wide frame). The attribute is a categorical column with 2 to 32 levels, no missing values, and every level repeated at least twice; the index is every remaining column, also complete. The candidate is verified by checking that (index, attribute) is jointly unique over all rows. Frames under 30 rows are not scanned, and at most three pivot findings are reported; when two attributes complete the same key with the same measure, only the reading with fewer levels is kept, because levels become columns.

The evidence records n_rows, n_index_tuples, n_levels, the level names, and rectangularity — rows divided by (index tuples × levels), which is 1.0 for a complete panel and below it when some (index, level) cells are absent. The finding is advice: the frame the other detectors see is unchanged. Executing the recipe is the reader’s step, and it is mechanical, because the definition carries the three arguments df.pivot needs.

The wide frame itself produces no pivot finding — with three measure columns, there is no single measure left to widen around. A sparse long frame still reports, with its rectangularity below 1.0 stating how much of the panel is present.

The relationships between the levels exist only in the wide form, so at the standard and deep budgets the profiler does not stop at the pivot finding: it profiles the pivoted frame as well and merges its findings into the same report. They are tagged source_form: "pivoted wide form" in their evidence metrics, and the coverage entry pivot_profile states the wide form’s shape and how many findings it contributed. The quick budget skips the wide profile, and so do wide forms over 64 columns or under 8 index tuples.

Drift findings: periods and the permutation null¶

The drift detector engages only on a declared time axis. Select it explicitly or let the standard selection include it, and declare the axis with hints. Its findings carry one evidence entry whose metrics hold the per-period scores (periods, each with range, drift, and concept_drift), the reference period’s holdout fit (reference_r2), the largest period score (max_score, at max_period), and null_max — the largest maximum produced by permuting the period labels and rescoring, the no-drift bar the real maximum had to beat. A target whose reference model does not reach a holdout R² of 0.2 produces no finding: there is no stable relationship whose drift could be scored.

Evidence payloads: the table inside a finding¶

Some finding types carry a bespoke payload in their evidence metrics: a small table, shaped for that type, that lets the claim be inspected without a follow-up query. The scalar metrics always carry the full counts; the payload is a capped sample of the rows behind them, and it renders as a table under the finding in Markdown.

The functional-dependency payload is mapping: for an approximate dependency, the keys that map to more than one value, worst first (up to 8 keys, 3 values per key), with mapping_ambiguous_keys and mapping_keys giving the totals; for an exact dependency, the 8 largest keys. Break findings carry segments — the segmentation each breakpoint was drawn from — duplicate-row findings carry duplicate_groups with the largest groups’ values rendered, and drift findings carry periods. Each detector defines its payload with its finding type; there is no shared envelope.