Extreme values - onceover

The surprises pass reports numeric values that sit apart from the rest of their column — far enough that the distance itself is the finding. It runs at every budget and emits intervals findings of type extreme_value (form isolation_gap): the extreme values and their row indices, the edge of the remaining bulk, and the size of the gap between them, stated as a multiple of the bulk’s central spread.

The pass exists because a column card does not show this. A maximum is one number among the summary statistics whether it sits just past the bulk or a full spread away from it, and a mean or a standard deviation absorbs the extreme into the quantity meant to expose it.

The same pass runs a second scan over categorical columns. It looks for the mirror image of an extreme value: not a number far from its column’s bulk, but a level that appears inside a context where almost every row carries a different one. Those findings are type categorical_outlier; the test is described under Pattern-breaking categorical values below.

The isolation test¶

Each tail of a numeric column is tested separately. The candidates are the values past the Tukey fence (Q3 + 1.5 · IQR at the top, Q1 − 1.5 · IQR at the bottom). Candidates are tested as a cluster of one to three values, not one at a time: the cluster is isolated when the gap between its nearest member and the edge of the remaining bulk is at least 1.0 × the bulk’s central spread (p90 − p10 of the remainder).

Testing the cluster rather than each value matters. Two adjacent extremes have a near-zero gap between them, so a test that measures each value’s distance to its nearest neighbour sees the pair as ordinary company for each other and reports neither. Measured against the bulk instead, the pair’s joint gap is what it is.

The second scale¶

For a column whose values are all positive, isolation must hold on two scales: the raw scale above, and the log scale at half the bar (a log gap of at least 0.5 × the bulk’s log-scale central spread). A right-skewed positive column — a lab measurement, a count, an amount — has a long raw-scale tail by construction; distances that look isolated in raw units are the distribution’s ordinary shape in ratio units. The log test is what separates that shape from a value that is extreme relative to the column’s own proportions.

Extremes within a subset¶

A value can be unremarkable in its column and isolated inside one category of it. The pass therefore repeats the test within each level of every categorical column with 2 to 32 levels, for scopes holding at least 30 target values. A finding from a subset names the condition in its claim. Values already reported by the column-wide test are not reported again per subset.

The example below builds all three cases into one frame: a pair of adjacent extremes in age (148 and 151 against a bulk that ends at 82), two values (79 and 82) that are ordinary in the column but isolated inside cohort_b = true, whose other ages stop at 45, and a lognormal lab_value whose raw tail — maximum 208.8 against a median near 20 — produces no finding under the log-scale test.

import random

import polars as pl

import onceover

ages, cohort = [], []
for i in range(600):
    if i % 4 == 0:
        cohort.append(True)
        ages.append(20 + (i // 4) % 26)        # 20..45 within cohort_b
    else:
        cohort.append(False)
        ages.append(20 + i % 41)               # 20..60 outside it
ages[301] = 79
cohort[301] = True                             # subset extremes
ages[302] = 82
cohort[302] = True
ages[500] = 148
ages[501] = 151                                # the adjacent pair

rng = random.Random(11)
lab = [round(rng.lognormvariate(3.0, 0.9), 2) for _ in range(600)]

df = pl.DataFrame({"age": ages, "cohort_b": cohort, "lab_value": lab})
report = onceover.profile(df)

for f in report.findings:
    if f["specification"].get("form") == "isolation_gap":
        print(f["claim"])
        print("   evidence:", {k: f["evidence"][0]["metrics"][k]
              for k in ("bulk_edge", "gap", "gap_ratio", "extreme_rows")})
print("lab_value maximum:", df["lab_value"].max(),
      "- no extreme_value finding for it")

age reaches 151 — isolated from the rest of the column: 99.67% of values are <= 82, and the gap of 66 is 2.1x the central spread (p90-p10 = 32) of the remainder
   evidence: {'bulk_edge': 82.0, 'gap': 66.0, 'gap_ratio': 2.0625, 'extreme_rows': [501, 500]}
within cohort_b = true (152 rows), age reaches 82 — isolated from the rest of the subset: all other values are <= 45, a gap of 34 (1.7x the subset's central spread, p90-p10 = 20.2)
   evidence: {'bulk_edge': 45.0, 'gap': 34.0, 'gap_ratio': 1.6831683168316818, 'extreme_rows': [302, 301]}
lab_value maximum: 208.8 - no extreme_value finding for it

Pattern-breaking categorical values¶

A categorical value has no distance from a bulk, so the isolation test does not transfer. The categorical scan instead fixes a context — a conjunction of levels from other columns, such as stem_cankers = above-sec-nde — and asks whether one level of the target column is near-universal inside it. A context qualifies when it holds at least 30 rows, one level accounts for at least 99% of them, and at most 3 rows carry a different level. Those rows are the finding.

Contexts run to three conditions. Single levels of every eligible column are tested first. Deeper contexts are built from the 12 condition columns whose single levels come closest to the bar for that target: pairs among their levels, and triples formed by extending pairs whose dominant share is already at least 0.95. Candidates are ranked by the binomial probability of the dominant count under the dominant level’s rate in the full column, so a context that merely restates the column’s overall distribution ranks below one where the level is ordinary globally and near-universal locally. At most 3 findings are reported per target column, and a breaking row is reported once per target.

The example below plants one breaking row in a frame built from four 100-row blocks. In the first block (c1 = a, c2 = b), outcome is y in every row but one: row 42 carries x. No single condition separates anything — c1 = a and c2 = b each hold outcome = x in about half their rows. The same pattern also reads in the other directions: within c2 = b and outcome = x, c1 is w except at row 42. The scan reports the readings as one finding with an evidence record for each, because the same row breaks all of them.

c1, c2, outcome = [], [], []
blocks = [("a", "b", "y"), ("a", "z", "x"), ("w", "b", "x"), ("w", "z", None)]
for x1, x2, val in blocks:
    for i in range(100):
        c1.append(x1)
        c2.append(x2)
        outcome.append(val if val is not None else ("x" if i % 2 == 0 else "y"))
outcome[42] = "x"          # the one breaker of the (a, b) pattern

df = pl.DataFrame({"c1": c1, "c2": c2, "outcome": outcome})
report = onceover.profile(df)

for f in report.findings:
    if f.get("type") == "categorical_outlier":
        print(f["claim"])
        for e in f["evidence"]:
            print("   evidence:", e["metrics"])

within c2 = b and outcome = x (101 rows), c1 is w in 99.0% of rows — 1 row breaks the pattern: row 42 (c1 = a)
   evidence: {'dominant_level': 'y', 'dominant_share': 0.99, 'log10_significance': -40.65831127154081, 'n': 100, 'outlier_count': 1, 'outlier_rows': [42], 'outlier_values': ['x']}
   evidence: {'dominant_level': 'w', 'dominant_share': 0.9900990099009901, 'log10_significance': -28.395426288515697, 'n': 101, 'outlier_count': 1, 'outlier_rows': [42], 'outlier_values': ['a']}
   evidence: {'dominant_level': 'z', 'dominant_share': 0.9900990099009901, 'log10_significance': -28.395426288515697, 'n': 101, 'outlier_count': 1, 'outlier_rows': [42], 'outlier_values': ['b']}

Reading the evidence¶

The evidence record carries the test’s quantities, so the claim can be checked without re-running it: bulk_edge (the largest bulk value on the tested side), gap and gap_ratio (the distance to the nearest extreme, and that distance divided by the bulk’s central spread), central_spread_p90_p10, extreme_values with extreme_rows (their 0-based row indices), side, and n, the number of values in the scope tested.

A categorical_outlier finding’s evidence carries the context’s quantities: n (rows in the context), dominant_level and dominant_share, outlier_count, outlier_rows with outlier_values, and log10_significance, the ranking statistic. The claim states one framing of the pattern; the finding’s subject names every column involved, and each further framing the scan found appears as its own evidence record.

What the pass does not report¶

A value that is surprising only against a prediction. A result that is ordinary for its column but wrong for its row — high given the patient’s other measurements, low given the account’s history — is surprising relative to a model of the other columns, not relative to the bulk. No finding here speaks to that form.
Clusters of more than three extremes. A separated group of four or more values is a second mode or a merged subpopulation, not an extreme of this bulk; other passes describe it.
Small scopes. A subset or context with fewer than 30 values is not tested: its spread or share estimate would rest on the same handful of values being judged.
Categorical contexts past the search limits. The categorical scan tests text and boolean targets, contexts of at most three conditions, and — below single conditions — only the 12 condition columns whose single levels come closest to the bar. A pattern that needs a fourth condition, or a first condition far from the bar, is out of reach. The unconditioned case is not reported either: a column that is 99% one level overall is a concentration finding, not a row-level surprise.