Step discontinuities

The breaks pass finds level changes in an ordered series. This chapter covers the cross-sectional counterpart: a level jump of one column across a threshold in another column, or across a diagonal boundary in two columns, in a table with no meaningful row order. The detector id is discontinuities; it runs at the standard and deep budgets and reports category changes, type step_discontinuity, form step.

Two candidate families are scanned for each numeric target column:

axis steps: a step indicator 1{x > c} for every other numeric column x, at 20 interior quantile cutpoints of x;
diagonal steps: a step indicator across a line cos(theta) * zx + sin(theta) * zy > c in each pair of columns (standardized), over 16 directions. Directions within one step of an axis are skipped; the axis scan covers them. When there are more than 20 numeric columns, pairs are restricted to the 10 axes whose axis scan scored best for that target.

Each candidate is fitted by ordinary least squares against a smooth background — [1, u, u^2] for an axis step (with u the standardized axis), [1, zx, zy] for a diagonal step — so the step coefficient measures the jump net of the smooth relationship. A grid-selected threshold is then refined on a finer grid inside the winning interval, because a threshold placed one grid cell away from the true boundary dilutes the jump estimate (the smooth background absorbs part of it).

Selection honesty: split halves¶

Rows are split deterministically into a fit half and an evaluation half. The fit half selects the candidate (largest |t_fit|) and ranks candidates across units; the evaluation half, which selection never reads, estimates the jump, its standard error, and the held-out gain (the fit-half step model’s evaluation-half mean squared error against the fit-half smooth-only model). Because selection and estimation use different rows, the winner’s evaluation-half z keeps its null calibration no matter how many candidates were searched: under no jump anywhere, |z| >= 2 fires on about 4.6% of unit winners. Reporting applies stricter bars, because the screen runs once per numeric target: |z| >= 4 and a held-out gain of at least 0.5% of the target’s evaluation-half variance.

Typing: a jump is scale-invariant¶

A steep smooth transition (a logistic crossing, a kink) also produces a large step coefficient against a quadratic background. The screen therefore types each flagged candidate with a local test on the evaluation half: the level gap between the nearest points on each side of the boundary at two scales, the nearest 15 and the nearest 60 points per side. A true jump shows the same gap at both scales (ratio near 1); a smooth transition’s gap grows with scale (ratio around 2 or more), because the wider window spans more of the slope. A candidate is reported as a jump only when the small-scale gap is significant (type_gap_t > 3) and the ratio stays below 1.5 (type_ratio). The evidence also carries type_z: the step’s t-statistic when the background is augmented with quadratic and cubic terms along the projection. The resolution limit is stated plainly: a smooth transition narrower than the small scale’s span is indistinguishable from a jump, and is typed as one.

The demonstration below plants three structures in one frame: an axis step, a diagonal step that no single column expresses, and a steep logistic transition that must not be reported.

import numpy as np
import polars as pl

import onceover

rng = np.random.default_rng(0)
n = 1200
x = rng.uniform(0, 1, n)
s0 = rng.uniform(0, 1, n)
s1 = rng.uniform(0, 1, n)
u = rng.uniform(0, 1, n)

# axis step: +2.0 where x crosses 0.6, on a linear trend
y_axis = 2.0 * (x > 0.6) + 1.5 * x + 0.3 * rng.normal(size=n)
# diagonal step: +2.0 across s1 = 0.45 + 0.35 * s0
y_diag = 2.0 * (s1 > 0.45 + 0.35 * s0) + 1.5 * s0 + 0.5 * rng.normal(size=n)
# steep but smooth: logistic transition in a driver that is not
# itself a column of the frame, width 0.08
y_smooth = 2.0 / (1.0 + np.exp(-(u - 0.5) / 0.08)) + 0.3 * rng.normal(size=n)

df = pl.DataFrame(
    {"x": x, "s0": s0, "s1": s1,
     "y_axis": y_axis, "y_diag": y_diag, "y_smooth": y_smooth}
)
report = onceover.profile(df, budget="standard", detectors=["discontinuities"])
steps = [f for f in report.findings if f["type"] == "step_discontinuity"]
for f in steps:
    print(f["claim"])
print(f"{len(steps)} step findings")

s1 steps by 0.453839 across a diagonal boundary in s0 and y_diag (direction 112 degrees, standardized threshold 0.198008): evaluation-half jump 0.453839 (se 0.031079, z 14.602823), held-out gain 0.010642; the local gap is scale-invariant (ratio 0.93), typing it as a jump rather than a steep smooth transition
y_diag steps by 1.620865 across a diagonal boundary in s0 and s1 (direction 112 degrees, standardized threshold 0.420663): evaluation-half jump 1.620865 (se 0.089635, z 18.082974), held-out gain 0.189325; the local gap is scale-invariant (ratio 1.23), typing it as a jump rather than a steep smooth transition
s0 steps by -0.554768 across a diagonal boundary in s1 and y_diag (direction 45 degrees, standardized threshold 0.195177): evaluation-half jump -0.554768 (se 0.04552, z -12.187234), held-out gain 0.015755; the local gap is scale-invariant (ratio 0.90), typing it as a jump rather than a steep smooth transition
y_axis steps by 1.773857 where x crosses 0.588449: evaluation-half jump 1.773857 (se 0.061643, z 28.77615), held-out gain 0.175928; the local gap is scale-invariant (ratio 0.85), typing it as a jump rather than a steep smooth transition
s0 steps by -0.323716 across a diagonal boundary in y_axis and y_diag (direction 79 degrees, standardized threshold 0.233278): evaluation-half jump -0.323716 (se 0.040681, z -7.957443), held-out gain 0.007863; the local gap is scale-invariant (ratio 0.69), typing it as a jump rather than a steep smooth transition
5 step findings

The axis step is recovered with its threshold in the axis column’s original units; the diagonal step names both columns and the boundary direction in degrees, with the threshold in standardized units. The logistic column produces no finding: its best candidate flags on the evaluation half, but the two-scale gap grows with scale, so it is typed as a smooth transition and counted in the coverage string instead of reported.

The screen also reports mirror readings: s1 steps across a boundary drawn in (s0, y_diag), and s0 across boundaries involving y_diag. These are the same planted jump read from the other side — a step finding is a conditional level difference, and a column that records which side of a boundary each row sits on (here y_diag, which was built from the boundary) shows the boundary again when it serves as an axis for another target. Read a step finding as “the target’s level differs across this boundary”, not as evidence about which column causes the jump.

What the pass does not report¶

A jump whose target relationship is only visible against a residual. The screen scans raw columns. Screening the residuals of a fitted smooth model — the recommended second pass in the originating implementation — is not part of this detector.
Steps in a categorical direction. Axes are numeric columns; a level shift between the levels of a categorical column is the conditional-rules and concentration passes’ territory.
More than one jump per (target, axis set). Each unit reports its single best boundary. A target with two genuine steps along the same axis produces one finding, for the stronger jump.
Flagged candidates below the reporting bars. Candidates with 2 <= |z| < 4, or a held-out gain under 0.5% of the target’s variance, are counted in coverage but not reported.