Drift along a time axis

The drift detector answers a narrower question than the breaks pass: given a column the caller declares as the time axis, do the relationships behind a numeric target change between the earliest period and later ones?

It engages only when the time axis is declared, with hints={"time": "<column>"}. The rows are cut into contiguous periods of roughly equal row counts along that axis. For each numeric target, the predictability screen’s gradient-boosting model is fitted on the earliest period alone, with the time column withheld from the predictors — a model that splits on the time axis itself routes every later period to one side by construction, which would read as maximal drift on stable data. Each later period is then scored against the reference period by routing both periods’ rows through the fitted trees: at every split, the two periods’ left/right counts form a 2x2 table, and the drift score is the mean chi-square across splits, weighted by each split feature’s share of the model’s total gain, so splits on features the model barely uses contribute almost nothing. Under no drift the score averages near 1.

import random

import polars as pl

import onceover


def frame(shift: bool) -> pl.DataFrame:
    rng = random.Random(7)
    t, x, y = [], [], []
    for i in range(4000):
        xv = rng.uniform(-1, 1) + (2.0 if shift and i >= 2000 else 0.0)
        t.append(i)
        x.append(xv)
        y.append(xv + rng.uniform(-0.05, 0.05))
    return pl.DataFrame({"t": t, "x": x, "y": y})


report = onceover.profile(frame(True), hints={"time": "t"})
for f in report.findings:
    if f["specification"].get("form") == "drift":
        print(f["claim"])

the predictors of x drift over t: split-routing drift score reaches 343.0 in period 4 (2000 to 2499), against a no-drift scale of ~1
the predictors of y drift over t: split-routing drift score reaches 336.7 in period 4 (2000 to 2499), against a no-drift scale of ~1

The planted shift starts halfway through the series, at t = 2000. With eight periods of 500 rows, that is the boundary between period 3 and period 4. The evidence carries one entry per later period, so the onset is read directly from the table: scores sit near the no-drift scale of 1 before the shift and jump after it. concept_drift repeats the mean over only the splits that have a leaf child. Both sides of every comparison use the same row count (the smaller period’s), because the chi-square grows with the number of compared rows as well as with the size of the shift. A finding takes more than a large score. The maximum over periods must also beat a permutation null: the reference rows and every period’s compared rows are pooled and reassigned to a pseudo-reference group and pseudo-period groups of the original sizes, every period is rescored against the shared pseudo-reference, and the maximum is retaken — 16 replicates. The null prices the max-over-periods selection itself, including the fact that all periods share one reference (an atypical reference draw elevates every comparison at once, and the replicates reproduce that). The evidence carries the largest replicate maximum as null_max; a finding requires the real maximum to exceed it and to reach 5.0.

metrics = next(
    f["evidence"][0]["metrics"]
    for f in report.findings
    if f["specification"].get("form") == "drift"
    and f["subject"]["columns"] == ["t", "y"]
)
print("reference period:", metrics["reference_period"])
print("reference rows:", metrics["reference_rows"], " reference R2:", round(metrics["reference_r2"], 3))
for p in metrics["periods"]:
    print(p["period"], p["range"], "drift:", None if p["drift"] is None else round(p["drift"], 2),
          " concept:", None if p["concept_drift"] is None else round(p["concept_drift"], 2))

reference period: 0 to 499
reference rows: 500  reference R2: 0.994
1 500 to 999 drift: 1.03  concept: 1.24
2 1000 to 1499 drift: 0.72  concept: 0.73
3 1500 to 1999 drift: 0.79  concept: 0.51
4 2000 to 2499 drift: 336.71  concept: 316.32
5 2500 to 2999 drift: 336.71  concept: 316.32
6 3000 to 3499 drift: 336.71  concept: 316.32
7 3500 to 3999 drift: 336.71  concept: 316.32

Two controls. The same generator without the shift produces no drift finding, and the shifted frame without the time hint produces none either — the detector does not guess an axis.

def drift_count(rep):
    return sum(
        1 for f in rep.findings if f["specification"].get("form") == "drift"
    )


print("stable, with hint:", drift_count(onceover.profile(frame(False), hints={"time": "t"})))
print("shifted, no hint:", drift_count(onceover.profile(frame(True))))

stable, with hint: 0
shifted, no hint: 0

Limits¶

The score sees only the variables the reference-period model splits on. Drift confined to a variable the model does not use, or a change in the target’s own distribution with its predictors unchanged, does not register. The reference model must reach a holdout R2 of at least 0.2 on the reference period; a target with no learnable structure there is not scored. Periods with fewer than 100 compared rows record null scores rather than noisy ones.