The predictability screen

The predictability screen tests how well other columns predict each eligible target. It uses a small gradient boosting model, which can detect interactions and nonlinear patterns that pairwise correlation can miss. A dependencies/predictive finding reports the inputs used, the target, a holdout score, and a majority baseline for categorical targets.

The encoded matrix¶

All model passes share one encoded matrix, built from a deterministic stratified sample of at most 20,000 rows (deep: 50,000). Each usable column is encoded as follows:

numeric columns and date/datetime columns (as day numbers) into at most 32 quantile bins, with the real bin edges retained so rules can quote actual thresholds;
strings with at most 32 distinct levels into nominal codes (strings with more levels are skipped, and the skip is counted in coverage);
booleans into two levels.

All usable columns are encoded. Each target model uses at most 48 predictors (64 on deep), selected as described below.

The matrix is built once per profile; the predictability screen, the decision-table pass, PRISM, and the association miner all read it. The bin edges and level labels travel with the matrix, so every finding can name conditions in original units.

The model¶

The screener uses histogram gradient boosting with depth-3 trees, shrinkage 0.3, L2 regularization λ = 1, at most 30 rounds with early stopping (patience 5) on a deterministic holdout. Bins are packed row-major, so each round builds histograms in a single pass over contiguous memory.

For each target: all present rows are collected (at least 300 required), shuffled with a per-target seeded generator, and split 75/25 into train and holdout; fitting uses at most 5,000 rows (10,000 on deep). Predictors are shortlisted by normalized mutual information between the target and every other feature over up to 2,048 training rows, truncated to the feature cap; the target’s own column never enters. Minimum leaf size scales with the training rows (n/200, clamped to [10, 1,000]).

What it reports¶

Numeric targets: holdout R².
Categorical targets (≤ 24 levels, or ≤ 12 distinct for a numeric target): holdout accuracy against the majority baseline.
inputs: every predictor actually used by the fitted model — the complete list.
Top features by gain: explanatory metadata, separate from the inputs, because gain ranking alone under-reports correlated contributors.

A finding is emitted when the holdout score clears the target-type threshold. The evidence carries n_train, n_holdout, the score, the baseline, top_features, and deterministic provenance. Validation is a diagnostic_holdout that participated in early stopping — it is marked used_for_selection, and it is not an independent confirmation sample.

import numpy as np
import polars as pl

import onceover

rng = np.random.default_rng(21)
n = 6_000
x1 = rng.uniform(-1, 1, n)
x2 = rng.uniform(-1, 1, n)
y = x1 * x2 + rng.normal(0, 0.05, n)      # interaction-only signal

df = pl.DataFrame({
    "x1": x1,
    "x2": x2,
    "y": y,
} | {f"noise_{i:02}": rng.random(n) for i in range(6)})

report = onceover.profile(df)
pred = [f for f in report.findings
        if f["category"] == "dependencies" and f.get("type") == "predictive"
        and f["subject"].get("target") == "y"]
print("predictive findings for y:", len(pred))
screen = [f for f in pred
          if any(e["method"]["detector"] == "gbm" for e in f["evidence"])]
for f in screen:
    e = next(e for e in f["evidence"] if e["method"]["detector"] == "gbm")
    print(" claim:", f["claim"])
    print(" inputs:", e["metrics"]["inputs"])
    print(" top features:", e["metrics"]["top_features"])

predictive findings for y: 3
 claim: y is predictable from [x1, x2, noise_00, noise_01, noise_02, noise_03, noise_04, noise_05] (holdout R² = 0.91)
 inputs: ['x1', 'x2', 'noise_00', 'noise_01', 'noise_02', 'noise_03', 'noise_04', 'noise_05']
 top features: ['x1', 'x2', 'noise_00']

The example’s target depends on x1 · x2. Pairwise correlation with either input is weak, while the boosted model can split on both features to predict the target.

Two detectors write to dependencies/predictive for this target: the screen and the piecewise regime model. The loop selects the screen’s evidence by detector before printing gain, because “top features” is a property of the fitted model rather than of the statement. Note that the screen’s input list includes all six noise columns: a depth-3 model with 30 rounds can touch many predictors, and the evidence reports what it used in its input list.

What the screen cannot do¶

A per-feature shortlist can drop a signal that only exists jointly (two individually uninformative columns may both be excluded on a wide table). Early stopping on a small holdout can miss weak structure. A high R² measures prediction quality on the holdout; it does not establish an exact mapping. The functional-dependency pass checks candidate mappings. The screen’s caps (targets, features, rows) are disclosed in predictability_screen coverage on every run.