Dependence measures

For selected numeric column pairs, the dependence screen calculates Pearson, Spearman, and Chatterjee’s ξ in both directions. It reports statistical associations; these measures alone do not show that one column determines another.

Three coefficients¶

Pearson — measures linear association.
Spearman — measures monotone association using average ranks for ties.
Chatterjee’s ξ (JASA 2021) — an asymmetric measure of dependence that can detect non-monotone patterns. The screen calculates both directions because ξ(x → y) and ξ(y → x) can differ.

When x has ties, the estimator randomizes their order using a fixed seed. Sorting ties by row order can introduce dependence on row position; for example, a block ID compared with a row counter can produce a high ξ on independent values. ξ is undefined if either column is constant or if fewer than 30 pairs are available.

Screening and caps¶

The pass scores all numeric pairs using normalized mutual information over encoded bins from up to 2,048 rows. It then calculates Pearson, Spearman, and ξ on up to 496 pairs (standard) or 2,016 (deep), using at most 20,000 or 50,000 sampled rows. Findings survive when the strongest signal reaches 0.5 (or |Pearson| is exactly 1), the top 20 are kept, and pairs with signal ≥ 0.98 become redundancy links that feed the connected-groups view. Each finding records the four coefficients, pair count, and sample scope.

import numpy as np
import polars as pl

import onceover

rng = np.random.default_rng(31)
n = 4_000
x = rng.uniform(-1, 1, n)
df = pl.DataFrame({
    "x": x,
    "linear": 3 * x + 1,            # Pearson ~ 1
    "monotone_bend": x ** 3,        # Spearman 1, Pearson < 1
    "parabola": x ** 2,             # xi high, Pearson ~ 0
})

report = onceover.profile(df)
for f in report.findings:
    if f["category"] == "associations":
        m = f["evidence"][0]["metrics"]
        a, b = m["variables"]
        print(f"{a:>14s} × {b:<14s} pearson={m['pearson']:+.3f} "
              f"spearman={m['spearman']:+.3f} xi(a→b)={m['xi_ab']:.3f} xi(b→a)={m['xi_ba']:.3f}")

        linear × monotone_bend  pearson=+0.917 spearman=+1.000 xi(a→b)=0.999 xi(b→a)=0.999
             x × linear         pearson=+1.000 spearman=+1.000 xi(a→b)=0.999 xi(b→a)=0.999
             x × monotone_bend  pearson=+0.917 spearman=+1.000 xi(a→b)=0.999 xi(b→a)=0.999
        linear × parabola       pearson=-0.002 spearman=+0.009 xi(a→b)=0.999 xi(b→a)=0.259
 monotone_bend × parabola       pearson=-0.002 spearman=+0.009 xi(a→b)=0.999 xi(b→a)=0.259
             x × parabola       pearson=-0.002 spearman=+0.009 xi(a→b)=0.999 xi(b→a)=0.259

In the controlled examples, Pearson is highest for a linear pattern. Spearman remains high for a monotone curve. For the non-monotone relationship y = x², the two ξ directions differ: x determines y, but y does not determine x.

Interpretation¶

Association coefficients describe patterns in the evaluated pairs. They do not establish a functional mapping or a causal effect. A low ξ on a sample means no strong ξ signal was detected at that sample size; coverage records the scope (Coverage, budgets, and deadlines).

Shape against a designated target¶

The measures above are response-free. When you already know which column you intend to model, profile(df, target="y") adds the target_shapes pass: for each numeric predictor it computes linear R², a binned-curve R² (up to 20 equal-count bins, adjusted for the bins’ degrees of freedom), and the best single-step R², all on the rows where predictor and target are both present. It reports a predictor only when a shape is decisive — a single step carries the relationship, or the curve adds materially over the line. Every other pass behaves exactly as in a profile without a target.

rng = np.random.default_rng(11)
n = 1_200
dose = rng.uniform(0, 10, n)
dose_response = np.where(dose < 4.0, 1.0, 3.0) + 0.1 * rng.normal(size=n)
df2 = pl.DataFrame({"dose": dose, "response": dose_response})

report2 = onceover.profile(df2, target="response")
for f in report2.findings:
    if f["specification"].get("form") in ("step_shape", "curved_shape"):
        m = f["evidence"][0]["metrics"]
        print(f"{f['subject']['inputs']} -> response: form={f['specification']['form']} "
              f"step_at={m.get('step_at')} step_r2={m['step_r2']:.2f} linear_r2={m['linear_r2']:.2f}")

level = rng.uniform(-1, 1, n)
curved_response = level ** 3 + 0.05 * rng.normal(size=n)
df3 = pl.DataFrame({"level": level, "response": curved_response})

report3 = onceover.profile(df3, target="response")
for f in report3.findings:
    if f["specification"].get("form") in ("step_shape", "curved_shape"):
        m = f["evidence"][0]["metrics"]
        print(f"{f['subject']['inputs']} -> response: form={f['specification']['form']} "
              f"curve_r2={m['curve_r2']:.2f} linear_r2={m['linear_r2']:.2f} spearman={m['spearman']:.2f}")

['dose'] -> response: form=step_shape step_at=4.0013640035324025 step_r2=0.99 linear_r2=0.72
['level'] -> response: form=curved_shape curve_r2=0.97 linear_r2=0.83 spearman=0.95

The step finding records the located cut (step_at) — the same quantity the discontinuity screen reports for its jumps, so the two corroborate each other when both fire on a pair. A linear predictor and an unrelated predictor produce no shape findings: the dependence screen already reports their association, and the shape adds no interpretation. The pass’s coverage line names the designated target and how many predictors were evaluated, so a reader can tell the profile was target-aware.

Interactions: the H statistic¶

The coefficients above are model-free and pairwise: they say two columns are related. A different question is whether two columns act jointly on a third — whether an additive model in the pair leaves structure unmodelled. The interactions pass answers it for the numeric targets the predictability screen fits well. It refits the screen’s GBM for the target, evaluates partial dependences for the strongest predictor pairs, and computes the Friedman–Popescu H statistic: the share of the pair’s joint partial-dependence variance that the sum of the two single partial dependences does not explain.

A pair is reported only when the surrogate first beats an additive (depth-1) refit of the same target out of sample, and the pair itself clears three bars on h2, on the interaction variance’s separation from zero (interaction_z), and on the interaction variance as a share of the target’s variance. The example plants a product term; the generating pair is the first interaction finding for y.

rng = np.random.default_rng(3)
n = 1_500
x1 = rng.uniform(-1, 1, n)
x2 = rng.uniform(-1, 1, n)
x3 = rng.uniform(-1, 1, n)
y = 2.0 * x1 * x2 + 0.3 * x3 + 0.1 * rng.normal(size=n)
df2 = pl.DataFrame({"x1": x1, "x2": x2, "x3": x3, "y": y})

report2 = onceover.profile(df2)
for f in report2.findings:
    if f["specification"].get("form") == "interaction_h" and f["subject"].get("target") == "y":
        m = f["evidence"][0]["metrics"]
        print(f"{f['subject']['inputs']} -> y: h2={m['h2']:.3f} "
              f"interaction_variance={m['interaction_variance']:.3f} "
              f"z={m['interaction_z']:.1f} holdout_r2={m['holdout_r2']:.2f} "
              f"additive_holdout_r2={m['additive_holdout_r2']:.2f}")

['x1', 'x2'] -> y: h2=0.943 interaction_variance=0.352 z=7.4 holdout_r2=0.91 additive_holdout_r2=0.06

The same profile run also reports interactions for the reverse targets x1 and x2 — the inverse of a product is itself non-additive, so those readings are correct for the fitted functions. A purely additive target produces no interaction findings: its surrogate does not beat the additive refit by the required margin, so its pairs are never evaluated. The additive gate exists because a tree ensemble’s staircase fit of an additive surface is not additive as a function, and its partial dependences alone would report that artifact as an interaction.