Case study: distinguishing a level shift from a trend

Support says churn “went up in June.” The churn table is a weekly snapshot with tenure, a churn score, and the snapshot date. We want to check whether a model based on tenure accounts for the increase, or whether its residuals show a level shift. The changes/residual_level finding estimates one such shift under a stated reference model.

The fixture adds 0.15 to the score from a fixed date and omits one week from the snapshot calendar. The analysis checks whether the score change looks like a gradual trend or a shift in the model residuals.

import datetime as dt
import random

import polars as pl

import onceover

rng = random.Random(88)

base = dt.date(2024, 1, 8)
missing_week = dt.date(2024, 5, 20)                  # one snapshot week skipped
snapshot_date = []
for w in range(30):
    d = base + dt.timedelta(weeks=w)
    if d == missing_week:
        continue
    snapshot_date.extend([d] * 200)       # 200 rows per week
n = len(snapshot_date)

tenure = [rng.randrange(1, 60) for _ in range(n)]
score = [round(0.3 + 0.01 * t + rng.uniform(-0.05, 0.05), 4) for t in tenure]
# Level shift from 2024-06-03: the score gains +0.15 everywhere after that date.
cutoff = snapshot_date.index(dt.date(2024, 6, 3))
churn_score = [
    round(score[i] + (0.15 if i >= cutoff else 0.0) + rng.uniform(-0.02, 0.02), 4)
    for i in range(n)
]

df = pl.DataFrame({
    "snapshot_date": snapshot_date,
    "tenure": tenure,
    "churn_score": churn_score,
})

report = onceover.profile(df)
print(report.overview["headline"])

5800 rows × 3 columns (standard budget); 0 functional dependencies, 2 predictive dependencies, 0 conditional rules

First, check the snapshot dates and their cadence:

for f in report.findings:
    if f["category"] in {"gaps", "intervals", "ordering"}:
        print(f"[{f['category']:9s}] {f['claim']}")

[gaps     ] column snapshot_date is missing 1 week at 2024-05-13..2024-05-27 (14 days elapsed)
[ordering ] column snapshot_date is non-decreasing with row order

The dates are non-decreasing and have a modal cadence of seven days. One expected week is missing. The residual-shift result should be read with that calendar gap in mind.

changes = [f for f in report.findings
            if f["category"] == "changes" and f["subject"].get("target") == "churn_score"]
print("residual-shift findings for churn_score:", len(changes))
for f in changes:
    m = f["evidence"][0]["metrics"]
    print(f["claim"])
    print()
    print("   breakpoint:", m["breakpoint"])
    print("   residual mean before/after:",
          round(m["residual_mean_before"], 4), "/", round(m["residual_mean_after"], 4))
    print("   holdout error reduction:", f"{100 * m['holdout_error_reduction']:.1f}%")
    print("   reference model: churn_score ~ tenure +",
          m["reference_model"]["coefficients"])

residual-shift findings for churn_score: 1
churn_score shifts after 2024-05-27 along snapshot_date: a training-selected residual correction reduces holdout squared error by 83.9%

   breakpoint: 2024-05-27
   residual mean before/after: -0.0464 / 0.1024
   holdout error reduction: 83.9%
   reference model: churn_score ~ tenure + {'tenure': 0.009948939540210531}

Reading the finding¶

The reference model is a linear fit of churn_score on tenure; its coefficients are recorded in the finding. The residuals, ordered by snapshot date, show one step. The centered cumulative-sum scan places the breakpoint at 2024-05-27, within the missing week. The estimated offset applies from the 2024-06-03 block onward. A fixed before/after offset correction reduces holdout squared error by 83.9%. Under this model, the June increase is a residual level shift rather than a trend.

with pl.Config(tbl_rows=40, tbl_width_chars=90):
    monthly = df.group_by(
        pl.col("snapshot_date").dt.strftime("%Y-%m").alias("month")
    ).agg(pl.col("churn_score").mean().round(4).alias("mean_score")).sort("month")
    print(monthly)

┌─────────┬────────────┐
│ month   ┆ mean_score │
╞═════════╪════════════╡
│ 2024-01 ┆ 0.5976     │
│ 2024-02 ┆ 0.6095     │
│ 2024-03 ┆ 0.6084     │
│ 2024-04 ┆ 0.6057     │
│ 2024-05 ┆ 0.5923     │
│ 2024-06 ┆ 0.7516     │
│ 2024-07 ┆ 0.7428     │
└─────────┴────────────┘

The monthly means show a step around June. The finding supplies an estimated breakpoint, offsets, and the reference model used to compute residuals.

What the finding does not claim¶

No p-value is attached: this is a measured level change under a stated model, on disclosed train/holdout rows, from a pass that tests one breakpoint on the first date column. Multiple breaks, gradual ramps, and breaks driven by a column other than the first temporal one are out of scope today (Directions). The calendar finding also reports a missing week beginning May 20. Check the source that supplies the snapshot dates before interpreting the breakpoint.

For a reported change, the profile can test for a residual level shift under the specified model. The result provides a breakpoint and effect estimate for follow-up; it does not identify the cause.