The views detector groups columns that can share a clustering of rows.
Each group of columns is called a view. For example, temperature and
pressure may vary together across operating conditions, while humidity
requires a separate grouping.
This structure is inspired by CrossCat (Mansinghka et al., 2016). CrossCat uses Bayesian inference over column and row partitions. Onceover uses a deterministic greedy search with fixed limits and returns one partition. Its scores compare candidate fits; they are not probabilities that a relationship is true.
The detector is experimental and disabled by default.
Select the detector¶
Request detectors=["views"] to run the pass at the standard or deep
budget. The quick budget skips it. available_detectors() lists all
registered detectors, including this opt-in pass.
import onceover
ids = [d["id"] for d in onceover.available_detectors()]
print("views" in ids)
True
Example: two columns with shared states¶
Temperature and pressure each have two fixed operating settings. The settings change together, so both columns define the same two groups of rows. Humidity is generated independently. The assertions check that the first two columns share a view and humidity does not join it.
The pass does not use row order. We arrange the groups consecutively to make the fixture easy to read; it would still be a clustering example if the rows were shuffled.
import random
import polars as pl
import onceover
rng = random.Random(3)
temperature = [20.0] * 400 + [32.0] * 400
pressure = [1010.0] * 400 + [990.0] * 400
humidity = [50.0 + rng.gauss(0, 10) for _ in range(800)]
df = pl.DataFrame(
{"temperature": temperature, "pressure": pressure, "humidity": humidity}
)
report = onceover.profile(df, budget="standard", detectors=["views"])
for finding in report.findings:
print(finding["category"], finding["subject"]["columns"], "::", finding["claim"])
shared = [set(f["subject"]["columns"]) for f in report.findings
if f["category"] == "associations"]
assert {"temperature", "pressure"} in shared
assert all("humidity" not in columns for columns in shared)
changes ['humidity'] :: Rows split into 8 clusters; headline column humidity shifts by 2.63 between clusters (Greedy deterministic partition; scores are marginal-likelihood values, not posterior probabilities.).
changes ['pressure', 'temperature'] :: Rows split into 2 clusters; headline column temperature shifts by 1.00 between clusters (Greedy deterministic partition; scores are marginal-likelihood values, not posterior probabilities.).
concentration ['pressure'] :: 400 of 400 rows in cluster 0 (rows 0-399) hold pressure at 1010.0 (100% of the cluster); 400 of 400 rows in cluster 1 (rows 400-799) hold pressure at 990.0 (100% of the cluster) (Greedy deterministic partition; scores are marginal-likelihood values, not posterior probabilities.).
concentration ['temperature'] :: 400 of 400 rows in cluster 0 (rows 0-399) hold temperature at 20.0 (100% of the cluster); 400 of 400 rows in cluster 1 (rows 400-799) hold temperature at 32.0 (100% of the cluster) (Greedy deterministic partition; scores are marginal-likelihood values, not posterior probabilities.).
associations ['pressure', 'temperature'] :: Columns temperature, pressure form a dependence block under the greedy view partition (merge gain 40.1 nats over separate views on the search sample; Greedy deterministic partition; scores are marginal-likelihood values, not posterior probabilities.).
Read the findings¶
The pass uses these report categories:
| Category and type | Meaning |
|---|---|
associations/statistical | Several columns share a fitted row partition. The merge gain compares a shared view with separate views on the search sample. |
changes/residual_level | The fitted clusters differ in their means or categorical distributions. Despite the category name, the pass has not tested for a change over time. |
concentration/modal_share | One modeled numeric value or categorical level occupies at least 90% of a qualifying cluster’s observed values. |
concentration/spike_cells | Cells have low predictive density relative to the other scored cells in their column. |
A shared view can score better because it uses fewer clusters than separate views. The merge gain includes that complexity penalty. It is not a general measure of dependence: some related categorical columns do not merge.
The output also includes a cluster-difference finding for humidity, even though that column was generated independently. This is a difference between fitted groups of values, not evidence of a separate operating state.
Example: extreme values¶
The fixture contains three values of 999.9 among observations near 20.
Inspect the spike_cells evidence for the row indices and scores. The pass
can also flag unplanted observations, so the number of reported cells need
not equal the number of injected spikes.
rng = random.Random(5)
series = [20.0 + rng.gauss(0, 1) for _ in range(200)]
for i in (30, 100, 170):
series[i] = 999.9
df = pl.DataFrame({"temperature": series})
report = onceover.profile(df, budget="standard", detectors=["views"])
for finding in report.findings:
print(finding["category"], finding["subject"]["columns"], "::", finding["claim"])
flagged_rows = {cell["row"] for f in report.findings if f.get("type") == "spike_cells"
for e in f["evidence"] for cell in e["metrics"].get("cells", [])}
planted_rows = {30, 100, 170}
assert planted_rows <= flagged_rows
print(f"Planted extreme values found: {len(flagged_rows & planted_rows)}/{len(planted_rows)}")
print(f"Other reported cells: {len(flagged_rows - planted_rows)}")
changes ['temperature'] :: Rows split into 6 clusters; headline column temperature shifts by 632.61 between clusters (Greedy deterministic partition; scores are marginal-likelihood values, not posterior probabilities.).
concentration ['temperature'] :: 13 cells in temperature have log predictive density below -9.9 nats under their cluster (lowest 999.9 at row 30; Greedy deterministic partition; scores are marginal-likelihood values, not posterior probabilities.).
Planted extreme values found: 3/3
Other reported cells: 10
Data and work limits¶
The pass samples up to 20,000 rows at standard or 50,000 at deep;
sample_rows overrides that limit. It searches a further subsample of at
most 2,000 rows, then assigns and scores rows from the larger sample.
At most 64 eligible columns are modeled. When more columns are eligible, a pre-screen ranks each column by its strongest pairwise dependence (normalized mutual information on the search sample) and keeps the top 64; a column with more distinct values than half its rows - an identifier - gets no ranking signal. The remaining names are recorded as dropped. Each view has at most eight clusters, and the search allows up to 256 merge attempts and 300 million counted cell visits.
Numeric columns, dates, and datetimes are modeled as continuous values after median/IQR scaling. Strings and booleans use categorical models; levels beyond the 32 most frequent are pooled. Nulls contribute no score. See the method reference for formulas, assignment rules, and all configuration fields.
Interpretation¶
Cluster row ranges give the smallest and largest observed indices. Rows within those ranges may belong to other clusters; the ranges are not continuous segments.
The 10-nat spike threshold is a density comparison, not a calibrated probability of error. It can flag valid rare observations.
Missingness patterns, calendar structure, and relationships within a cluster are outside the model. Pooling categories also hides differences among rare levels.
Samples, the column cap, and the greedy search can miss relationships. A singleton view does not establish independence.
Checks¶
The implementation tests the component formulas, deterministic output, aligned distributions, independent binary columns, constant columns, category pooling, and the separation of modal and spike findings.
The method reference includes a reproducible sensor benchmark and the remaining evaluation questions.