Relationship discovery uses row samples and search caps to limit its cost. This chapter describes how rows are chosen, which passes use each sampler, and what the samples can miss. Coverage records the sample sizes. Column cards, keys, duplicates, sortedness, and identity residual checks use full scans or exact deductions.
The stratified sampler¶
sample_rows(n, cap) splits the row range into cap strata and chooses
one row from each. A fixed arithmetic hash of the stratum index determines
the offset within each stratum. The result is sorted and reproducible for
the same row count and cap.
The first and last strata both contribute rows, so the sample spans the table. Hashed offsets avoid the fixed spacing of a plain stride, which can repeatedly select the same part of periodic data, such as one weekday in a weekly export. They do not guarantee that every periodic pattern is represented. When the table fits under the cap, every row is used.
The example has 300,000 rows: an id, ten random columns, a target, and a
planted lookup pair (region → region_name). Its coverage block shows
the row counts used by the different passes.
import numpy as np
import polars as pl
import onceover
rng = np.random.default_rng(3)
n = 300_000
region = rng.integers(0, 5, n)
big = pl.DataFrame(
{"id": np.arange(n, dtype=np.int64)}
| {"region": region}
| {"region_name": [f"r{r}" for r in region]}
| {f"x{i:02}": rng.random(n) for i in range(10)}
| {"y": rng.random(n)}
)
report = onceover.profile(big)
for name in ["encoding", "predictability_screen", "functional_dependencies", "dependence_screen"]:
print(f"{name:24s}", report.coverage[name]["detail"])
print()encoding 20000 rows (20000 of 300000), 14 features (all usable columns; per-model predictor budget 48, 0 skipped/unusable)
predictability_screen 14 target columns considered over 20000 encoded rows / 14 features; 2 predictable; predictors shortlisted from all features on training rows (max 48 predictors / 5000 fitting rows per target)
functional_dependencies all 2 eligible lhs / 14 eligible rhs columns screened (26 directed pairs over 2048 rows); per-target lhs cap 16; levels <= 2; verification rows 200000 (stratified sample); 2 exact FDs, 0 AFDs; composite keys screen all eligible pairs, verify up to 16 candidates (skipped when a single-column key exists); deep triples use up to 12 shortlisted columns
dependence_screen 78 of 78 numeric pairs (full); all 78 pairs screened; detailed statistics over 20000 of 300000 rows; 0 strong, 0 redundancy links
The coverage detail pairs every sample with its population — “20000 rows (20000 of 300000)” — so a reader always knows which claims were measured where. On this frame the encoded matrix and the dependence statistics run over 20,000 stratified rows; the FD screen looks at 26 directed pairs over 2,048 rows, then verifies its two discovered candidates over a 200,000-row sample; the column cards and keys never sample at all.
Which pass uses which rows¶
| Pass | Rows used (standard / deep) | Mechanism |
|---|---|---|
| Column cards, duplicates, single keys, sortedness, dates | all rows | exact |
| FD broad screen | 2,048 | stratified sample of hashed values |
| FD verification | ≤ 200,000 / 500,000 | stratified sample, canonical value bytes |
| Composite-key screen / verification | 2,048 screen; ≤ 16 / 24 verified pairs, full data | hash screen, exact count |
| Encoded matrix | ≤ 20,000 / 50,000 | stratified |
| Predictability screen | ≤ 5,000 / 10,000 fitting rows | seeded shuffle, 75/25 holdout per target |
| Decision tables | encoded rows | 5-fold cross-validation |
| PRISM rules | encoded rows | bitset masks over the encoded sample |
| Rule evaluation | 1/5 of sampled rows | row-id hash partition |
| Association mining | 4/5 train, 1/5 holdout | row-id hash partition |
| Dependence statistics | ≤ 20,000 / 50,000 | stratified, pair-complete |
| Identity fitting | ≤ 20,000 | stratified |
| Identity residual check | all pair-complete rows | exact |
| Regimes | ≤ 5,000 / 10,000 | 1/5 row-id holdout |
Two partition mechanisms recur. The predictability screen shuffles
positions with a seeded generator and holds out 25% per target — the
holdout never trained that model. The rule evaluator, the association
miner, and the regimes pass use a stable hash of the original row id to
split rows; because the same hash defines evaluation rows across passes, a
rule’s evaluation partition is reproducible — but the mining passes may
already have seen those rows during discovery, which is why their
validation is labeled diagnostic_partition (or diagnostic_holdout) and
marked used_for_selection.
What samples can miss¶
A 20,000-row sample may miss a pattern present in 0.01% of rows, or include
too few examples to detect it. Per-feature shortlisting can drop signals
that appear only through interactions. The FD search stops at its
per-target determinant shortlist. Coverage records sample sizes, caps,
and shortlists. Overrides such as sample_rows, max_predictors,
max_pairs, and budget deep increase the corresponding search budgets.
A finding from a sample describes the evaluated rows. If a finding is missing, increase the budget or test the hypothesis directly. A bounded search cannot establish that two columns are independent.