Large and wide tables

Column summaries and integrity checks examine every row. Relationship discovery screens broadly, then limits follow-up work by budget. This lets the profiler handle tables with millions of rows and hundreds of columns, while the amount of relationship evidence depends on the configured budget.

What stays exact¶

Pass	At any size
Column cards	every row: exact cardinality, null counts, statistics, quantiles, top values, concentration
Duplicate rows	full-row hash plus exact verification
Single-column keys	exact distinct counts over the full column
Sortedness	exact scan of adjacent values
Dates	exact calendar scan of distinct days
Arithmetic identities	fitted on a sample, residuals verified on every pair-complete row
Functional dependencies	screened broadly; reported candidates verified over up to 200,000 rows (`standard`) or 500,000 (`deep`) — full data when the table is smaller

Dense and nullable Int64/Float64 columns borrow Arrow buffers directly; other dtypes can require a normalized copy. This is an in-memory profiler, so input size and exact statistics still drive memory and runtime.

Bounded relationship work¶

Screening is broad and cheap; follow-up is bounded. The budget table in Coverage, budgets, and deadlines gives the per-tier numbers: at standard, the encoded relationship sample is 20,000 rows over all usable columns, each target is screened against all columns and fitted against at most 48 shortlisted predictors over 5,000 rows, all numeric pairs are screened and the 496 strongest get rank statistics, and FD candidates are verified over 200,000 rows. deep raises each of those. Coverage reports the limits used for each run.

Here is a wide synthetic table with relationships planted in the last columns — deliberately at the far end of the column order, because a screener that only looked at the first few dozen columns would miss them:

import numpy as np
import polars as pl

import onceover

rng = np.random.default_rng(20241024)
n, width = 100_000, 60

data = {"id": np.arange(n, dtype=np.int64)}
data.update({f"noise_{i:02}": rng.random(n) for i in range(width - 7)})
x = rng.uniform(-1, 1, n)
codes = rng.integers(0, 4, n, dtype=np.int64)
data.update({
    "late_x": x,
    "late_affine": 3 * x + 2,       # arithmetic identity with late_x
    "late_curve": x * x,            # nonlinear response
    "late_code": codes,             # code -> label dependency
})

df = pl.DataFrame(data).with_columns(
    pl.col("late_code").cast(pl.String).alias("late_label")
)
print(df.shape)

report = onceover.profile(df)
print(report.overview["headline"])

(100000, 59)

100000 rows × 59 columns (standard budget); key candidate: id; 3 functional dependencies, 6 predictive dependencies, 0 conditional rules

The report recovers the planted affine dependency, nonlinear response, and code→label lookup in the final columns, and ranks the integer id as the key candidate. Screening considers every usable column before shortlisting.

The coverage block records the work performed:

from great_tables import GT, loc, style

rows = [{"pass": p, "status": i["status"], "detail": i["detail"]}
        for p, i in report.coverage.items()]

(GT(pl.DataFrame(rows))
 .tab_header(title=f"Coverage on {df.height:,} rows × {df.width} columns (standard)",
             subtitle="Note the caps: the detectors screened everything, then bounded the deep work")
 .tab_options(table_width="100%", table_layout="fixed",
              table_font_size="14px", data_row_padding="9px")
 .tab_style(style.css("white-space: normal; overflow-wrap: anywhere; line-height: 1.5;"),
            loc.body()))

The scale benchmark¶

The reproducible scale benchmark is the pre-publish gate for this contract. It generates a fixed-seed table — independent noise, an integer primary key, a key with one repeated value, and relationships planted in the final columns: an affine identity, a nonlinear response, and a category-to-label dependency — then fails unless every planted structure is recovered and the exact cardinality checks hold:

pip install 'onceover[benchmark]'
python benchmarks/scale.py --rows 2000000 --columns 200 --threads 4 \
  --output /tmp/onceover-scale.json

The recorded run — 2,000,000 rows × 200 columns on macOS arm64, 24 GiB system memory, four Rayon threads:

Recorded measurement	Value
Profiling time	5.68 s
Peak process RSS during profiling	3.22 GiB
Increase from pre-profile RSS	0.15 GiB

All eight checks passed, including recovery of the late-column relationships, and no independent noise target was flagged as predictable. Peak RSS includes the input DataFrame and the Python process itself; the RSS increase measures the change from the pre-profile process baseline. These are reproducible synthetic checks, not guarantees for every data distribution — see the benchmark notes for methodology and the recorded JSON.

What bounded search can miss¶

Samples can miss rare patterns: a relationship visible in 0.01% of rows may not appear in a 20,000-row sample. Shortlisting can miss weak signals and interaction-only signals: a pair of columns that predict a target only together may not survive per-column screening. Capped searches stop early: beyond 496 pairs (standard), the profiler still screens every numeric pair but computes detailed statistics only for the strongest.

Coverage records sample sizes and search caps, and each finding includes its validation scope. A run that reports no relationship may still have missed one. Increase the budget (deep, sample_rows, max_pairs) or test a specific hypothesis directly.

Tuning for a big table¶

The overrides from Coverage, budgets, and deadlines apply directly:

# wider sampling when the standard caps feel tight
report = onceover.profile(df, sample_rows=50_000, max_predictors=96,
                          max_pairs=2_000)

# or a wall-clock budget; coverage lists passes skipped after the deadline
report = onceover.profile(df, deadline_seconds=30.0)

Sample and pair overrides change the amount of relationship work. A deadline can skip later passes. Neither reduces the rows examined by full-data passes that complete.