Crate architecture and the pipeline

onceover is a Cargo workspace of six crates with one direction of dependencies, plus a thin Python package bound by pyo3 and built with maturin. Every profiling algorithm is implemented in Rust; Python provides the API.

onceover-core      typed column extraction from polars, exact value
                   counting, column cards, duplicates, date utilities
onceover-deps      keys and exact relationships: FD/AFD discovery,
                   unique column combinations, sortedness
onceover-mines     the model and statistical passes: encoding,
                   predictability screen (histogram GBM), decision
                   tables, PRISM, dependence measures, arithmetic
                   identities, local linear regimes, nominal
                   associations, date/sequence intelligence
onceover-report    finding types, evidence, consolidation and selection,
                   report serialization (no polars dependency)
onceover           the pipeline: budget tiers, detector registry,
                   orchestration, coverage
onceover-py        the pyo3 bindings: profile(df) and the report object

onceover-report depends on neither Polars nor the mining crates. It defines the finding schema and serialization; the pipeline and detectors depend on that schema. A finding’s category describes the property found; its evidence.method.detector identifies the detector that found it.

The pipeline¶

The profile pipeline runs these passes in order:

Extraction — the polars DataFrame becomes a TypedTable of compact typed columns (zero-copy where the dtype allows it).
Column cards — exact per-column statistics, roles, concentration, format patterns; the card findings are emitted here.
Row duplicates — exact, with the logical shortcut when a proven key makes duplicates impossible.
Keys — single-column uniqueness from the cards’ exact counts, composite-key screening and verification.
Functional dependencies — screen, lattice, verify; approximate dependencies and suppression.
Sortedness and dates — exact scans.
Encoding — the shared matrix for every model pass.
Predictability screen — every eligible column as a target.
Decision tables, PRISM, regimes — the model follow-ups, reading the screen’s shortlists and fit in that order.
Nominal associations — pairwise value implications over the encoded matrix.
Dependence and identities — pairwise statistics, then affine probes with full-row verification.
Rule evaluation — the shared predicate evaluator over every conditional-rule candidate.
Consolidation and selection — equivalent statements merge, report-wide limits apply, ids are hashed from the canonical statement.

Between passes a deadline check (when one is set) records skipped work in coverage — nothing is interrupted mid-computation.

The detector registry¶

Detectors are registered separately from the taxonomy: each declares an id, the categories it can produce, and the detectors it requires as resources. Selecting ["prism", "association_rules"] runs those detectors and their required resources. Resource detectors may run without adding their own findings. An empty selection still produces column cards and coverage.

import json

import onceover

for d in onceover.available_detectors():
    print(f"{d['id']:24s} -> {d['categories']}"
          + (f"   requires: {d['requires']}" if d["requires"] else ""))

column_profile           -> ['missingness', 'concentration', 'column_roles', 'formats']
row_duplicates           -> ['duplicates']
keys                     -> ['uniqueness']
pivot                    -> ['column_roles']
functional_dependencies  -> ['dependencies']
duplicate_encodings      -> ['column_roles']
sortedness               -> ['ordering']
granularity              -> ['formats']
sentinels                -> ['missingness']
digits                   -> ['formats']
dates                    -> ['gaps', 'intervals']
expectations             -> ['gaps', 'ordering']
gbm                      -> ['dependencies']
decision_tables          -> ['dependencies']   requires: ['gbm']
prism                    -> ['conditional_rules']   requires: ['gbm']
regimes                  -> ['dependencies', 'changes']   requires: ['gbm']
interactions             -> ['dependencies']   requires: ['gbm']
target_shapes            -> ['dependencies']
breaks                   -> ['changes']
surprises                -> ['intervals']
discontinuities          -> ['changes']
drift                    -> ['changes']   requires: ['gbm']
association_rules        -> ['conditional_rules']
hotspots                 -> ['concentration']
numeric_dependence       -> ['associations']
identities               -> ['dependencies']   requires: ['numeric_dependence']
period_associations      -> ['associations']   requires: ['numeric_dependence']
views                    -> ['associations', 'changes', 'concentration']

Rust extensions implement the Detector trait with a DetectorDescriptor and a discover(&DetectorContext) returning finding candidates. The pipeline validates their output categories, subjects, and statement definitions; invalid results are rejected and reported in coverage.

Budget tiers¶

TierConfig sets the search and sample limits for each budget. Coverage reports the work performed under those limits:

	`quick`	`standard`	`deep`
Exact passes (cards, duplicates, keys, sortedness, dates)	full	full	full
Composite keys	—	pairs, verify ≤ 16	pairs + triples, verify ≤ 24
FD: max LHS size / verification rows	—	2 / 200,000	3 / 500,000
Relationship sample rows / per-model predictor limit	—	20,000 / 48	50,000 / 64
GBM fitting rows / rounds	—	5,000 / 30	10,000 / 40
Decision-table / PRISM targets	—	6 / 6	10 / 10
Regimes: targets / rows / depth	—	4 / 5,000 / 3	8 / 10,000 / 4
Dependence pairs (capped)	—	496	2,016
Identity candidate pairs	—	100	200
Association findings	—	20	40

quick runs cards, duplicate checks, single-column key checks, sortedness, and date analysis; it skips relationship models. sample_rows, max_predictors, and max_pairs change the sampled work, while the full-row checks remain unchanged (Coverage, budgets, and deadlines).