Use onceover.profile(df) to inspect an orders table for candidate keys,
relationships between columns, and duplicate rows.
The fixture contains a primary key, a region lookup, a nearly-unique external reference with two repeated values, and a derived column. This known structure lets us check the report against the input.
import json
import random
import polars as pl
import onceover
rng = random.Random(7)
n = 2000
region_id = [rng.randrange(3) for _ in range(n)]
region_name = [["emea", "amer", "apac"][r] for r in region_id]
order_id = list(range(n))
external_ref = [f"R{i:05}" for i in range(n)]
external_ref[-1] = external_ref[0] # two duplicated values
external_ref[-2] = external_ref[1]
quantity = [rng.randrange(1, 11) for _ in range(n)]
double_quantity = [2 * q for q in quantity]
df = pl.DataFrame({
"order_id": order_id,
"external_ref": external_ref,
"region_id": region_id,
"region_name": region_name,
"quantity": quantity,
"double_quantity": double_quantity,
})
df.head(4)Profile it¶
One call runs the detectors enabled by the default budget. Large and wide tables describes runtime and coverage on larger tables.
report = onceover.profile(df)
print(report.overview["headline"])2000 rows × 6 columns (standard budget); key candidate: order_id; 5 functional dependencies, 4 predictive dependencies, 16 conditional rules
The headline reports a key candidate, five functional dependencies, predictable columns, and conditional rules for this fixture.
Inspect findings¶
report.findings is sorted by its priority score, interestingness:
for f in report.findings[:8]:
print(f"[{f['interestingness']:.2f}] {f['claim'][:95]}")[0.95] order_id uniquely identifies rows (primary key candidate)
[0.87] double_quantity = 2.000000·quantity within tolerance on all 2000 non-missing row pairs
[0.82] column external_ref is nearly unique with 2 repeated non-null values
[0.82] region_id is predictable from [region_name] (holdout accuracy 1.00 vs 0.36 majority baseline)
[0.82] When double_quantity <= 2, quantity = 1 holds in 35 of 35 matching diagnostic rows (100.00%; ba
[0.82] When quantity <= 1, double_quantity = 2 holds in 35 of 35 matching diagnostic rows (100.00%; ba
[0.82] When double_quantity > 10 and double_quantity <= 12, quantity = 6 holds in 50 of 50 matching di
[0.82] When double_quantity > 18 and double_quantity <= 20, quantity = 10 holds in 43 of 43 matching d
The report verifies mappings between quantity and double_quantity,
checks their affine formula, and finds the region lookup. It also reports
that order_id is unique while external_ref has two repeated values.
The following count shows the categories present in this report:
from collections import Counter
print(Counter((f["category"], f.get("type")) for f in report.findings))Counter({('conditional_rules', 'if_then'): 16, ('dependencies', 'functional'): 5, ('dependencies', 'predictive'): 4, ('column_roles', 'categorical_role'): 3, ('uniqueness', 'uniqueness'): 2, ('column_roles', 'duplicate_encoding'): 2, ('concentration', 'hotspot'): 2, ('column_roles', 'identifier_role'): 2, ('associations', 'statistical'): 1, ('ordering', 'column_ordering'): 1})
To inspect findings about one column, filter by subject.columns. For example, these findings refer to external_ref:
for f in report.findings:
if "external_ref" in f["subject"]["columns"]:
print(f"[{f['category']}] {f['claim']}")[uniqueness] column external_ref is nearly unique with 2 repeated non-null values
[column_roles] column external_ref looks like a unique identifier (1998 distinct of 2000 values)
external_ref looks like an identifier, but it contains two repeated
values. Joining on it as though it were unique can produce extra rows.
Inspect the uniqueness finding before choosing it as a key.
Inspect a complete finding¶
f = report.findings[0]
print(json.dumps(f, indent=1)[:1600]){
"id": "uniqueness-1ac6146d2c27c2e2",
"category": "uniqueness",
"type": "uniqueness",
"subject": {
"columns": [
"order_id"
],
"unit": "column_values"
},
"specification": {
"definition": {
"comparison": "canonical_equality"
}
},
"claim": "order_id uniquely identifies rows (primary key candidate)",
"evidence": [
{
"method": {
"detector": "keys",
"algorithm": "uniqueness",
"provenance": "exact uniqueness + folklore ranking (onceover v0.1)"
},
"basis": "measured",
"scope": {
"sampling": "full",
"population_rows": 2000,
"rows_evaluated": 2000,
"missing_values": "non-missing values counted; missingness reported separately"
},
"validation": {
"method": "direct_comparison",
"used_for_selection": false
},
"metrics": {
"distinct": 2000,
"n": 2000,
"plausibility_score": 1.0
}
}
],
"interestingness": 0.95,
"detector_interestingness": 0.95,
"interpretation": "use as join key; exclude from numeric features"
}
The result names its columns in subject. specification describes the
comparison, mapping, or formula, and evidence contains the evaluated row
counts and metrics. The findings chapter explains these
fields and shows how to filter on them.
identity = next(f for f in report.findings if "within tolerance" in f["claim"])
print(identity["claim"])
print("category:", identity["category"], "| type:", identity.get("type"))
print("form:", identity["specification"].get("form"))
ev = identity["evidence"][0]
print("basis:", ev["basis"], "| sampling:", ev["scope"]["sampling"])
print("validation:", ev["validation"]["method"])
print("metrics:", ev["metrics"])double_quantity = 2.000000·quantity within tolerance on all 2000 non-missing row pairs
category: dependencies | type: functional
form: affine
basis: measured | sampling: full
validation: direct_comparison
metrics: {'a': 2.0, 'absolute_tolerance': 1e-09, 'b': 0.0, 'exact_fraction': 1.0, 'matching_rows': 2000, 'n': 2000, 'relative_tolerance': 1e-09, 'tolerance': '1e-9 absolute + 1e-9 relative', 'violations': 0}
The identity check fitted y = a*x + b on a sample, then checked the
residuals on all 2,000 rows. Every row matched within the stated tolerance.
Inspect column summaries¶
report.columns contains counts, statistics, and suggested roles, keyed
by column name. For example:
print(json.dumps(report.columns["region_id"], indent=1)[:900])
print(json.dumps(report.columns["double_quantity"], indent=1)[:700]){
"name": "region_id",
"dtype": "int64",
"role": "categorical",
"n": 2000,
"null_count": 0,
"null_fraction": 0.0,
"distinct": 3,
"distinct_ratio": 0.0015,
"numeric_stats": {
"min": 0.0,
"max": 2.0,
"mean": 0.952,
"std": 0.816514543654919,
"q25": 0.0,
"q50": 1.0,
"q75": 2.0,
"zeros": 717,
"negatives": 0,
"integral": true
},
"concentration": {
"hhi": 0.3344935,
"entropy_nats": 1.0968765707316228,
"effective_cardinality": 2.994797362652193,
"top1_share": 0.3585,
"top3_share": 1.0
},
"top_values": [
{
"value": "0",
"count": 717,
"share": 0.3585
},
{
"value": "1",
"count": 662,
"share": 0.331
},
{
"value": "2",
"count": 621,
"share": 0.3105
}
],
"action_hints": [
"treat_as_categorical"
]
}
{
"name": "double_quantity",
"dtype": "int64",
"role": "categorical",
"n": 2000,
"null_count": 0,
"null_fraction": 0.0,
"distinct": 10,
"distinct_ratio": 0.005,
"numeric_stats": {
"min": 2.0,
"max": 20.0,
"mean": 10.922,
"std": 5.762110377283693,
"q25": 6.0,
"q50": 10.0,
"q75": 16.0,
"zeros": 0,
"negatives": 0,
"integral": true
},
"concentration": {
"hhi": 0.1002955,
"entropy_nats": 2.3011108360022776,
"effective_cardinality": 9.985268291912343,
"top1_share": 0.1085,
"top3_share": 0.32
},
"top_values": [
{
"value": "2",
"count": 217,
"share": 0.1085
},
{
"value": "12",
"count": 216,
"share": 0.108
},
{
"value": "20",
region_id has three distinct numeric values and receives a categorical
role with a treat_as_categorical suggestion. double_quantity includes
exact distribution summaries. These counts use every input row; see
Column summaries for the available fields.
Check which passes ran¶
report.coverage lists completed and skipped passes, along with their
sample sizes and limits:
for name, info in report.coverage.items():
print(f"{name:28s} {info['status']:14s} {info['detail'][:72]}")arithmetic_identities ran_exact 1 candidate pairs checked, 1 identities
association_rules ran_sampled 1 of 1 nominal features (all eligible columns, level cap 16); 1588 train
breaks ran_sampled no date, datetime, or year-like integer column; break scan skipped
column_cards ran_exact 6 cards (exact)
dates ran_exact skipped: no temporal columns
decision_tables ran_sampled 4 categorical targets searched, 1 tables found; 4 of 4 eligible targets
dependence_screen ran_exact 6 of 6 numeric pairs (full); all 6 pairs screened; detailed statistics o
digits ran_exact 4 numeric columns digit-scanned; 0 Benford-eligible (others fail the pos
discontinuities ran_exact 3 numeric targets screened over 6 axis units and 42 diagonal units (16 d
drift ran_exact no time axis declared
duplicate_encodings ran_exact 4 label-like columns, 6 pairs evaluated on 2000 rows; 2 duplicate encodi
encoding ran_sampled 2000 rows (2000 of 2000), 5 features (all usable columns; per-model pred
expectations skipped no hints provided
extraction ran_exact 2000 rows × 6 columns
functional_dependencies capped all 4 eligible lhs / 6 eligible rhs columns screened (20 directed pairs
granularity ran_exact 0 float columns scanned for granularity changes
hotspots ran_sampled 4 eligible columns, 6 combinations examined (0 triples), 287 cells exami
interactions ran_sampled 0 numeric targets above the holdout R² bar (0.1), 0 fitted, 0 clearing t
keys ran_exact 1 single-column unique, 0 composite
multiterm_identities ran_exact 16 multi-term candidates screened, 0 multi-term identities
period_associations not_applicable no date column or `year` column defines at least two periods of 30+ rows
pivot not_applicable needs exactly one continuous measure column and a categorical attribute
predictability_screen ran_sampled 5 target columns considered over 2000 encoded rows / 5 features; 4 predi
prism_rules ran_sampled 4 targets eligible, 16 rules; 4 of 4 eligible targets selected (cap 6)
regimes ran_sampled 0 of 0 eligible numeric targets (cap 4), 0 fitted over at most 5000 rows
row_duplicates ran_exact no exact duplicate rows (exact)
rule_evaluation ran_sampled 16 of 16 candidates passed a common diagnostic partition; support and ba
sentinels ran_exact 4 numeric columns scanned for sentinel values
sortedness ran_exact 1 columns sorted by row order
surprises ran_exact 3 numeric columns scanned against 4 conditioning columns; 0 surprise fin
target_shapes skipped detector not enabled
views skipped detector not enabled
Declare what you know¶
Inference has limits: a monthly cadence cannot be inferred in days,
and panel completeness cannot be tested without knowing the
dimensions. hints declares the intended structure; the
expectations pass checks it exactly and tags its evidence
reference_source="declared". Here March is missing from a
monthly series — five observations, below the inferred gap
check’s floor:
import datetime
monthly = pl.DataFrame({
"month": [datetime.date(2024, m, 1) for m in (1, 2, 4, 5, 6)],
"revenue": [10.0, 11.0, 13.0, 12.0, 14.0],
})
hinted = onceover.profile(monthly, hints={
"time": "month",
"no_gaps": ["month"],
"cadence": {"month": "monthly"},
})
for f in hinted.findings:
if f["category"] == "gaps":
print(f["claim"])
print(f["evidence"][0]["metrics"])column month is missing 1 month at 2024-02-01..2024-04-01 (60 days elapsed)
{'approximate': False, 'elapsed_days': 60, 'from': '2024-02-01', 'missing': 1, 'n': 5, 'reference': 'months', 'reference_source': 'declared', 'to': '2024-04-01'}
row_duplicates can rule out duplicate rows without comparing whole
rows because order_id has already been found unique and null-free.
functional_dependencies reports how many rows it checked. This fixture
fits within its verification cap; larger inputs may be sampled.
Save the report¶
Use save() to write a report. JSON output is repeatable for the same
package version, data, and settings when no deadline is set.
import os
import tempfile
path = report.save(tempfile.mkdtemp() + "/profile.json")
print("saved", os.path.getsize(path), "bytes to", path)
print(report.to_markdown()[:400])saved 68453 bytes to /tmp/tmpff3qnycr/profile.json
# onceover report
2000 rows × 6 columns (standard budget); key candidate: order_id; 5 functional dependencies, 4 predictive dependencies, 16 conditional rules
## Overview
- **rows × columns**: 2000 × 6
- **dtypes**: 4×int64, 2×string
- **roles**: 4×categorical, 1×free_text, 1×identifier
## Findings
- **[0.95·measured] uniqueness/uniqueness** — order_id uniquely identifies rows (primary key ca
save() writes JSON by default; report.data returns the parsed report.
to_markdown() produces a text summary suitable for sharing in an issue
or pull request.
Further examples¶
Reading findings: fields, scores, and filtering.
Column summaries: roles and distribution statistics.
Coverage: budgets, sampling, and detector selection.
Analysis notebook: a larger fixture with charts and checks against planted relationships.