This synthetic vendor extract contains two re-sent orders that repeat the internal key and create exact duplicate rows. It also has a nearly unique external reference, invoice numbers in two formats, and a missing business day.
import datetime as dt
import random
import polars as pl
import onceover
rng = random.Random(2024)
n = 4_000
order_id = list(range(n))
external_ref = [f"ORD-{i:06}" for i in range(n)]
for k in range(3): # three duplicated references
external_ref[n - 1 - k] = external_ref[k]
region_id = [rng.randrange(3) for _ in range(n)]
region_name = [["emea", "amer", "apac"][r] for r in region_id]
invoice_no = [f"INV-{i:05}" if i % 2 == 0 else f"{i:05}" for i in range(n)]
quantity = [rng.randrange(1, 11) for _ in range(n)]
unit_price = [round(rng.uniform(5, 25), 2) for _ in range(n)]
line_total = [round(q * p, 2) for q, p in zip(quantity, unit_price)]
base = dt.date(2024, 3, 4)
skip = {dt.date(2024, 4, 2), dt.date(2024, 4, 3)} # two weekdays missing
business, i = [], 0
while len(business) < n:
d = base + dt.timedelta(days=i // 20)
if d not in skip:
business.append(d)
i += 1
df = pl.DataFrame({
"order_id": order_id,
"external_ref": external_ref,
"region_id": region_id,
"region_name": region_name,
"invoice_no": invoice_no,
"quantity": quantity,
"unit_price": unit_price,
"line_total": line_total,
"business_date": business,
})
df = pl.concat([df, df[500:502]]) # two duplicated rows, on purpose
report = onceover.profile(df)
print(report.overview["headline"])4002 rows × 9 columns (standard budget); 3 functional dependencies, 9 predictive dependencies, 6 conditional rules
The headline reports no key candidate. The two repeated orders mean no single column is unique across all rows. Review the ranked findings to see which issues affect joins, date checks, and invoice-number parsing.
for f in report.findings:
kind = "/".join(p for p in (f["category"], f.get("type")) if p)
print(f"[{f['interestingness']:.2f} {kind}] {f['claim']}")[0.93 changes/step_discontinuity] quantity steps by 1.870979 across a diagonal boundary in unit_price and line_total (direction 124 degrees, standardized threshold 0.108533): evaluation-half jump 1.870979 (se 0.064488, z 29.012934), held-out gain 0.317601; the local gap is scale-invariant (ratio 0.78), typing it as a jump rather than a steep smooth transition
[0.93 changes/step_discontinuity] line_total steps by 34.256058 across a diagonal boundary in quantity and unit_price (direction 22 degrees, standardized threshold 1.334258): evaluation-half jump 34.256058 (se 1.246533, z 27.481079), held-out gain 77.366891; the local gap is scale-invariant (ratio 0.78), typing it as a jump rather than a steep smooth transition
[0.93 changes/step_discontinuity] unit_price steps by 5.313018 across a diagonal boundary in quantity and line_total (direction 79 degrees, standardized threshold -1.458529): evaluation-half jump 5.313018 (se 0.250124, z 21.241533), held-out gain 1.38034; the local gap is scale-invariant (ratio 0.84), typing it as a jump rather than a steep smooth transition
[0.87 dependencies/predictive] line_total follows 8 local linear regimes (first split quantity ≤ 4.000000; holdout R² 0.998 versus 0.917 for one linear model)
[0.86 dependencies/functional] line_total = quantity · unit_price within tolerance on all 4002 non-missing rows
[0.83 dependencies/predictive] unit_price follows 8 local linear regimes (first split line_total ≤ 112.640000; holdout R² 0.942 versus 0.791 for one linear model)
[0.82 uniqueness/uniqueness] column external_ref is nearly unique with 5 repeated non-null values
[0.82 uniqueness/uniqueness] column invoice_no is nearly unique with 2 repeated non-null values
[0.82 uniqueness/uniqueness] column order_id is nearly unique with 2 repeated non-null values
[0.82 dependencies/predictive] region_id is predictable from [region_name] (holdout accuracy 1.00 vs 0.35 majority baseline)
[0.81 dependencies/predictive] region_name is predictable from [region_id] (holdout accuracy 1.00 vs 0.34 majority baseline)
[0.81 dependencies/predictive] line_total is predictable from [quantity, unit_price] (holdout R² = 1.00)
[0.80 dependencies/predictive] order_id is predictable from [region_name, quantity, line_total, business_date] (holdout R² = 1.00)
[0.80 dependencies/predictive] unit_price is predictable from [order_id, region_id, region_name, quantity, line_total, business_date] (holdout R² = 0.98)
[0.78 conditional_rules/if_then] When region_name = "amer", region_id = 1 holds in 250 of 250 matching diagnostic rows (100.00%; baseline 30.86%)
[0.78 duplicates/duplicate_rows] 2 rows repeat earlier values across the compared columns (2 groups, largest 2)
[0.78 conditional_rules/if_then] When region_id > 0 and region_id <= 1, region_name = "amer" holds in 250 of 250 matching diagnostic rows (100.00%; baseline 30.86%)
[0.78 conditional_rules/if_then] When region_name = "apac", region_id = 2 holds in 267 of 267 matching diagnostic rows (100.00%; baseline 32.96%)
[0.77 conditional_rules/if_then] When region_id > 1 and region_id <= 2, region_name = "apac" holds in 267 of 267 matching diagnostic rows (100.00%; baseline 32.96%)
[0.77 conditional_rules/if_then] When region_id <= 0, region_name = "emea" holds in 293 of 293 matching diagnostic rows (100.00%; baseline 36.17%)
[0.77 conditional_rules/if_then] When region_name = "emea", region_id = 0 holds in 293 of 293 matching diagnostic rows (100.00%; baseline 36.17%)
[0.77 column_roles/duplicate_encoding] region_id and region_name encode the same grouping: 3 levels, paired one-to-one on all 4002 rows
[0.74 dependencies/functional] region_id → region_name (region_name is a function of region_id; 0 violations in 4002 rows)
[0.74 dependencies/functional] region_name → region_id (region_id is a function of region_name; 0 violations in 4002 rows)
[0.73 gaps/gap] column business_date is missing 2 business days at 2024-04-01..2024-04-04 (3 days elapsed)
[0.72 dependencies/predictive] unit_price depends on line_total and quantity jointly: 15.4% of the pair's joint partial-dependence variance is interaction (H² = 0.15); the interaction variance is 4.570895 against a target variance of 30.359352 (15.1%), from a surrogate with holdout R² 0.98 against 0.96 for an additive refit
[0.71 dependencies/predictive] line_total depends on quantity and unit_price jointly: 7.8% of the pair's joint partial-dependence variance is interaction (H² = 0.08); the interaction variance is 235.721584 against a target variance of 3049.199557 (7.7%), from a surrogate with holdout R² 1.00 against 0.91 for an additive refit
[0.69 associations/statistical] quantity and line_total show numeric association (maximum measured effect 0.809)
[0.64 formats/format_variation] column invoice_no mixes value formats (2: 50% '#####' vs 50% 'AAA-#####')
[0.64 concentration/hotspot] cell (region_id = '(0, 1]', region_name = 'amer') holds 1268 of 4002 scanned rows (31.7%) against 401.8 expected under independence of region_id, region_name — 3.2x the baseline (z = 43.2, bar 3.9 over 89 cells examined)
[0.60 associations/statistical] unit_price and line_total show numeric association (maximum measured effect 0.571)
[0.60 column_roles/identifier_role] column external_ref looks like a unique identifier (3997 distinct of 4002 values)
[0.60 column_roles/identifier_role] column invoice_no looks like a unique identifier (4000 distinct of 4002 values)
[0.60 column_roles/identifier_role] column order_id looks like a unique identifier (4000 distinct of 4002 values)
[0.55 column_roles/categorical_role] numeric column quantity behaves like a category (10 distinct values)
[0.55 column_roles/categorical_role] numeric column region_id behaves like a category (3 distinct values)
Review exact duplicates before using the key. order_id is flagged nearly
unique, with two repeated values. The duplicates finding reports two
rows that repeat earlier rows across every compared column. Its
specification records exact equality with missing values comparing equal. In this fixture the repeated rows
are planted re-sends. In a real extract, confirm whether repeated rows are
redundant before removing them; equality alone does not establish why
they occurred.
The external reference is not a key. external_ref is nearly unique
with five repeated values, and the duplicates finding does not explain
all of them: some rows with repeated references differ. Before using
external_ref as a join key, confirm that it is intended to be unique
and investigate the repeated references.
dirty = next(f for f in report.findings
if f["category"] == "uniqueness" and "external_ref" in f["subject"]["columns"])
for e in dirty["evidence"]:
if "examples" in e["metrics"]:
print("repeated external_ref values:", e["metrics"]["examples"])repeated external_ref values: ['ORD-000000', 'ORD-000001', 'ORD-000002']
decisions = pl.DataFrame({
"finding": [
"order_id nearly unique; 2 rows repeat earlier values exactly",
"external_ref nearly unique (5 repeats, rows differ)",
"invoice_no mixes 2 formats",
"2 business dates missing",
"line_total predictable from quantity × unit_price",
],
"decision": [
"deduplicate on order_id; then load key",
"do not join on it; query vendor",
"normalize in staging before parsing",
"send dates back to vendor",
"keep as recomputation check column",
],
"basis": ["measured · full", "measured · full",
"measured · full", "measured · full", "estimated · holdout"],
})Other relationships and coverage¶
There is no functional dependency from quantity alone to line_total
in this fixture: the total also depends on unit_price. The region
lookup is verified in both directions with zero violations. That
dependency is expected structure, rather than a data-quality error.
Coverage records which passes ran and their limits. FD verification uses the full 4,002-row table. Model passes use the encoded rows with their stated fitting limits and validation partitions (Coverage, budgets, and deadlines).
Use the counts and comparisons to identify records to review. Confirm the intended keys, formats, and calendar with the data owner before changing the extract.