Column cards - onceover

A column card summarizes one column using all rows. Counts and summary statistics are exact; the role label is inferred from the dtype and counts. Cards also report concentration measures and, for strings, format signatures. This chapter describes how these fields are computed.

Exact counting at any cardinality¶

Every count on a card — distinct, nulls, top values, concentration — is exact. The counting routine is adaptive over a compact value key: 64-bit integers as-is, floats as bit patterns (NaN never stored, −0.0 normalized to +0.0), strings, and booleans. It accumulates in a hash map until the map holds 4,096 distinct keys; if the column has more, it switches to sorting the compact keys and counting runs. Both paths examine every row and produce an exact distinct count; the limit determines which counting algorithm runs.

Only the reported top values (top 64, ordered by count descending then key ascending) own their string data; the high-cardinality sort borrows strings. Concentration is computed over the full value distribution, not over only the top-k values. effective_cardinality uses the counts of all values, including on high-cardinality columns. The implementation notes live in the counting chapter.

Role inference¶

Roles are assigned deterministically from dtype and exact counts:

Role	Rule
`all_null`	no non-null values
`constant`	one distinct value
`boolean`, `date`, `datetime`	boolean / date / datetime dtype
`categorical` (strings)	≤ 500 distinct values
`free_text` (strings)	more than 500 distinct values
`categorical` (numeric)	≤ 12 distinct with n ≥ 30, or distinct ratio ≤ 2% with ≤ 100 distinct and n ≥ 100
`identifier` (numeric)	distinct ratio ≥ 0.98
`continuous` (numeric)	otherwise
`other`	dtype the profiler treats as opaque (reported in coverage)

Role labels can differ from a column’s intended meaning. Numeric measurements with no repeats, such as random floats, have distinct ratio 1.0 and receive the identifier role even when they are not identifiers. Date and datetime dtypes have their own roles. Similarly, numeric codes can fall outside the categorical thresholds. Check the label against the column’s meaning. Numeric columns inferred categorical carry treat_as_categorical in the card’s action_hints list; the finding that says the same thing (column_roles) is a separate, inferred-basis statement.

import polars as pl

import onceover

n = 2_000
df = pl.DataFrame({
    "flag": [True, False] * (n // 2),
    "code": [i % 7 for i in range(n)],          # 7-level numeric code
    "tiny": [i % 2 for i in range(n)],           # 2-level numeric code
    "random_float": [i * 0.5 for i in range(n)],  # distinct ratio 1.0
})
report = onceover.profile(df, budget="quick")
for name in ["flag", "code", "tiny", "random_float"]:
    card = report.columns[name]
    print(f"{name:14s} role={card['role']:12s} distinct={card['distinct']:6d} "
          f"hints={card.get('action_hints', [])}")

flag           role=boolean      distinct=     2 hints=[]
code           role=categorical  distinct=     7 hints=['treat_as_categorical']
tiny           role=categorical  distinct=     2 hints=['treat_as_categorical']
random_float   role=identifier   distinct=  2000 hints=[]

Concentration¶

The concentration block summarizes the value distribution with three complementary measures, all exact:

HHI — the Herfindahl–Hirschman index, the sum of squared shares. A constant scores 1.0; k equally frequent values score 1/k.
Shannon entropy in nats, and its exponent, effective cardinality — “this column behaves like it has N distinct values”. A six-level column dominated by one level scores near 1; a balanced six-level column scores near 6.
top-1 and top-3 shares — the mass of the most frequent values.

Concentration findings use fixed thresholds, and each finding states the number that triggered it: a constant (distinct ≤ 1), a near-constant (top-1 share ≥ 0.99), and a high-concentration flag when the top value holds ≥ 90% of mass, reported with the effective cardinality. A column below these thresholds receives no concentration finding.

Format patterns¶

String cards carry format_patterns: signatures in which digits become #, lowercase letters a, uppercase letters A, and other characters are retained. Signatures are truncated at 24 characters. For example, invoices split between AAA-#### and #### produce two patterns, each with its share. A formats finding requires the top pattern to account for less than 90% of values and the second for at least 5%. The finding includes both shares and example values.

These signatures can expose mixed date formats, padding, or inconsistent code formats. They describe character shapes; they do not validate dates, currencies, phone numbers, or other domain-specific values.