Column cards - onceover

A column card summarizes a column’s inferred role, counts, statistics, and value distribution. report.columns contains one dict per column. Cards use every input row; relationship sampling does not affect them.

The fixture includes constant and all-null columns, repeated categories, numeric measurements, dates, and invoice numbers with two formats.

import random

import polars as pl

import onceover

rng = random.Random(11)
n = 1500

base = pl.date_range(pl.date(2024, 1, 1), pl.date(2024, 1, 1) + pl.duration(weeks=n - 1),
                     interval="1w", eager=True)
plan = rng.choices(["starter", "growth", "enterprise"], weights=[70, 20, 10], k=n)
channel = ["web" if rng.random() < 0.995 else "app" for _ in range(n)]
unit = ["kg"] * n
legacy_flag = [None] * n
risk_score = [round(rng.uniform(0, 10), 1) for _ in range(n)]
email_domain = rng.choices(["gmail.com", "outlook.com", "corp.io", "proton.me"],
                           weights=[50, 25, 20, 5], k=n)
invoice_no = [f"INV-{i:04d}" if i % 2 == 0 else f"{i:04d}" for i in range(n)]
status_code = [rng.randrange(1, 7) for _ in range(n)]
is_active = [rng.random() < 0.8 for _ in range(n)]

df = pl.DataFrame({
    "signup_date": base,
    "plan": plan,
    "channel": channel,
    "unit": unit,
    "legacy_flag": legacy_flag,
    "risk_score": risk_score,
    "email_domain": email_domain,
    "invoice_no": invoice_no,
    "status_code": status_code,
    "is_active": is_active,
})

report = onceover.profile(df, budget="standard")
print(report.overview["headline"])

1500 rows × 10 columns (standard budget); key candidate: invoice_no; 1 functional dependencies, 0 predictive dependencies, 0 conditional rules

A card at a glance¶

import json

print(json.dumps(report.columns["risk_score"], indent=2))

{
  "name": "risk_score",
  "dtype": "float64",
  "role": "continuous",
  "n": 1500,
  "null_count": 0,
  "null_fraction": 0.0,
  "distinct": 101,
  "distinct_ratio": 0.06733333333333333,
  "numeric_stats": {
    "min": 0.0,
    "max": 10.0,
    "mean": 4.930866666666659,
    "std": 2.8732421725678146,
    "q25": 2.475,
    "q50": 4.9,
    "q75": 7.3,
    "zeros": 9,
    "negatives": 0,
    "integral": false
  },
  "concentration": {
    "hhi": 0.010659555555555556,
    "entropy_nats": 4.576751934201698,
    "effective_cardinality": 97.19817486700717,
    "top1_share": 0.017333333333333333,
    "top3_share": 0.050666666666666665
  },
  "top_values": [
    {
      "value": "6.1",
      "count": 26,
      "share": 0.017333333333333333
    },
    {
      "value": "6",
      "count": 25,
      "share": 0.016666666666666666
    },
    {
      "value": "7.3",
      "count": 25,
      "share": 0.016666666666666666
    },
    {
      "value": "5.5",
      "count": 24,
      "share": 0.016
    },
    {
      "value": "6.8",
      "count": 24,
      "share": 0.016
    }
  ]
}

risk_score has the inferred role continuous. Its n, null_count, distinct, and distinct_ratio describe all input rows. numeric_stats contains the mean, standard deviation, quartiles, zero and negative counts, and whether every value is integral. concentration describes how often values repeat; top_values lists the most frequent values.

The statistics block depends on dtype: strings have string_stats, booleans have bool_stats, and dates have temporal_stats. The date scan adds the most frequent interval to temporal_stats. For signup_date, that interval is one week:

print(json.dumps(report.columns["signup_date"]["temporal_stats"], indent=2))

{
  "min": "2024-01-01",
  "max": "2052-09-23",
  "span_days": 10493,
  "modal_interval_days": 7
}

Roles¶

Role inference applies rules to dtype and exact counts. The resulting label describes observed behavior and may differ from the column’s meaning in the application:

Role	Inferred when	Notes
`all_null`	no non-null values	flagged as a finding too
`constant`	one distinct value	drop candidate
`boolean`	boolean dtype
`categorical`	strings with ≤ 500 distinct values; numeric with ≤ 12 distinct (n ≥ 30), or ≤ 100 distinct and ≤ 2% of rows (n ≥ 100)	numeric categories carry a `treat_as_categorical` hint
`free_text`	strings with > 500 distinct values	excluded from nominal model passes
`identifier`	numeric with distinct ratio ≥ 0.98	check intended meaning before treating as a join key
`continuous`	numeric, otherwise
`date` / `datetime`	date or datetime dtype	datetime analyzed at day granularity for gaps
`other`	dtype the profiler treats as opaque	reported in coverage

A numeric measurement with no repeated values, such as random floats, has a distinct ratio of 1.0 and receives the identifier role. That rule cannot establish whether the column identifies an entity; check its definition and use in the application. Date and datetime dtypes retain their own roles. The numeric-categorical thresholds also need context: a 13-level ordered code may still represent categories.

Concentration¶

The concentration block describes the frequencies of non-null values. The Herfindahl–Hirschman index is the sum of squared value shares: 1.0 for a constant column, or $1/k$ for $k$ equally frequent values. Shannon entropy is reported in nats. Effective cardinality, $\exp(\text{entropy})$ , is the number of equally frequent values that would have the same entropy. A six-level column dominated by one value has effective cardinality near 1; a balanced six-level column has 6. top1_share and top3_share give the fractions occupied by the most frequent value and the three most frequent values.

The email_domain card illustrates these measures:

print(json.dumps(report.columns["email_domain"]["concentration"], indent=2))
print(json.dumps(report.columns["email_domain"]["top_values"], indent=2))

{
  "hhi": 0.34874666666666665,
  "entropy_nats": 1.1724469750309252,
  "effective_cardinality": 3.229886428078974,
  "top1_share": 0.48533333333333334,
  "top3_share": 0.9506666666666667
}
[
  {
    "value": "gmail.com",
    "count": 728,
    "share": 0.48533333333333334
  },
  {
    "value": "outlook.com",
    "count": 402,
    "share": 0.268
  },
  {
    "value": "corp.io",
    "count": 296,
    "share": 0.19733333333333333
  },
  {
    "value": "proton.me",
    "count": 74,
    "share": 0.04933333333333333
  }
]

All cards at a glance¶

Render one row per column to compare roles, null fractions, and value frequencies. The Great Tables styles below wrap long column names and roles:

from great_tables import GT, loc, style

rows = []
for name, card in report.columns.items():
    conc = card.get("concentration")
    rows.append({
        "column": name,
        "dtype": card["dtype"],
        "role": card["role"],
        "nulls": f"{card['null_fraction']:.1%}",
        "distinct": card["distinct"],
        "top1 share": f"{conc['top1_share']:.1%}" if conc else "—",
        "eff. cardinality": f"{conc['effective_cardinality']:.2f}" if conc else "—",
    })

(GT(pl.DataFrame(rows))
 .tab_header(title="Column cards at a glance", subtitle="Every count is exact, over all rows")
 .tab_options(table_width="100%", table_layout="fixed",
              table_font_size="14px", data_row_padding="9px")
 .tab_style(style.css("white-space: normal; overflow-wrap: anywhere; line-height: 1.5;"),
            loc.body()))

The table shows unit as constant, legacy_flag as all null, plan and email_domain as concentrated categoricals, risk_score as continuous, and invoice_no as free text.

invoice_no has the free_text role because it exceeds the 500-value threshold for string categories. It is also null-free, with 1,500 distinct values in 1,500 rows, so its values can identify rows in this table. The role excludes it from nominal model passes; uniqueness is checked separately from its exact counts.

Cards are also the input to charts. The top values of a categorical column make a quick bar chart — Altair over the card’s top_values:

import altair as alt

top = pl.DataFrame([
    {"value": t["value"], "share": t["share"]}
    for t in report.columns["email_domain"]["top_values"]
])

alt.Chart(top).mark_bar().encode(
    x=alt.X("share:Q", title="share of non-null values", axis=alt.Axis(format="%")),
    y=alt.Y("value:N", sort="-x"),
).properties(width=460, height=160)

Format patterns¶

String cards carry one more block: format_patterns, the shape signatures of the column’s values. Digits become #, lowercase letters a, uppercase letters A, other characters are kept, and signatures truncate at 24 characters. Two clusters of signatures in a column that should have one can indicate mixed formats, as in this invoice_no fixture:

print(json.dumps(report.columns["invoice_no"]["format_patterns"], indent=2))

[
  {
    "pattern": "####",
    "count": 750,
    "share": 0.5,
    "example": "0001"
  },
  {
    "pattern": "AAA-####",
    "count": 750,
    "share": 0.5,
    "example": "INV-0000"
  }
]

Half the invoice numbers have the pattern AAA-#### and half have ####. The card reports both signatures and their shares without listing all 1,500 values. The mixed-format finding suggests normalization before parsing; whether to change these values depends on the expected invoice format.

Card-level findings¶

The profiler also creates findings from card counts: constant and all-null columns, nearly constant two-level columns such as channel, concentrated values, numeric categories such as status_code, and identifier-like columns. These receive lower selection priority than relationship findings. The loop below prints the findings for unit, legacy_flag, channel, and status_code, along with any suggested action.

single = {"unit", "legacy_flag", "channel", "status_code"}
for f in report.findings:
    if len(f["subject"]["columns"]) == 1 and f["subject"]["columns"][0] in single:
        print(f"[{f['category']}] {f['claim']}")
        if f.get("interpretation"):
            print(f"    → {f['interpretation']}")

[concentration] column channel is nearly constant: 99.7% 'web' vs 4 'app'
    → treat the rare value as the signal
[concentration] column unit is constant ('kg')
    → drop from modeling; carries no information
[column_roles] numeric column status_code behaves like a category (6 distinct values)
    → treat as categorical in models; do not scale
[missingness] column legacy_flag is entirely null
    → consider dropping

The thresholds for role rules, concentration flags, and format findings are listed in Column cards.