Exact counting at any cardinality

The profiler counts distinct values over every row. It uses a hash map for columns with few distinct values and switches to sorting when cardinality exceeds 4,096.

The value key¶

Counting operates on VKey, a compact enumerable form: 64-bit integers as-is; floats as their bit patterns (NaN excluded, −0.0 normalized to +0.0); strings; booleans. VKey is ordered — by value within a variant, so a million distinct integers sort as integers, not as strings — and the order is total, which is what makes the sorted-run fallback deterministic under parallelism.

The card keeps only what it needs: the exact distinct count over all non-null values, the non-null row count, and the top 64 values by (count descending, key ascending). High-cardinality sorting borrows string views rather than owning them — only the reported top values ever own their strings — so a 10-million-distinct string column does not become a 10-million-allocation column.

Two algorithms, one exact answer¶

build_counts starts with a hash map using rustc_hash’s FxHashMap. It uses a non-cryptographic hash for speed and compares keys for equality, so collisions do not merge distinct values. It accumulates exact counts until the map holds 4,096 distinct keys. If the column has more, the routine switches: sort the compact keys, walk the sorted array once, and count runs. Both paths examine every row; both produce the exact distinct count; the limit chooses the algorithm, never the accuracy.

The map is efficient when few distinct values repeat across many rows. For high-cardinality columns, sorting stores compact values in one array instead of maintaining a large map. Both algorithms count all input rows.

Concentration in the same pass¶

HHI, entropy, and effective cardinality are accumulated while counting — over the full value distribution, not a top-k sample — so effective_cardinality uses every distinct value even on a million-distinct column. To keep floating-point results deterministic, the hash-map path sorts its entries by key before accumulating the concentration summary, so the floating-point summation order is fixed regardless of hash-table layout or thread count (Determinism under parallelism).

Key discovery and role inference use the counts. Dependency and duplicate checks use the same normalized values. The full report retains the distinct count and non-null count used by each pass.