Each input column is converted to a TypedCol once per profile. Only the
extraction module uses Polars APIs; the analysis passes operate on the
normalized column representation.
TypedCol is a closed enum: I64, F64, Str, StrOwned, Bool,
Date, Datetime, Null, Other. Dates are day numbers since the
epoch; datetimes are normalized to microseconds; Other covers dtypes the
profiler treats as opaque (reported in coverage, never silently
misread).
Borrowed, chunked, owned¶
Int64 and Float64 columns are the hot path, and they avoid copies in
the common cases. NumericValues has three shapes:
Borrowed — a single dense Arrow buffer plus a validity bitmap, sliced in place. This is the zero-copy case.
Chunked — multiple Arrow chunks, each with its own buffer and validity; lookups binary-search the chunk by row offset. Still no copy.
Owned — the fallback: a
Vec<Option<T>>materialized for dtypes that need conversion (other numeric widths, casts).
String columns come in two flavors — borrowed string views when the
chunk layout allows it, owned Strings otherwise — and the distinction
exists for one reason: high-cardinality sorting in the counting pass can
borrow views instead of allocating a million small strings
(Exact counting at any cardinality).
Canonical scalar rules¶
Extraction applies these rules before counts, hashing, and comparisons:
Floats: NaN never enters a map or a count (it is null-like); −0.0 is normalized to +0.0 so the two zeros compare and hash equal.
Nulls: validity comes from Arrow bitmaps; a null is a null, whether the chunk stores it as a bitmap or a sentinel.
Bools, dates, datetimes: stored as their numeric forms; dates compare as day numbers, which is what the temporal pass counts.
These rules affect results. An identity probe checking
y = 2·x at 1e-9 tolerance needs −0.0 to equal 0.0; a distinct count
needs an explicit policy for NaN; and gap analysis on day
numbers is only correct if every date path produces the same day
number.
The Python boundary¶
The pyo3 module accepts a Polars DataFrame (via the Arrow interchange
that pyo3-polars provides), extracts the TypedTable, and releases the
GIL for the whole profile: the compute is rayon-parallel across columns,
pairs, and targets, so threads that call profile concurrently do not
serialize behind the interpreter. The report crosses back as serialized
JSON plus a thin object; there is no lazy handle into Rust memory, and
the package does not require pandas.
The binding exposes profile, available_detectors, and the report
object. Polars integration is concentrated in onceover-core’s extraction
module and onceover-py, so changes to interchange APIs can usually be
handled there.