Testing strategy

Tests compare findings with planted data, exercise edge cases, and use negative controls that should not produce a finding.

Planted structure, asserted recovery¶

Unit and integration tests build tables with known structure and check that the report recovers the expected statements. A fixture with three FD violations asserts the approximate dependency with exactly three violations; a decision-table fixture asserts the cross-validated accuracy against a hand-computable baseline; a regimes fixture asserts the recovered slopes and the breakpoint. Tests also check cases that should not produce a finding: a near-majority target must yield no PRISM rules, a unique column must suppress the “X → everything” flood, a failed rule evaluation must drop the candidate.

The taxonomy has its own tests: equivalent statements from different detectors consolidate into one finding without pooling counts; different directions, conditions, and forms stay distinct; selection limits apply across detectors; a custom detector emitting an existing category is consolidated like a built-in.

Adversarial regressions¶

Regression tests preserve cases that exposed errors. The dependence test checks tied and independent samples. Chatterjee’s ξ breaks x-ties uniformly at random because a fixed ordering can introduce dependence on row position. A block-id column against a row counter must report ξ ≈ 0, not the ≈ 0.99 a stable-order implementation produced; tied x against independent y must not inflate; a step function in y must still score high. Each of those is a planted table and an assertion on the number.

Regression tests also cover the sampler (strata coverage, no periodic aliasing), the FD screen (near-unique determinants excluded), the identity probe (violations counted, tolerance honored), and the rule evaluator (thresholds inclusive of their boundaries).

Python round-trip and notebooks¶

The Python suite runs the compiled wheel end-to-end: headline shape, finding categories, coverage statuses, determinism of the serialized report, the detector-selection contract, and the schema-version field. CI executes onceover_analysis.ipynb cell by cell against the freshly built wheel and regenerates its stored outputs. This checks that the example runs against the package being tested. The book chapters you are reading execute the same way during the documentation build.

The scale gate¶

Before any publication, a reproducible benchmark profiles a wide, known table: 2,000,000 rows × 200 columns, fixed seed, independent noise, an exact integer key, a deliberately dirty key, and three relationships planted in the last columns — an affine identity, a nonlinear response, and a code-to-label dependency — where a screener that only examined the first few dozen columns would miss them.

The script fails unless every planted structure is recovered and the exact cardinality checks hold. The recorded run, on macOS arm64 with four Rayon threads:

All eight checks pass, including recovery of the late-column relationships, and no independent noise target is flagged as predictable — the false-positive control. A smaller regression subset runs in CI on every push; the full gate is the pre-publish step, with the methodology and the recorded JSON in the benchmark notes.

These tests verify behavior on controlled fixtures. They do not establish that findings on a new dataset are correct; the evidence and coverage fields describe what to verify.