Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays
Per-field selective risk control in document extraction silently violates its own trust contract on real documents due to three diagnosed failure modes: document clustering (design effect 1.84–2.45), score-refit leakage (coverage 0.416 at risk 0.127, violating alpha=0.10 in 95% of splits), and tie-mass pathology (degenerate score collapsing threshold grid from 0.030 to 0.001) A "validity ladder" organizes progressively rigorous fixes with guarantee forms stated per tier, including a fit/val spli
Analysis
TL;DR
- Per-field selective risk control in document extraction silently violates its own trust contract on real documents due to three diagnosed failure modes: document clustering (design effect 1.84–2.45), score-refit leakage (coverage 0.416 at risk 0.127, violating alpha=0.10 in 95% of splits), and tie-mass pathology (degenerate score collapsing threshold grid from 0.030 to 0.001)
- A "validity ladder" organizes progressively rigorous fixes with guarantee forms stated per tier, including a fit/val split protocol that restores expected-selective-risk control for learned fusion (coverage 0.318 at risk 0.096 at nominal alpha=0.10)
- Mondrian Learn-then-Test with exact binomial tails yields per-group PAC certificates across field-iid (0.171 at risk 0.068), cluster-corrected (0.140), and doc-iid (0.060) tiers, with the document-level tier being the only one matching real deployment conditions but currently near-vacuous in coverage
- Support-bin (pre-specified provenance taxonomy) wins every rigor tier on sonnet CORD capture (p<1e-4, Bonferroni-corrected) but fails to replicate under haiku or qwen on the same documents; pooled thresholds win on higher-accuracy corpora instead
- A blind three-annotator human-gold audit verifies the practical tier's accepted-set risk at 1.3% against a 10% budget (Fleiss' kappa=0.83), and the full Apache-2.0 release includes seed-pinned, regression-gated procedures
Why It Matters
This work directly addresses a critical reliability gap in production document-extraction systems: selective risk control—the mechanism by which systems decide which fields to auto-accept versus route for human review—is systematically broken on real-world data. For AI practitioners building extraction pipelines, the findings mean that naive confidence-thresholding will silently overpromise accuracy, and the validity ladder provides a practical roadmap to progressively stronger guarantees. The paper also establishes an important empirical principle: no single certification strategy universally wins, with conditioning helping where pooled methods cannot certify and vice versa.
Technical Details
- Failure mode diagnosis: On 13,859 fields from 800 CORD receipts (49.0% baseline accuracy with Claude Sonnet 5), the authors identify document clustering inducing design effects of 1.84–2.45, score-refit leakage causing severe overfitting to validation thresholds, and tie-mass pathology where degenerate scores collapse the usable threshold range
- Mondrian Learn-then-Test framework: Applies conformal prediction with exact binomial tails to produce per-group PAC certificates, stratifying by field-iid, cluster-corrected, and document-iid assumptions; the document-iid tier achieves the tightest certificate (0.060) but at near-vacuous coverage
- Support-bin taxonomy: A pre-specified provenance-based stratification that dominates on low-accuracy corpora under rigorous tiers but fails to generalize across model families (Sonnet vs. Haiku vs. Qwen on identical documents)
- Fit/val split protocol: A learned fusion approach using held-out validation restores expected risk control (0.096 realized at nominal 0.10) but is not a certificate—realized risk exceeds alpha in 47.5% of resplits
- Human validation: Three-annotator blind audit confirms 1.3% actual error rate in the accepted set against a 10% budget, with high inter-annotator agreement (kappa=0.83) and labels biased one-sidedly pessimistic
Industry Insight
- Teams deploying document extraction at scale should treat per-field selective risk control as a certification problem, not a threshold-tuning problem; the three failure modes are likely endemic across receipt, invoice, and form extraction pipelines using off-the-shelf LLM APIs
- The non-replicability of Support-bin across model families suggests that provenance-based conditioning is model-dependent and should be validated per-deployment, while pooled thresholds remain the more portable default for high-accuracy systems
- The Apache-2.0 release with seed-pinned, regression-gated procedures offers a rare reproducible benchmark for the emerging field of certified selective prediction in NLP, and the validity ladder framework could become a standard reference for auditing production extraction systems
Disclaimer: The above content is generated by AI and is for reference only.