When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide
Weak overlap in off-policy evaluation is driven by logger-target action alignment, not logging sharpness alone; action-level disagreement between logger and target policy collapses support Cross-fitting the outcome nuisance does not resolve the optimizer's curse when the policy is fit on evaluation data; honest policy-level splitting is required instead Propensity-estimation error is the largest source of estimator degradation, disproportionately hurting IPS while leaving doubly-robust estimator
Analysis
TL;DR
- Weak overlap in off-policy evaluation is driven by logger-target action alignment, not logging sharpness alone; action-level disagreement between logger and target policy collapses support
- Cross-fitting the outcome nuisance does not resolve the optimizer's curse when the policy is fit on evaluation data; honest policy-level splitting is required instead
- Propensity-estimation error is the largest source of estimator degradation, disproportionately hurting IPS while leaving doubly-robust estimators largely unaffected
- Effective sample size can rank risk across logging environments but is unreliable for ranking candidates within a single log, and its cut point does not transfer across contexts
- The authors release a controlled, reproducible benchmark with synthesized logging and a propensity floor of 0.02, reducing tuned hybrids to their untuned parents and isolating four practically distinct estimators
Why It Matters
This work directly addresses a critical pain point for AI practitioners deploying budget-constrained targeting systems: knowing whether offline evaluation metrics reliably predict real-world policy performance. The findings challenge common assumptions about cross-fitting and overlap diagnostics, providing actionable guidance for practitioners who rely on off-policy evaluation before deployment.
Technical Details
- Benchmarks six off-policy estimators across five datasets and two known-effect sweeps, validated against a non-simulated paired reference, with all exact-value surfaces being synthetic or semi-synthetic
- Propensities are floored at 0.02 to ensure bounded weights, which collapses two tuned hybrid estimators to their untuned parents, leaving four practically distinct estimators for comparison
- Demonstrates that sharpening a logger built from the target's own score barely improves overlap, while action-level disagreement between logger and target policy is the dominant factor
- Shows that out-of-fold propensity estimation degrades Inverse Propensity Scoring (IPS) more than any other stress condition, can invert the overlap diagnostic itself, and has minimal impact on doubly-robust estimation
- Proposes honest policy-level splitting as the correct remedy for optimizer's curse, reframing the problem as a change of estimand (targeting the learning procedure's value) rather than attempting to de-bias the full-sample policy
Industry Insight
- Practitioners should prioritize logger-target action alignment over logging sharpness when designing or selecting logging systems for off-policy evaluation; investing in better action coverage matters more than refining score distributions
- Cross-fitting alone is insufficient to protect against optimizer's curse in policy evaluation; organizations should adopt honest policy-level splitting or hold out data specifically for policy evaluation rather than reuse
- Propensity-estimation quality should be treated as the primary diagnostic for offline evaluation reliability, especially for IPS-based approaches; doubly-robust estimators offer more resilience when propensity models are imperfect
Disclaimer: The above content is generated by AI and is for reference only.