A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It
Standard marginal conformal prediction fails to provide reliable uncertainty estimates for minority classes in imbalanced drug discovery datasets, despite meeting global coverage targets. The study demonstrates that minority class coverage can drop drastically (e.g., to 4.2% for toxicity) due to a conservation identity where majority surplus amplifies minority shortfall based on imbalance ratios. Class-conditional (Mondrian) conformal prediction effectively restores per-class reliability and min
Analysis
TL;DR
- Standard marginal conformal prediction fails to provide reliable uncertainty estimates for minority classes in imbalanced drug discovery datasets, despite meeting global coverage targets.
- The study demonstrates that minority class coverage can drop drastically (e.g., to 4.2% for toxicity) due to a conservation identity where majority surplus amplifies minority shortfall based on imbalance ratios.
- Class-conditional (Mondrian) conformal prediction effectively restores per-class reliability and minority coverage to target levels with only a modest increase in prediction set size.
- The failure is robust across different model architectures (Random Forest, Graph Networks, Chemical Language Models) and persists under realistic scaffold splits, indicating a fundamental issue with marginal calibration under imbalance.
Why It Matters
This research highlights a critical blind spot in applying conformal prediction to real-world scientific domains like drug discovery, where class imbalance is common. Relying solely on global coverage metrics can lead to dangerously overconfident models for rare but high-stakes events, such as clinical trial toxicity or blood-brain barrier penetration. Practitioners must adopt class-aware calibration methods to ensure safety and reliability in virtual screening campaigns.
Technical Details
- Problem Identification: Evaluated standard marginal conformal prediction across four chemical datasets, revealing that while global 90% coverage was achieved, minority class coverage fell to 64.8% (blood-brain-barrier) and 4.2% (toxicity).
- Theoretical Explanation: Introduced a "conservation identity" showing that the minority's coverage deficit equals the majority's surplus, scaled by the class imbalance ratio, allowing precise prediction of the gap magnitude.
- Model Agnosticism: Confirmed the failure across diverse architectures including Random Forests, Graph Neural Networks, and frozen chemical language models, proving the issue stems from data imbalance rather than specific model flaws.
- Solution Validation: Demonstrated that Mondrian (class-conditional) conformal prediction successfully restored minority coverage to the target level across all datasets, with minimal impact on prediction set efficiency.
- Practical Impact: Proposed a one-number diagnostic for detecting this failure and showed via cost modeling that abstaining on affected compounds using class-conditional methods can flip screening campaigns from net-negative to net-positive utility.
Industry Insight
- Audit Global Metrics with Caution: Do not rely exclusively on aggregate accuracy or global coverage rates when deploying calibrated models in imbalanced settings; always inspect per-class coverage statistics.
- Adopt Class-Conditional Calibration: Implement Mondrian or similar class-conditional conformal prediction techniques in drug discovery pipelines to mitigate risks associated with rare adverse events or properties.
- Implement Diagnostic Protocols: Integrate simple diagnostic checks for minority coverage gaps early in the model validation phase to prevent silent failures in virtual screening campaigns.
Disclaimer: The above content is generated by AI and is for reference only.