Mitigating Class-Tail Undercoverage in Medical Vision-Language Models under Clinical Shift
Medical vision-language models (VLMs) can maintain high marginal coverage after clinical shift while severely under-covering individual disease classes, a failure invisible to source prevalence metrics CALCoDe (Class-Tail Adaptive Localized Conformal Deferral) is introduced as a post-hoc reliability layer for frozen medical VLMs that addresses both localized and tail-aware coverage gaps simultaneously The method uses cross-fitted validation predictions to identify at-risk classes and disjoint ca
Analysis
TL;DR
- Medical vision-language models (VLMs) can maintain high marginal coverage after clinical shift while severely under-covering individual disease classes, a failure invisible to source prevalence metrics
- CALCoDe (Class-Tail Adaptive Localized Conformal Deferral) is introduced as a post-hoc reliability layer for frozen medical VLMs that addresses both localized and tail-aware coverage gaps simultaneously
- The method uses cross-fitted validation predictions to identify at-risk classes and disjoint calibration splits to estimate class-conditional tail thresholds, combining them via one-sided maximum with localized conformal thresholds
- CALCoDe includes an independently calibrated support audit that defers cases with insufficient inlier support, providing finite-sample coverage guarantees under exchangeability assumptions
- Evaluated across two dermatology shifts and four frozen VLM backbones, CALCoDe is the only method achieving both 0.95 marginal and worst-class accepted coverage across all eight settings, outperforming sTACP (0.926) and LCP-VLM (0.864) on worst-class coverage
Why It Matters
This work addresses a critical reliability gap in deploying medical VLMs in real-world clinical settings, where distribution shifts between training and deployment data can silently degrade coverage for rare or underrepresented disease classes. For AI practitioners building healthcare systems, it demonstrates that standard conformal prediction methods may provide misleading guarantees of reliability, and introduces a practical, model-agnostic reliability layer that can be applied to frozen VLMs without retraining.
Technical Details
- CALCoDe Architecture: A post-hoc conformal prediction framework that operates on frozen medical VLMs without requiring model fine-tuning. It combines class-tail adaptive thresholds with localized conformal prediction using a one-sided maximum operation, ensuring that protected classes receive additional coverage guarantees beyond standard localized rules.
- Validation and Calibration Strategy: Uses cross-fitted validation predictions to identify classes at risk of undercoverage, then employs a disjoint calibration split to estimate class-conditional tail thresholds for those identified classes. This separation prevents overfitting and ensures valid finite-sample coverage guarantees.
- Support Audit Mechanism: An independently calibrated support audit component defers prediction cases that lack sufficient inlier support, providing an additional safety layer for out-of-distribution or ambiguous inputs.
- Theoretical Guarantees: Under exchangeability assumptions among accepted examples within each protected class, CALCoDe provides finite-sample coverage at the prespecified guard level and provably contains the corresponding localized conformal sets.
- Empirical Evaluation: Tested across two dermatology domain shifts (HAM10000→ISIC 2019 and HAM10000→PAD-UFES-20) with four frozen VLM backbones (BiomedCLIP, OpenAI CLIP ViT-B/32, PubMedCLIP ViT-B/32, MedSigLIP-448), achieving 0.970 average worst-class accepted coverage on HAM10000→ISIC 2019 compared to 0.926 (sTACP) and 0.864 (LCP-VLM).
Industry Insight
- Healthcare AI developers should treat marginal coverage metrics with skepticism when deploying VLMs clinically, as they can mask severe class-wise coverage failures that disproportionately affect rare conditions or underrepresented patient populations
- Post-hoc reliability layers like CALCoDe offer a practical deployment pathway for existing frozen medical VLMs without costly retraining, making conformal prediction more accessible for clinical integration
- The finding that the affected undercovered class varies with both acquisition protocol and backbone geometry suggests that robustness auditing should be performed across multiple model architectures and data collection protocols rather than relying on a single deployment configuration
Disclaimer: The above content is generated by AI and is for reference only.