Who Gets Missed in the Tail? Thresholded Subgroup Underdiagnosis in Long-Tailed Chest X-ray Classification
High-ranking metrics like mAP do not guarantee fair diagnosis for rare conditions, as thresholding can leave vulnerable subgroups with unacceptably high false negative rates. Combining group-tail weighting during training with tail-aware threshold selection significantly reduces worst-group False Negative Rates (FNR) across sex, age, and other demographics. Aggregate group robustness techniques alone are insufficient to eliminate rare subgroup misses; explicit audit of the score-to-decision conv
Analysis
TL;DR
- High-ranking metrics like mAP do not guarantee fair diagnosis for rare conditions, as thresholding can leave vulnerable subgroups with unacceptably high false negative rates.
- Combining group-tail weighting during training with tail-aware threshold selection significantly reduces worst-group False Negative Rates (FNR) across sex, age, and other demographics.
- Aggregate group robustness techniques alone are insufficient to eliminate rare subgroup misses; explicit audit of the score-to-decision conversion process is required.
- The study establishes that rare-label fairness in medical imaging is a joint function of the specific finding, patient subgroup, and the chosen operating threshold.
Why It Matters
This research highlights a critical gap between standard model evaluation and clinical safety, demonstrating that models appearing robust on average may systematically fail minority or rare-disease populations when deployed. For AI practitioners, it underscores the necessity of moving beyond global accuracy or ranking metrics to include subgroup-specific threshold audits before deployment. This ensures that medical AI systems do not inadvertently perpetuate health disparities by missing diagnoses in vulnerable groups.
Technical Details
- Problem Scope: Analyzes pre-deployment fairness in long-tailed multi-label Chest X-ray (CXR) classification, specifically focusing on the transition from continuous scores to binary decisions.
- Methodology: Employs a "diagnostic ladder" approach to isolate effects of class-level long-tail losses, subgroup-aware weighting, group robustness, and threshold selection.
- Datasets: Evaluates performance on VinDr-CXR and MIMIC-CXR/CXR-LT datasets.
- Key Results (VinDr-CXR): Applying group-tail weighting followed by tail-aware thresholding reduced tail FNR from 0.665 to 0.269, sex worst-group FNR from 0.705 to 0.157, and age worst-group FNR from 0.822 to 0.133. Macro-mAP improved slightly from 0.611 to 0.635.
- Key Results (MIMIC-CXR/CXR-LT): Reduced tail FNR from 0.866 to 0.741 and lowered worst-group FNR across sex, age, race, and insurance categories, though residual missed-positive rates remained high.
- Statistical Validation: Used paired bootstrap contrasts to support FNR reductions and compared against GroupDRO reference runs to show that aggregate group robustness alone fails to remove rare subgroup misses.
Industry Insight
- Audit Protocols: Healthcare AI developers must implement subgroup-specific threshold audits as a standard part of the validation pipeline, rather than relying solely on global performance metrics.
- Fairness Engineering: Training strategies should incorporate subgroup-aware weighting tailored to rare classes, as generic group robustness methods are inadequate for addressing tail-end disparities.
- Regulatory Compliance: Regulatory frameworks for medical AI should require evidence that decision thresholds do not disproportionately increase false negatives for protected subgroups or rare conditions.
Disclaimer: The above content is generated by AI and is for reference only.