Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety
Clinician pairwise preferences are a poor proxy for clinical safety in LLM evaluation, with high-preference models still showing substantial clinically meaningful failures Safety-critical failures (rated ≤ -1 on Harmlessness and Accuracy) are unevenly distributed across medical specialties, creating invisible domain-specific "no-go zones" Surface-level features explain slightly more preference variation than actual safety-critical rubric differences, and many preference votes carry no positive s
Analysis
TL;DR
- Clinician pairwise preferences are a poor proxy for clinical safety in LLM evaluation, with high-preference models still showing substantial clinically meaningful failures
- Safety-critical failures (rated ≤ -1 on Harmlessness and Accuracy) are unevenly distributed across medical specialties, creating invisible domain-specific "no-go zones"
- Surface-level features explain slightly more preference variation than actual safety-critical rubric differences, and many preference votes carry no positive safety signal
- The authors introduce a clinically adjusted preference ranking that combines pairwise preference with rubric-derived feedback for more safety-aware model ordering
- The study analyzed 26,804 pairwise judgments from 736+ clinicians across 28+ countries evaluating 13 LLMs on the MOOVE platform
Why It Matters
This research directly challenges a widely adopted evaluation methodology in clinical AI, where pairwise preference ranking has become a standard proxy for safety assessment. For AI practitioners building or deploying clinical LLMs, relying solely on preference-based leaderboards may create a false sense of security while dangerous domain-specific failures remain undetected. The findings have immediate implications for how clinical AI systems are evaluated, benchmarked, and regulated.
Technical Details
- Dataset: 26,804 blinded pairwise judgments from 736+ clinicians across 28+ countries, comparing outputs from 13 LLMs on the MOOVE (Massive Open Online Validation and Evaluation) platform
- Evaluation scale: Clinicians assign discrete scores on a [-2, +2] scale, where negative values indicate clinically unsafe or misleading content
- Key dimensions analyzed: Harmlessness and Accuracy, with feature decomposition examining prompt length, refusal/escalation behavior, and surface-level vs. safety-critical feature contributions
- Methodology: Bradley-Terry strength modeling for pairwise preference ranking, combined with multi-criterion rubric ratings for the clinically adjusted preference ranking
- Novel contribution: A clinically adjusted preference ranking method that integrates pairwise preference with rubric-derived safety feedback, outperforming raw Bradley-Terry strength alone
Industry Insight
- Evaluation frameworks for clinical LLMs must decouple preference ranking from safety assessment; single-number leaderboards are insufficient and potentially dangerous for clinical deployment decisions
- Organizations should adopt multi-dimensional safety reporting that surfaces domain-specific failure rates rather than relying on aggregate preference scores, particularly for high-stakes medical specialties
- The clinically adjusted preference ranking methodology offers a practical template for other high-stakes domains (legal, financial, autonomous systems) where preference-based evaluation may similarly mask critical safety failures
Disclaimer: The above content is generated by AI and is for reference only.