Large Language Models Show Metacognitive Sensitivity in Medical Reasoning
A psychophysics-inspired clinical benchmark was developed to evaluate diagnostic accuracy and confidence calibration in medical LLMs, focusing on distinguishing Alzheimer-type neurocognitive disorder (AT-NCD) from depression-related cognitive impairment (DRCI) Pilot testing with gpt-4.1-nano achieved 93.5% diagnostic accuracy, 78.4% mean confidence, and an AUROC2 of 0.876 across 135 trials from 45 synthetic vignettes The model demonstrated partial metacognitive sensitivity: confidence increased
Analysis
TL;DR
- A psychophysics-inspired clinical benchmark was developed to evaluate diagnostic accuracy and confidence calibration in medical LLMs, focusing on distinguishing Alzheimer-type neurocognitive disorder (AT-NCD) from depression-related cognitive impairment (DRCI)
- Pilot testing with gpt-4.1-nano achieved 93.5% diagnostic accuracy, 78.4% mean confidence, and an AUROC2 of 0.876 across 135 trials from 45 synthetic vignettes
- The model demonstrated partial metacognitive sensitivity: confidence increased with evidence distance from diagnostic boundaries, decreased with missing information, and remained higher on correct trials after adjustments
- Errors clustered in moderate, conflicting AT-NCD cases where the model biased toward DRCI while retaining unjustifiably high confidence, revealing localized calibration failure
- The study establishes a reproducible framework for directly measuring confidence quality rather than inferring it from benchmark accuracy or model capability alone
Why It Matters
This research addresses a critical gap in deploying LLMs for clinical decision-making: accuracy alone is insufficient when confidence does not reliably track evidence quality and uncertainty. For AI practitioners building medical diagnostic tools, the findings demonstrate that even high-accuracy models can exhibit systematic calibration failures in diagnostically ambiguous scenarios, which could have serious consequences in real-world healthcare settings.
Technical Details
- Benchmark Design: 45 synthetic clinical vignettes varying evidence strength, conflicting evidence, and missing information, each presented under three prompt variants, yielding 135 total trials
- Diagnostic Focus: Differentiation between probable AT-NCD and DRCI, two conditions with overlapping cognitive symptoms that pose genuine diagnostic challenges
- Evaluation Metrics: Diagnostic accuracy (93.5%), mean confidence (78.4%), AUROC2 (0.876), and analysis of confidence-evidence relationships including distance from diagnostic boundaries
- Model Tested: gpt-4.1-nano pilot with 100% valid structured output rate across all trials
- Key Finding: Confidence behavior was modulated by evidence strength and completeness but showed systematic overconfidence in moderate, conflicting AT-NCD cases where the model shifted toward DRCI diagnoses
Industry Insight
- AI developers should implement direct confidence calibration measurement in medical LLM pipelines rather than assuming calibration quality from accuracy benchmarks alone
- Diagnostic boundary cases—particularly moderate presentations with conflicting evidence—represent high-risk zones where models may exhibit overconfidence, requiring targeted safeguards or human-in-the-loop review
- The psychophysics-inspired benchmark framework offers a transferable methodology for evaluating metacognitive sensitivity in other specialized diagnostic domains beyond neurocognitive disorders
Disclaimer: The above content is generated by AI and is for reference only.