MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation
MTDiag introduces a large multi-turn diagnostic dialogue dataset addressing the gap between static QA benchmarks and the interactive, incremental nature of real clinical diagnosis The dataset combines three heterogeneous sources (DDXPlus, MIMIC-IV, and published case reports from AJCR) covering both common ED presentations and long-tail rare/atypical conditions All cases are normalized into a canonical schema using UMLS concept identifiers and ICD-10 diagnosis codes, with physician-validated con
Analysis
TL;DR
- MTDiag introduces a large multi-turn diagnostic dialogue dataset addressing the gap between static QA benchmarks and the interactive, incremental nature of real clinical diagnosis
- The dataset combines three heterogeneous sources (DDXPlus, MIMIC-IV, and published case reports from AJCR) covering both common ED presentations and long-tail rare/atypical conditions
- All cases are normalized into a canonical schema using UMLS concept identifiers and ICD-10 diagnosis codes, with physician-validated conversion of structured clinical evidence into natural-language utterances
- A UserLM-8B-based utterance-generation pipeline was developed to simulate patient interactions in multi-turn diagnostic dialogues
- The authors introduce clinical knowledge-grounded metrics that evaluate LLMs as diagnostic agents beyond simple diagnostic accuracy, capturing the dynamics of multi-turn differential diagnosis
Why It Matters
This work directly addresses a critical limitation in current medical LLM evaluation: static benchmarks fail to capture the interactive, incremental nature of real clinical diagnosis, where LLMs show significant accuracy and reliability degradation in multi-turn settings. For AI practitioners building clinical decision-support systems, MTDiag provides both a realistic evaluation framework and the metrics needed to assess whether models can function as genuine diagnostic agents rather than one-shot question-answering tools.
Technical Details
- Data Sources: MTDiag aggregates cases from DDXPlus (diagnostic challenge cases), MIMIC-IV (real-world ICU electronic health records), and published case reports from the American Journal of Case Reports (AJCR), ensuring coverage of both high-prevalence and rare/atypical conditions
- Canonical Schema: All cases are normalized using UMLS concept identifiers and ICD-10 diagnosis codes, anchoring the dataset in widely-adopted medical knowledge bases for interoperability and standardization
- Utterance Generation Pipeline: A UserLM-8B-based pipeline converts structured clinical evidence into natural-language patient utterances, simulating realistic multi-turn dialogues where the model must progressively gather information
- Physician Validation: The entire dataset underwent physician validation to ensure clinical authenticity and correctness of the simulated patient interactions
- New Evaluation Metrics: The paper introduces clinical knowledge-grounded metrics specifically designed for multi-turn differential diagnosis, moving beyond binary accuracy to assess the quality of the diagnostic reasoning process across turns
Industry Insight
- The degradation of LLM performance in multi-turn clinical settings signals that current evaluation paradigms are insufficient for real-world deployment; practitioners should prioritize multi-turn interactive benchmarks when developing clinical AI systems
- The release of a physician-validated, standardized schema (UMLS + ICD-10) provides a reusable foundation for building and evaluating other clinical dialogue systems, reducing the cost of dataset creation in the space
- The introduction of process-oriented evaluation metrics (beyond accuracy) reflects a broader industry shift toward assessing reasoning quality and interaction dynamics in clinical AI, which should inform how teams design evaluation pipelines for diagnostic agents
Disclaimer: The above content is generated by AI and is for reference only.