MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
Introduction of MedRealMM, a large-scale multimodal benchmark derived from 5,620 de-identified patient-doctor interactions across 64 clinical departments in Chinese internet hospitals. Development of the Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments and convert them into standardized next-response generation tasks with text-image context. Implementation of physician-refined, case-specific rubrics that reward desirable clinical behaviors
Analysis
TL;DR
- Introduction of MedRealMM, a large-scale multimodal benchmark derived from 5,620 de-identified patient-doctor interactions across 64 clinical departments in Chinese internet hospitals.
- Development of the Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments and convert them into standardized next-response generation tasks with text-image context.
- Implementation of physician-refined, case-specific rubrics that reward desirable clinical behaviors and penalize unsafe, unsupported, or contradictory responses, moving beyond simple lexical overlap metrics.
- Evaluation of 19 general-purpose and medical-specialized LLMs reveals that while some frontier models match physicians in positive criteria, they exhibit higher rates of negative safety-critical errors.
- Demonstration that incorporating patient-uploaded medical images is critical for reliable clinical performance, highlighting a significant gap between current AI capabilities and real-world online physician standards.
Why It Matters
This benchmark addresses the critical misalignment between existing AI evaluation methods and real-world clinical practice by utilizing authentic, multimodal data rather than synthetic simulations. It provides AI practitioners with a rigorous tool to assess not just factual accuracy but also safety and clinical appropriateness in high-stakes medical contexts. For the industry, it underscores the necessity of multimodal integration and robust safety mechanisms in deploying LLMs for telemedicine applications.
Technical Details
- Data Source: 5,620 real-world cases collected from a nationwide Chinese internet hospital, covering 64 distinct clinical departments, ensuring diverse and representative clinical scenarios.
- MCCP Framework: A novel extraction method that identifies "Multimodal Clinical Challenge Points" within consultation trajectories, structuring them as next-response generation tasks while preserving prior text and image context.
- Evaluation Rubrics: Case-specific scoring criteria refined by medical professionals, designed to quantify both positive clinical behaviors and negative safety violations, offering a nuanced assessment of model output quality.
- Model Evaluation: Comprehensive testing of 19 LLMs, including both text-only and multimodal variants, assessing their ability to handle complex, real-world medical queries involving visual data.
- Key Finding: Quantitative results indicate that image modality significantly impacts performance, and current state-of-the-art models struggle with error avoidance, triggering more negative criteria than human physicians despite comparable positive scores.
Industry Insight
- Safety Over Accuracy: Developers must prioritize error avoidance and safety constraints in medical AI, as high factual accuracy alone is insufficient if the model generates unsafe or contradictory advice.
- Multimodal Necessity: Integrating visual data (e.g., medical images, photos of symptoms) is not optional but essential for achieving clinical-grade performance in online consultation systems.
- Real-World Validation: Benchmarks relying on synthetic data or multiple-choice questions may overestimate model capabilities; future evaluations should incorporate authentic, open-ended clinical interactions with expert-defined rubrics to ensure practical utility.
Disclaimer: The above content is generated by AI and is for reference only.