MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
英文标题：MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
arXiv:2607.09142v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in online medical consultation, yet existing benchmarks remain poorly aligned with real clinical practice. Many rely on synthetic conversations or patient simulators, omit patient-uploaded medical
2.1 KB
612 B
380 B
9.1 KB
32.7 KB
22.8 KB
24.2 KB
34.7 KB
中文
arXiv:2607.09142v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in online medical consultation, yet existing benchmarks remain poorly aligned with real clinical practice. Many rely on synthetic conversations or patient simulators, omit patient-uploaded medical
中文全文翻译正在补齐，可先查看下方英文原文。
English / 原文
[Submitted on 10 Jul 2026 (
), last revised 15 Jul 2026 (this version, v2)]
Title:MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
Authors:Runhan Shi
,
,
,
,
,
,
,
,
,
,
,
,
View a PDF of the paper titled MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation, by Runhan Shi and Quan Zhou and Yuqian Xu and Shuai Yang and Xin Wu and Zitong Zhou and Hui Liu and Bin Zha and Zheming Wang and Liya Li and Wei Wei and Haoyuan Hu and Jun Xu
Abstract:Large language models (LLMs) are increasingly deployed in online medical consultation, yet existing benchmarks remain poorly aligned with real clinical practice. Many rely on synthetic conversations or patient simulators, omit patient-uploaded medical images, or evaluate open-ended clinical responses using multiple-choice or lexical-overlap metrics that poorly reflect clinical quality. We introduce \textbf{MedRealMM}, a large-scale benchmark for multimodal online medical consultation built from de-identified patient-doctor interactions collected from a nationwide Chinese internet hospital. MedRealMM uses a Multimodal Clinical Challenge Point (MCCP) extraction framework to identify clinically demanding moments in authentic consultation trajectories and converts each into a standardized next-response generation task while preserving the preceding text-image context. Each instance is paired with a case-specific rubric refined by physicians that rewards clinically desirable behaviors and penalizes unsafe, unsupported, or contradictory responses. The current release contains 5,620 real-world multimodal cases spanning 64 clinical departments. We evaluate 19 general-purpose and medical-specialized LLMs, including text-only and multimodal systems. Our results show that image information is critical for reliable clinical performance and that current frontier models remain below the online physician response. Although some frontier models satisfy as many or more positive clinical criteria than physicians, they trigger more negative criteria, indicating that safety-sensitive error avoidance remains a central bottleneck. MedRealMM offers a realistic and reproducible benchmark for evaluating multimodal medical reasoning in real-world online consultation. The dataset will be publicly available on Hugging Face at this https URL.
Submission history
From: Runhan Shi [
]
Fri, 10 Jul 2026 06:52:05 UTC (5,636 KB)
[v2]
Wed, 15 Jul 2026 08:18:43 UTC (5,636 KB)