摘要：本文提出了一项新的基准测试 Medical-Checklist，用于评估医学多模态模型。多模态模型近期的发展已显示出其在医学视觉-语言任务领域的巨大潜力。然而，越来越明显的是，无论这些模型被应用于自然图像还是医学图像，对其性能进行评估都颇具挑战性。关键问题在于，模型能否准确理解输入图像，并将其与相关输入文本建立关联。为解决这一问题，Medical-Checklist 对模型施加了一项二元测试：向模型提供一张图像和两条描述，其中一条正确、另一条错误，模型必须选择正确的描述。错误描述中包含一个从正确描述中被不准确替换的医学概念（词语或短语）。尽管该任务较为简单，但这种简洁性使得研究者能够以统一方式评估按不同原理设计和训练的多种多模态模型。该测试还能够验证模型是否能正确理解覆盖不同医学子领域的大范围医学概念。Medical-Checklist 旨在减少数据中的潜在偏差，并支持评估模型处理分布外输入的能力，而这一能力在现有数据集中难以得到评估。在使用 Medical-Checklist 评估四个最先进的医学多模态模型时，研究发现，尽管这些模型在 Med-VQA 等特定任务中表现出色，它们可能并不能正确理解图像，这表明距离临床应用仍有很长的路要走。该数据集和代码将在论文被接收后公开。

Abstract:This paper introduces a new benchmark test, Medical-Checklist, for assessing medical multimodal models. The recent advancements in multimodal models have demonstrated significant potential in the field of medical vision-language tasks. However, it is becoming increasingly clear that evaluating these models' performance, whether they are applied to natural or medical images, is challenging. The critical question is whether the models can accurately understand an input image while associating it with relevant input text. To address this, Medical-Checklist imposes a binary test on the models: they are given an image and two captions, where one is correct and the other incorrect, and the model must select the correct one. The incorrect caption contains a single medical concept (word or phrase) that is inaccurately substituted from the correct caption. Although the task is simple, this simplicity enables the unified assessment of diverse multimodal models designed and learned on different principles. It also enables us to verify whether models correctly understand a wide range of medical concepts across various medical sub-domains. Medical-Checklist is designed to reduce potential biases in data and to enable evaluation of the models' ability to handle out-of-distribution inputs, which were difficult in existing datasets. When evaluating four state-of-the-art medical multimodal models with Medical-Checklist, it was revealed that despite their excellent performance in specific tasks such as Med-VQA, they may not correctly understand images, suggesting a long journey ahead for clinical application. The dataset and code will be made public upon acceptance.