摘要：视觉语言模型的基准测试会以某种约定提供答案选项：字母、颜色名称或像素坐标。这种约定通常被视为中性的。我们的发现表明，它并不中性；基准测试所报告的限制，可能属于读出方式，而非模型本身。
在 200 张 COCO 照片上，当以英文给出位置时，Qwen3-VL-4B 能以 68.5% 的准确率从九个位置中选出指定物体所在的正确位置；而当以像素坐标给出相同位置时，准确率仅为 20.0%。随机猜测的准确率为 11.1%。当答案选项采用坐标时会产生这一代价；相反，在问题中向模型提供一个坐标仅会带来 3.5 个百分点的损失，且不具有统计显著性。这一差距在 4x4 网格、采用 8 位量化而非 4 位量化，以及按物体尺寸、距边界距离和类别划分的每一个子集中均成立。它还会决定哪一个模型获胜。在英文名称条件下打平的两款模型，在一种坐标系统下相差 39 个百分点，在另一种坐标系统下相差 54 个百分点，且差距方向相反。
在颜色任务中，具备完成该任务能力的四个开放模型中有三个表现出这一惩罚；在照片任务中，三个开放模型中有两个如此，Gemini 也同样如此：在可解析答案上相差 11.1 个百分点（p = 1e-4）。GPT-4o 则没有出现这一现象。为检验模型是否真的能读取坐标，我们为每个坐标附上错误名称，并记录模型会遵循哪一个。以色相角书写的颜色选项，其被遵循的比例低于随机水平；标准化像素约定则被遵循至随机水平的四倍。这能够区分模型可使用和不可使用的约定，尽管它未能预测两种未经测试约定下的准确率。五个模型还会用五种不同方式命名同一个色轮，因此，固定答案词汇表在不同模型之间同样并不中性。
在这项工作中，我们曾五次因为评分器与模型对于答案形式的理解不一致，而将一个有能力的模型测量为没有能力。我们报告了每一个案例。它们正是这一现象的缩影。

Abstract:Benchmarks for vision-language models offer their answer choices in some convention: a letter, a color name, a pixel coordinate. That convention is treated as neutral. We find it is not, and that the limits a benchmark reports can belong to the readout rather than to the model.
On 200 COCO photographs, Qwen3-VL-4B picks the correct one of nine locations for a named object 68.5% of the time when the locations are given in English and 20.0% when the same locations are given as pixel coordinates. Chance is 11.1%. The cost arises when the answer options are coordinates; giving the model a coordinate in the question instead costs 3.5 points and is not significant. The gap holds on a 4x4 grid, under 8-bit rather than 4-bit quantization, and in every slice by object size, boundary distance and category. It also decides which model wins. Two models that tie under English names differ by 39 points in one coordinate system and by 54 in the other, in opposite directions.
On the color task, three of the four open models capable of the task show the penalty; on photographs, two of three open models do, and so does Gemini, at 11.1 points on parseable answers (p = 1e-4). GPT-4o does not. To ask whether a model reads a coordinate at all, we attach the wrong name to each one and record which the model follows. Color options written as hue angles are followed below chance; a normalized pixel convention is followed at four times chance. This tells apart conventions a model can use from ones it cannot, though it did not predict accuracy on two untried conventions. Five models also name the same color wheel five different ways, so a fixed answer vocabulary is not neutral across models either.
Five times during this work we measured a capable model as incapable because our scorer and the model disagreed about what an answer looks like. We report each case. They are the phenomenon in miniature.