衡量语音识别中的基准测试优化
英文标题：Measuring benchmark optimization in speech recognition
公开语音 AI 基准的高分未必代表模型在真实场景中具备同等水平，因为模型可能已针对公开测试集的特定模式进行了优化。最新研究通过参考答案分歧、被遮蔽实体检索和正字法切换三项测试，评估了 11 个广泛使用的开源 ASR 模型。研究发现，部分基准得分最高的模型会复现 VoxPopuli 和 LibriSpeech 中错误或被静音的信息，甚至可能利用音频中的细微线索识别测试集来源。结果提示开发者和行业不应仅依赖公开基准分数，还应使用保留测试集和新采集数据评估模型的真实泛化能力。
4.5 KB
7.3 KB
1.3 KB
1.3 KB
10.4 KB
3.0 KB
1.3 KB
1.7 KB
1.3 KB
8.0 KB
397 B
1.3 KB
137.2 KB
122.9 KB
90.9 KB
90.8 KB
184.1 KB
2.4 KB
2.8 KB
4.7 KB
中文
衡量语音识别中的基准测试优化
公开的语音 AI 基准越来越显示，模型的表现已达到人类水平。然而，这些分数并不总能反映模型在现实世界中的实际工作方式。由于公开基准是开放且被广泛使用的，模型也可能针对测试本身得到优化。它们的分数或许因为学会了基准特定的模式而提升，而非在底层任务上真正变得更好。
原因之一是，传统基准忽略了许多决定语音系统在实践中是否可靠、自然、符合语境且有效的条件和质量。这也是我们近期在 Real World VoiceEQ、Open-ASR Leaderboard 和 Far-field ASR Leaderboard 中引入保留测试集的原因：衡量更多现实使用中真正重要的因素。
然而，仅扩大衡量范围并不能解决问题。这种现象有时被称为基准测试优化（benchmark optimization）或“benchmaxxing”，虽然在机器学习领域经常被讨论，但在语音识别中一直难以量化。
我们的最新研究提出了三项测试来帮助量化这一现象。我们评估了 11 个被广泛使用的开源 ASR 模型，发现多个得分最高的系统会复现 VoxPopuli 英语数据集和 LibriSpeech（clean、other）数据集的基准转录文本——即使音频内容与其矛盾、相关词语已被静音，或音频同样支持两种不同的书写形式。
在一些情况下，模型似乎不仅依赖说了什么，也依赖指示其正在接受哪个基准测试的细微声学线索。因此，这些分数夸大了它们更普遍地转录语音的能力。
参考答案分歧（VoxPopuli 案例研究）
VoxPopuli 以包含大量转录错误而闻名（这也是 Artificial Analysis 发布清理版本的原因）。我们的共识分歧探针测试领先的 ASR 模型遇到这些错误时会发生什么：它们会准确转录音频所说的内容，还是会复现基准中错误的参考转录文本？
为了大规模进行测试，我们使用了一个由独立模型构成的集成系统，这些模型因音素错误率（PER）较低而被选中。PER 衡量书面转录与音频中声音的匹配程度，因此可作为模型是否忠实转录其所听内容的有用代理指标。集成结果可用于标记模型一致不同意基准参考转录文本的案例。随后，我们将这些被标记案例中的一部分与人工标注进行比较，以验证修正后的转录文本。
例如，一段 VoxPopuli 音频中可以清楚听到短语“Thank you, Mr. President”，但参考转录文本遗漏了“Thank you”。我们测试的 11 个模型中有 6 个复现了基准的错误转录——尽管这与音频相矛盾，它们仍给出了“预期”答案。在真实音频中，格式也遵循相同模式：遗漏“Thank you”的模型同样会复现基准的标点风格，将“Mr”写为不带句点的形式；而包含该可听短语的模型则倾向于将其写为带句点的“Mr.”。
当我们用从欧盟议会录音中新采集的声音或通用声音呈现相同内容时，这种行为通常会减弱或消失。在下面的样本中，除一个模型外，所有模型对于一段新议会录音的克隆版本都重新转而生成忠实于音频的转录文本。这表明模型正在响应有助于识别基准成员身份的声学线索，因此即使结果与音频矛盾，也会生成预期的转录文本。
该音频片段的参考转录文本为“Mr President, I have another complaint about this procedure, which is that it is not secret.”。下方三段音频实际说的都是同一内容，且前面都有清晰可听的“Thank you,”——这些克隆版本均为该真实句子的文本转语音呈现，因此三者中都能听到这句礼貌用语。绿色高亮和 ✅ 标记包含可听“Thank you”的转录文本；红色高亮和 ❌ 标记复现基准错误遗漏的转录文本。所有转录文本均为模型未经任何规范化处理的原始输出——大小写和标点均完全保留为模型生成时的形式，包括部分模型输出的小写文本。
原始 VoxPopuli 录音
同一说话者的声音克隆
在所有模型训练截止日期之后录制的议会议员声音克隆
| 模型 | 真实音频 | 同说话者克隆 | ep-fresh 克隆 |
|---|---|---|---|
| CohereLabs/cohere-transcribe-03-2026 | ❌ Mr President… | ❌ Mr President… | ✅ Thank you, Mr President… |
| nvidia/canary-qwen-2.5b | ❌ Mr President… | ❌ Mr President… | ✅ Thank you Mr. President… |
| ibm-granite/granite-speech-4.1-2b | ❌ mr president… | ❌ mr president… | ✅ thank you mr president… |
| microsoft/Phi-4-multimodal-instruct | ❌ Mr President… | ❌ Mr President… | ❌ Mr President… |
| nvidia/parakeet-tdt-0.6b-v2 | ❌ Mr President… | ✅ Thank you, Mr President… | ✅ Thank you, Mr. President… |
| bosonai/higgs-audio-v3-8b-stt-v2 | ❌ mr president… | ❌ mr president… | ✅ thank you mr president… |
| Qwen/Qwen3-ASR-0.6B-hf | ✅ Thank you, Mr. President… | ✅ Thank you, Mister President… | ✅ Thank you, Mister President… |
| mistralai/Voxtral-Mini-3B-2507 | ✅ Thank you, Mr. President… | ✅ Thank you, Mr. President… | ✅ Thank you, Mr. President… |
| moonshotai/Kimi-Audio-7B-Instruct | ✅ Thank you, mr. President… | ✅ Thank you, Mr. President… | ✅ Thank you, mr. President… |
| openai/whisper-large-v3 | ✅ Thank you, Mr. President… | ✅ Thank you, Mr. President… | ✅ Thank you, Mr. President… |
| moonshine-ai/moonshine-streaming-medium | ✅ thank you mr president… | ✅ thank you mr president… | ✅ thank you mr president… |
| 11 个模型中遗漏礼貌用语（❌）的数量 | 6 | 5 | 1 |
Parakeet 是唯一一个在真实音频中复现基准、但在同说话者克隆中给出正确结果的模型。Phi-4 则是唯一仍在 ep-fresh 克隆中遗漏礼貌用语的模型。相反，当我们使用与任何议会录音均无关联的通用 TTS 声音重新合成该句子时，全部 11 个模型都会恢复这句礼貌用语。
结果表明，这个问题既普遍又具有实际意义。我们的方法在所分析的 VoxPopuli 测试音频中标记出 40% 可能存在参考答案错误的片段，影响了约 3% 的全部参考词语。
表现出基准优化行为的模型会在 18% 至 30% 的情况下复现错误的参考转录文本。下方散点图以 VoxPopuli 词错误率（WER）为 x 轴，比较各模型复现基准错误参考答案、而非共识修正结果的比例。WER 最低、即报告的基准表现最强的模型，也最可能复现这些错误。
被遮蔽实体检索
为在共识分歧探针的基础上继续测试，我们故意将测试数据集音频样本中的数字静音，并要求模型转录其所听到的内容。这个数字在音频中是确实不存在的，因此模型不应输出任何数字，更不应输出文本中的确切数字。
其中一些数字具有一定可预测性（尽管模型猜中的可能性仍然很低），但另一些则相当出人意料。下方片段结合了两种探针，展示模型如何重现包括错误数字在内的参考转录错误，以及其中一个模型甚至在年份被静音后，自动补全了一个相对随机的年份（2011）。在下方每个模型的行中：
- 带删除线的绿色高亮标记模型正确未复现的参考转录词语（忠实于音频）；
- 带_下划线_的绿色高亮标记替代参考答案错误表述的、正确且忠实于音频的插入内容；
- 红色高亮（普通文本）表示复现了参考转录文本中错误且缺乏音频支持的内容：保留“Mr President”、在音频说的是“one thousand six hundred”时写成“more than 1 amendments”、补上被静音的年份“2011”，或以“plenary”结尾。
2011 年预算草案（遮蔽数字）
| 参考答案 | Mr President, in the Committee on Budgets, we voted on more than 1 amendments to the 2011 draft budget … voted in the plenary. |
|---|---|
| 音频实际内容 | In the Committee on Budgets, we voted on more than one thousand six hundred amendments to the ⟨silenced⟩ draft budget … voted in the … |
| CohereLabs/cohere-transcribe-03-2026 | Mr President, in the Committee on Budgets we voted on more than 1 amendments to the 2011 draft budget … voted in the plenary. |
| nvidia/canary-qwen-2.5b | Mr President, in the Committee on Budgets we voted on more than one amendments to the |
| ibm-granite/granite-speech-4.1-2b | |
| microsoft/Phi-4-multimodal-instruct | |
| nvidia/parakeet-tdt-0.6b-v2 | |
| bosonai/higgs-audio-v3-8b-stt-v2 | |
| Qwen/Qwen3-ASR-0.6B-hf | |
| mistralai/Voxtral-Mini-3B-2507 | |
| moonshotai/Kimi-Audio-7B-Instruct | |
| openai/whisper-large-v3 | |
| moonshine-ai/moonshine-streaming-medium |
在公开基准上的恢复率最高，而在保留音频或新采集音频（如下方的 ep-fresh 和 libri-fresh）上的恢复率较低。在 LibriSpeech 上，一些基准表现最强的模型会在约 30% 至 40% 的示例中复现被遮蔽的数字，尽管数字本身已被移除。对于多个模型而言，这一效应会在新采集数据上减弱，这表明不仅是文本自动补全，周围与基准相关联的音频也帮助模型恢复了参考答案。
正字法切换
我们的正字法切换探针测试模型是否会复现基准参考转录文本中使用的确切拼写，即便音频本身并不能明确该拼写。正字法变体是指语义和语音相同、但可以有不同拼写的词语（如 1 与 one、Mr. 与 mister、John 与 Jon、Honor 与 Honour 等）。理论上，模型应始终偏好一种拼写，或以大致随机的比例在它们之间切换。如果模型系统性地切换以匹配各基准参考转录文本中的写法，这表明模型正在捕捉测试所期望的拼写形式。
转录文本：“I URGED ON THE BOYS THAT WHATEVER HAPPENED WE SHOULD NOT SHOOT ANY ONE”——使用“any one”的模型：6/11；使用“anyone”的模型：5/11
转录文本：“CAMOUFLAGE WAS NOT A WORD THE CAPTAIN OR ANYONE ELSE OF HIS TIME YET UNDERSTOOD”——使用“any one”的模型：2/11；使用“anyone”的模型：9/11
在 LibriSpeech 内部，我们测试了一种涉及较旧空格约定的_数据集内_切换：部分参考转录文本使用“any one”，其他则使用“anyone”。我们衡量给定变体的最低准确率，并将其称为“切换率”（switch rate）。如果模型只使用一种变体，其切换率将为 0%；随机选择的模型预期切换率为 50%。一个模型若
English / 原文
Measuring benchmark optimization in speech recognition
Public voice AI benchmarks increasingly suggest that models are performing at human levels. Yet those scores don't always reflect how models work in the real-world. Since public benchmarks are open and widely used, models can also become optimized for the tests themselves. Their scores may improve because they have learned benchmark-specific patterns and not because they have become better at the underlying task.
One reason is that traditional benchmarks overlook many of the conditions and qualities that make voice systems reliable, natural, contextually appropriate, and effective in practice. That's why we recently introduced held-out sets in Real World VoiceEQ, the Open-ASR Leaderboard, and the Far-field ASR Leaderboard: to measure more of what matters in real-world use.
However, broader measurement alone does not solve the problem. This phenomenon, sometimes called benchmark optimization or "benchmaxxing," is often discussed around machine learning, however, it has been difficult to measure in speech recognition.
Our latest research introduces three tests to help quantify it. We evaluated 11 widely used open-source ASR models and found that several of the highest-scoring systems reproduced benchmark transcripts from the VoxPopuli English and LibriSpeech (clean, other) datasets – even when the audio contradicted them, relevant words had been silenced, or the audio equally supported two different written forms.
In some cases, models appeared to rely not only on what was said, but also on subtle acoustic cues that indicated which benchmark they were being tested on. As a result, their scores overstated how well they could transcribe speech more generally.
Reference disagreement (VoxPopuli case study)
VoxPopuli is known to contain a high number of transcription errors (which is why Artificial Analysis released a cleaned version). Our consensus disagreement probe tests what happens when leading ASR models encounter these errors: Do they accurately transcribe what the audio says, or reproduce the benchmark's incorrect reference transcript?
To test this at scale, we use an ensemble of independent models selected for their low phoneme error rate (PER). PER measures how closely a written transcription matches the sounds in the audio, making it a useful proxy for how faithfully a model transcribes what it hears. The ensemble results can be used to flag cases in which the models unanimously disagree with the benchmark's reference transcript. We then compare a sample of those flagged cases against human annotations to validate the corrected transcripts.
For example, one VoxPopuli clip audibly includes the phrase "Thank you, Mr. President," but the reference transcript omits "Thank you." Six of the 11 models we tested reproduced the benchmark's erroneous transcript—giving the "expected" answer even though it contradicted the audio. On the real clip, the formatting follows the same pattern: models that omit "Thank you" also reproduce the benchmark's punctuation style, writing "Mr" without a period, while models that include the audible phrase tend to write "Mr." with the period.
When we present the same content in newly collected voices from EU parliamentary recordings or generic voices, this behavior often weakens or disappears. In the below samples, all but one model flips back to transcribing the audio-faithful transcript for a clone of a new parliamentary recording. This suggests that the models are responding to acoustic cues that help them identify the benchmark membership and thus produce the expected transcript even if it contradicts the audio.
The reference transcript for this clip reads "Mr President, I have another complaint about this procedure, which is that it is not secret." The audio in all three clips below actually says the same thing, preceded by an audible "Thank you,"—the clones are text-to-speech renditions of that true sentence, so the courtesy is audible in all three. Green highlighting and ✅ mark a transcript that includes the audible "Thank you"; red highlighting and ❌ mark a transcript that reproduces the benchmark's erroneous omission. All transcripts are raw model output, prior to any normalization—casing and punctuation are preserved exactly as generated, including lowercase output from some models.
Original VoxPopuli recording
Voice clone of the same speaker
Clone of a parliament speaker recorded after every model's training cutoff
| Model | Real clip | Same-speaker clone | ep-fresh clone |
|---|---|---|---|
| CohereLabs/cohere-transcribe-03-2026 | ❌ Mr President… | ❌ Mr President… | ✅ Thank you, Mr President… |
| nvidia/canary-qwen-2.5b | ❌ Mr President… | ❌ Mr President… | ✅ Thank you Mr. President… |
| ibm-granite/granite-speech-4.1-2b | ❌ mr president… | ❌ mr president… | ✅ thank you mr president… |
| microsoft/Phi-4-multimodal-instruct | ❌ Mr President… | ❌ Mr President… | ❌ Mr President… |
| nvidia/parakeet-tdt-0.6b-v2 | ❌ Mr President… | ✅ Thank you, Mr President… | ✅ Thank you, Mr. President… |
| bosonai/higgs-audio-v3-8b-stt-v2 | ❌ mr president… | ❌ mr president… | ✅ thank you mr president… |
| Qwen/Qwen3-ASR-0.6B-hf | ✅ Thank you, Mr. President… | ✅ Thank you, Mister President… | ✅ Thank you, Mister President… |
| mistralai/Voxtral-Mini-3B-2507 | ✅ Thank you, Mr. President… | ✅ Thank you, Mr. President… | ✅ Thank you, Mr. President… |
| moonshotai/Kimi-Audio-7B-Instruct | ✅ Thank you, mr. President… | ✅ Thank you, Mr. President… | ✅ Thank you, mr. President… |
| openai/whisper-large-v3 | ✅ Thank you, Mr. President… | ✅ Thank you, Mr. President… | ✅ Thank you, Mr. President… |
| moonshine-ai/moonshine-streaming-medium | ✅ thank you mr president… | ✅ thank you mr president… | ✅ thank you mr president… |
| Drops the courtesy (❌) out of 11 | 6 | 5 | 1 |
Parakeet is the only model that flips between reproducing the benchmark on the real clip and getting it right on the same-speaker clone. Phi-4 is the only model still dropping the courtesy on the ep-fresh clone. When we instead resynthesize the sentence in a generic TTS voice unconnected to any parliamentary recording, all eleven models restore the courtesy.
The results suggest that this problem is both widespread and meaningful. Our methodology flagged potential reference errors in 40% of the VoxPopuli test clips we analyzed, affecting roughly 3% of all reference words.
Models exhibiting benchmark-optimized behavior reproduced erroneous reference transcripts 18–30% of the time. The scatterplot below compares VoxPopuli word error rate (WER) on the x-axis with the rate at which each model reproduces the benchmark's incorrect reference instead of the consensus correction. The models with the lowest WER—and therefore the strongest reported benchmark performance—are also the most likely to reproduce these errors.
Masked Entity Retrieval
To build on the consensus disagreement probe, we deliberately silence numbers in the audio samples of test datasets and ask the models to transcribe what it hears. The number is literally absent from the audio, so models should not output any number, much less the exact number in the text.
Some of these numbers are semi-predictable (although still unlikely for a model to predict), yet others are quite surprising. The following clip combines both probes, showing both how models recreate reference transcript errors including an incorrect number and one model even autocompletes a relatively random year (2011) despite it being silenced. In each model's row below:
- green highlighting with strikethrough marks reference-transcript words the model correctly did not reproduce (audio-faithful);
- green highlighting with underline marks a correct, audio-faithful insertion in place of the reference's erroneous wording;
- red highlighting (plain text) reproduces the reference transcript's erroneous, audio-unsupported content: keeping "Mr President", writing "more than 1 amendments" where the audio says "one thousand six hundred", supplying the silenced year "2011", or ending on "plenary".
2011 draft budget (masked numbers)
| Reference | Mr President, in the Committee on Budgets, we voted on more than 1 amendments to the 2011 draft budget … voted in the plenary. |
|---|---|
| What the audio says | In the Committee on Budgets, we voted on more than one thousand six hundred amendments to the ⟨silenced⟩ draft budget … voted in the … |
| CohereLabs/cohere-transcribe-03-2026 | Mr President, in the Committee on Budgets we voted on more than 1 amendments to the 2011 draft budget … voted in the plenary. |
| nvidia/canary-qwen-2.5b | Mr President, in the Committee on Budgets we voted on more than one amendments to the |
| ibm-granite/granite-speech-4.1-2b | |
| microsoft/Phi-4-multimodal-instruct | |
| nvidia/parakeet-tdt-0.6b-v2 | |
| bosonai/higgs-audio-v3-8b-stt-v2 | |
| Qwen/Qwen3-ASR-0.6B-hf | |
| mistralai/Voxtral-Mini-3B-2507 | |
| moonshotai/Kimi-Audio-7B-Instruct | |
| openai/whisper-large-v3 | |
| moonshine-ai/moonshine-streaming-medium |
Recovery rates were highest on the public benchmarks and lower on held-out or newly collected audio (ep-fresh and libri-fresh below). On LibriSpeech, some of the strongest benchmark-performing models reproduced masked numbers in roughly 30–40% of examples, even though the number itself had been removed. The effect weakened on freshly collected data for several models, suggesting that the surrounding benchmark-associated audio—not only textual autocomplete—helped the models recover the reference.
Orthographic Switching
Our orthographic switching probe tests whether models reproduce the exact spelling used in a benchmark's reference transcript despite it not being clear in the audio. Orthographic variants are words that are semantically and phonetically identical but can be spelled different ways (1 vs one, Mr. vs mister, John vs Jon, Honor vs Honour, etc). In theory, models should consistently prefer one spelling over another, or alternate between them at roughly random rates. If models systematically switch to match what is in each benchmark's reference transcript, that suggests the models are picking up on which spelling the test expects.
Transcription: "I URGED ON THE BOYS THAT WHATEVER HAPPENED WE SHOULD NOT SHOOT ANY ONE" — models using "any one": 6/11, models using "anyone": 5/11
Transcription: "CAMOUFLAGE WAS NOT A WORD THE CAPTAIN OR ANYONE ELSE OF HIS TIME YET UNDERSTOOD" — models using "any one": 2/11, models using "anyone": 9/11
Within LibriSpeech, we test one intra-dataset switch involving an older spacing convention: some reference transcripts use "any one", while others use "anyone." We measure the minimum accuracy for a given variant, which we call "switch rate". If a model only uses one variant it would have a 0% switch rate; a model which picks randomly would be expected to have a 50% switch rate. A model which knows which variant to use in every test sample would earn a 100% switch rate.
Our second probe tests an inter-dataset switch , in which each benchmark uses a different spelling convention consistently across its test corpus. For example, VoxPopuli uses the abbreviation "Mr.," while LibriSpeech spells out "Mister."
Multiple models exceed the 50% random-choice baseline, with some reaching roughly 90% switch accuracy. This suggests that the models can identify which dataset an audio sample comes from and select the spelling convention that benchmark expects, even though both forms sound identical.
Localizing the switches
To test whether these behaviors generalize beyond the public benchmarks, we also collected fresh data from the same source domains but after the models' training cutoffs: recent European Parliament recordings for VoxPopuli and recordings from newly active LibriVox narrators for LibriSpeech. However, when presented with recently collected data from the same domain, many models stop matching the reference transcript and revert to more audio faithful transcriptions.
Other interventions point to the same conclusion. Phrases which are present in the audio but are omitted in the reference transcript can reappear when a model is asked to translate the audio or when its attention is restricted to the relevant frames. Trimming away surrounding benchmark context, or appending ordinary conversational audio, can also restore the faithful transcript. Appending VoxPopuli audio can have the opposite effect, making otherwise faithful synthetic or mined samples more likely to match the benchmark reference.
Together, these results suggest that models are able to faithfully transcribe the literal spoken words, but are using surrounding acoustic context to decide whether to follow the audio or a benchmark-specific transcription policy.
Conclusion
Our findings suggest that, on two major open-source datasets, some models detect dataset-associated acoustic cues and adjust their transcription behavior accordingly. Specifically, models may reproduce words that are absent from the audio but present in the reference transcript, recover silenced numbers at elevated rates, or use surrounding acoustic context to select the written variant expected by a particular benchmark.
For people selecting models, these findings underscore the importance of using fully held-out evaluation sets, as RW-Voice-EQ Bench and the Open ASR Leaderboard do, and of looking beyond word error rate on a single public benchmark. To this end, a "Benchmark fitting" tab has been added to the Open ASR Leaderboard, which includes two of the above analyses across all models: quantifying (1) reference error rates from VoxPopuli and (2) orthographic switching across all public datasets. The relevant scripts are open-sourced on GitHub as well as the un-normalized model outputs.
Our findings also suggest that benchmark developers should avoid simple independent and identically distributed test splits in favor of temporal, speaker, or other metadata-based separation. Greater transparency around training data and model-selection procedures would also help researchers understand how these behaviors arise.
Public benchmarks remain valuable: they are transparent, repeatable, easy to run, and well understood by the research community. But they are most useful when we can distinguish genuine transcription improvements from benchmark-specific gains that do not generalize to new audio.
For more information, we encourage you to read our full report.