摘要：采用可验证奖励的强化学习（RLVR）是训练大语言模型进行数学推理的标准方法，其中答案验证器被用作语言中立的奖励函数。我们表明，这一假设在多语言环境中并不成立：精确匹配验证器会将格式和文字系统的差异转化为与语言相关的假阴性奖励噪声。我们提出了一套可复用的多语言 RLVR 奖励审计协议，包括验证器鲁棒性测试套件、Rollout 诊断流程，以及面向日语、英语和中文答案的语言条件化奖励误差指标。在 MGSM 的 k=8 Rollout 上，针对 Qwen3-4B、Qwen3-8B 和 Llama-3.1-8B-Instruct，精确匹配代理会以因语言而显著不同的比率拒绝经可信判定为正确的答案；对于 Qwen3-8B，日语（JP）的假阴性率达到 0.642，而英语（EN）为 0.122，中文（CN）为 0.073。纯数字探针将该机制定位到最终答案接口：接口模型将奖励误差 VLB 降至零，但残余准确率差距保持不变。随后，我们揭示了一个跨语言选择瓶颈：在 MGSM250 Rollout 上，一种无需可信标签的目标语言本地聚合规则可弥合平均选择差距的 55% 至 78%，且超过 95% 的修复需要真正的跨语言支持。这一瓶颈也在包含 483 道题目的 MATH-500 数据集上得到复现。一项受控训练审计显示，rule-GRPO 能够提升可信准确率，但奖励误差 VLB 仍然较高。统一的实践结论是：在对多语言 RLVR 奖励进行优化之前，应先按语言和答案接口对其进行审计。

Abstract:Reinforcement learning with verifiable rewards (RLVR) is a standard recipe for training large language models on mathematical reasoning, where an answer verifier serves as a language-neutral reward function. We show that this assumption fails in multilingual settings: an exact-match verifier turns format and script variation into language-dependent false-negative reward noise. We introduce a reusable protocol for auditing multilingual RLVR rewards: a verifier-robustness suite, a rollout-diagnosis procedure, and language-conditioned reward-error metrics for Japanese, English, and Chinese answers. On MGSM rollouts with k=8, the exact-match proxy rejects trusted-correct answers at sharply different rates by language across Qwen3-4B, Qwen3-8B, and Llama-3.1-8B-Instruct; for Qwen3-8B, the false-negative rate reaches 0.642 on JP against 0.122 on EN and 0.073 on CN. A plain-numeric probe localizes the mechanism to the final-answer interface: an interface model drives reward-error VLB to zero while the residual accuracy gap is unchanged. We then expose a cross-lingual selection bottleneck: on MGSM250 rollouts, a target-local aggregation rule using no trusted labels closes 55-78% of the average selection gap, and over 95% of repairs require genuine cross-lingual support. The bottleneck replicates on a 483-problem MATH-500 set. A controlled training audit shows that rule-GRPO raises trusted accuracy while the reward-error VLB stays high. The unifying message is operational: multilingual RLVR rewards should be audited by language and by answer interface before they are optimized.