用低资源语言思考:SFT 构建了什么、RL 修复了什么,以及准确率看不见什么
英文标题:Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See
研究团队以 Alibaba、OpenAI 和 NVIDIA 的三款前沿 MoE 模型为对象,将其微调为能够使用低资源语言进行推理;这些模型的活跃参数量均为 36 亿至 40 亿。研究发现,传统准确率基准几乎无法反映训练效果,单纯改变随机种子即可造成 7.7 个百分点的分数波动,超过了测得的所有数据和训练方案影响。SFT 可使模型在约 98% 的样本中使用提问语言进行推理,并提升语法表现,但仍存在格式遗漏、推理内容泄漏和难以遵从“用英语思考”等问题。预先注册的可验证奖励强化学习可显著修复前两类问题,并改善第三类问题,为低资源语言模型训练提供了可迁移的评估工具、控制实验和训练方法。
612 B
380 B
9.1 KB
32.7 KB
22.8 KB
24.2 KB
34.7 KB
中文
[提交于 2026 年 8 月 18 日]
标题:用低资源语言思考:SFT 构建了什么、RL 修复了什么,以及准确率看不见什么
查看 Ayoub Kirouane 及另一位作者撰写的论文《用低资源语言思考:SFT 构建了什么、RL 修复了什么,以及准确率看不见什么》PDF 版本
摘要:选取三款前沿混合专家(mixture-of-experts,MoE)模型(Alibaba、OpenAI、NVIDIA;每款均拥有 36 亿至 40 亿活跃参数),并将其微调为能够使用一种低资源语言进行推理。在准确率基准上,几乎没有任何变化;而在这一规模下,基准测试本身就充满噪声:仅改变随机种子便可使得分移动 7.7 个百分点,超过我们测得的所有数据和训练方案效应。这一无效结果是我们的第一个发现。真正的变化发生在准确率无法看见的地方。基础模型从不使用希腊语思考:即使问题以希腊语提出,在 1,000 条推理轨迹中也没有一条使用希腊语。因此,模型能够给出正确答案,却使用其用户无法阅读、审计或纠正的形式进行推理。经过监督微调(SFT)后,所有发布的检查点在约 98% 的样本中都会使用问题所用语言进行推理;其中一个模型家族的推理 token 数减少至原来的三分之一。四个模型的判定语法性均有提升,通用能力则维持在与各自基础模型相差几个百分点的范围内:没有遗忘能力,同时获得了流畅性。我们提出六个行为维度,使这类变化能够被测量;每个维度都设置了门槛,以拒绝任何与输出长度相关的指标。我们还报告了自身测量工具如何误导我们:六项失败,每一项均由一个控制实验发现。SFT 无法修复其自身缺陷:四分之一的答案跳过了指定格式,回答内容会泄漏到推理通道中,并且模型对明确的“用英语思考”指令的遵从率不足一半。训练前预先注册的、采用可验证奖励的强化学习,彻底修复了前两项问题(回退率从 24% 降至 2.5%,泄漏率从 3.5% 降至 0.0%;两者均相对于保持不变的随机奖励控制组),并改善了第三项问题(+9.1 个百分点);与此同时,希腊语推理习惯在仅以准确率为目标的梯度更新下毫发无损。我们发布了五个检查点。这些测量工具、控制实验和预注册方法可迁移至任何低资源语言;希腊语是使我们能够测量它们的案例。
提交历史
来自:Christos Petrocheilos [
]
[v1]
2026 年 8 月 18 日,周二,13:09:03 UTC(1,245 KB)
English / 原文
[Submitted on 18 Aug 2026]
Title:Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See
View a PDF of the paper titled Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See, by Ayoub Kirouane and 1 other authors
Abstract:Take three frontier mixture-of-experts models (Alibaba, OpenAI, NVIDIA; 3.6-4.0B active parameters each) and fine-tune them to reason in a low-resource language. On accuracy benchmarks almost nothing happens, and the benchmark itself is noise at this scale: changing only the random seed moves the score by 7.7 points, more than every data and recipe effect we measured. That null is our first result. The real changes live where accuracy cannot see. Base models never think in Greek: 0 of 1,000 reasoning traces, even when the question is Greek, so the model answers correctly while reasoning in a form its user cannot read, audit, or correct. After supervised fine-tuning (SFT), every released checkpoint reasons in the language of the question on ~98% of items, one family at 3x fewer tokens, with judged grammaticality improving on all four models and general ability within a few points of each base: nothing was forgotten, and fluency was gained. We propose six behavioural dimensions that make such changes measurable, each gated to reject any metric that correlates with output length, and we report how our own instruments lied: six failures, each caught by a control. What SFT cannot do is fix its own defects: a quarter of answers skip the requested format, answers leak into the reasoning channel, and an explicit "think in English" is obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training, fixes the first two outright (fallback 24% to 2.5%, leak 3.5% to 0.0%, both against a flat random-reward control) and moves the third (+9.1pp), while the Greek reasoning habit survives an accuracy-only gradient untouched. We release five checkpoints. The instruments, the controls and the pre-registration travel to any low-resource language; Greek is the case that let us measure them.
Submission history
From: Christos Petrocheilos [
]
[v1]
Tue, 18 Aug 2026 13:09:03 UTC (1,245 KB)