摘要：可验证奖励强化学习（Reinforcement Learning with Verifiable Rewards，RLVR）在奖励明确的问题上表现出色，例如数学和代码任务；但它是否同样适用于奖励不那么明确的场景，仍是一个开放问题。reason-over-search 方法将 RLVR 应用于开放域问答：检索过程为答案提供依据，而与参考答案的匹配则提供奖励。迄今为止，该方法已在大型模型上得到验证；对于参数量低于 10 亿的模型，则仅在借助更大教师模型蒸馏的情况下得到展示。我们在一个小模型上测试这一方法。我们使用 Group Relative Policy Optimization（GRPO）和交错式 Wikipedia 搜索工具，在 MuSiQue 上训练 Qwen3.5-0.8B；在每组 3 个随机种子的实验中，仅改变三种不同形态的奖励函数，并在每个训练检查点上，使用包含 7 个基准的问答套件进行留出评估。该方法有效：最佳实验的平均精确匹配率达到 0.352，而未经训练的基线为 0.092，提升 3.8 倍；训练循环中没有蒸馏步骤。奖励形态同样至关重要。在相同训练时长下，忠实于 Search-R1 的“仅精确匹配”奖励在每个随机种子中均为三者中最差；即使以它直接优化的精确匹配指标衡量，它仍然最差。我们得出结论：在数学和代码任务中作为 RLVR 默认方案的稀疏精确匹配奖励，并不是这一规模模型的正确起点。reason-over-search 场景能够为小模型上的 RLVR 提供合适的奖励，但小模型 RLVR 需要专门的奖励设计研究，而非简单缩小大型模型的方法配方。

Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) performs well on problems with clear rewards, such as mathematics and coding, but whether it also works where the reward is less clear remains open. The reason-over-search recipe applies RLVR to open-domain question answering, where retrieval grounds the answer and a match against the reference supplies the reward. So far it has been demonstrated on large models, and below one billion parameters only with distillation from a larger teacher. We test the recipe on a small model. We train Qwen3.5-0.8B with Group Relative Policy Optimization (GRPO) and an interleaved Wikipedia-search tool on MuSiQue, varying only the reward across three shapes over three seeds each, and we evaluate every checkpoint held-out on a seven-benchmark question-answering suite. The recipe works: the best run reaches 0.352 average exact match against a 0.092 untrained floor, a 3.8-fold gain, with no distillation step in the training loop. The reward shape also matters. The Search-R1-faithful exact-match-only reward is the worst of the three at every seed at the matched training horizon, and it is worst even on exact match, the metric it directly optimises. We conclude that the sparse exact-match reward, RLVR's default in mathematics and code, is the wrong starting point for models of this size. The reason-over-search setting can supply a suitable reward for RLVR on small models, but small-model RLVR needs its own reward-design study rather than a scaled-down copy of a large-model recipe.