摘要：具有可验证奖励的强化学习 (RLVR) 已成为改进语言模型数学推理的一个有前景的范式。然而大多数 RLVR 工作只奖励最终答案（基于结果的奖励），忽略了步骤级过程监督（过程奖励）的影响，尤其是对于缺乏在稀疏反馈下自我修正能力的小模型。我们在 GSM8K 上使用分组相对策略优化 (GRPO) 对 Qwen2.5-0.5B 进行微调，系统比较了五种奖励条件：无 RL 基线、仅过程、仅结果，以及三种混合权重（$\lambda \in {0.9, 0.5, 0.1}$ 过程权重）。仅过程监督的测试准确率达到 63.73%，而仅结果监督为 53.75%，相差近 10 个百分点，同时推理链具有更高的步骤有效性和更接近真实链长的偏差。混合奖励通常与过程权重正相关，但有一个异常：低过程/高结果配置（$\lambda=0.1$）表现不如纯结果监督，表明优化信号存在冲突。使用 GPT-4o 作为评判的错误分析揭示了不同的失效模式分布：过程模型生成的链结构不一致但算术基础扎实，而结果模型生成的链简洁但容易出错。我们的结果表明奖励粒度是 RLVR 的一级设计决策，过程级监督显著提高了小语言模型的准确性和推理链保真度。

Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for improving mathematical reasoning in language models. Yet most RLVR work rewards only the final answer (outcome-based rewards), leaving the impact of step-level process supervision (process rewards) underexplored especially for small models that lack the capacity to self-correct under sparse feedback. We systematically compare five reward conditions applied to Qwen2.5-0.5B fine-tuned with Group Relative Policy Optimization (GRPO) on GSM8K: a no-RL baseline, process-only, outcome-only, and three hybrid weightings ($\lambda \in \{0.9, 0.5, 0.1\}$ process weight). Process-only supervision achieves 63.73% test accuracy versus 53.75% for outcome-only, a nearly 10-percentage point gap while yielding reasoning traces with higher step validity and lower deviation from ground-truth chain length. Hybrid rewards generally correlate positively with process weight, with one notable anomaly: the low-process / high-outcome configuration ($\lambda=0.1$) underperforms pure outcome supervision, suggesting conflicting optimization signals. Error analysis using GPT-4o as a judge reveals distinct failure mode distributions: process models generate structurally inconsistent but arithmetically grounded traces, while outcome models produce concise but derivation-error-prone chains. Our results demonstrate that reward granularity is a first-order design decision for RLVR, with process-level supervision substantially improving both accuracy and trace fidelity in small language models.