摘要：经强化学习（RL）训练的大型推理模型，已越来越多地被证明在数学推理任务上优于其经监督微调（SFT）的对应模型；然而，这一优势背后的机制基础仍不明确。因此，我们提出问题：究竟是哪些内部表征差异，使 RL 模型能够实现更优的性能？本研究提供了两条相互印证的证据。首先，在逐层隐藏状态上训练的线性探针表明，相较于 SFT 模型，RL 模型在预测答案正确性方面通常具有更高准确率，这意味着其表征具有更强的线性可分性和更清晰的结构。其次，均值消融研究显示，RL 模型发展出一种层级化架构：更深层会逐步变得更为关键；而 SFT 模型则将重要性较均匀地分布在各层之间。综合来看，这些发现表明，RL 训练从根本上重构了模型表征和处理推理问题的方式。最后，我们通过分析不同问题在重复采样下的 token 数量变异性，评估模型是否会自适应地分配计算资源。尽管我们观察到部分经 RL 微调模型的变异性高于其 SFT 对应模型，但其他模型则表现出很强的一致性，这表明 token 分配可能更多取决于整体训练流水线，而不单纯是 RL 与 SFT 的区别。我们认为，这种 token 分配变异性揭示了合理的 on-policy 推理范围，能够反映哪些模型具有稳定策略，以及哪些模型呈现欠确定、可能不可识别的解题行为。

Abstract:Large reasoning models trained via reinforcement learning (RL) have been increasingly shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks; Yet the mechanistic basis for this advantage remains unclear. We therefore ask, what internal representational differences enable RL models' superior performance? Our work presents two converging lines of evidence: First, linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations. Second, mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers. Together, these findings demonstrate that RL training fundamentally restructures how models represent and process reasoning problems. Finally, we analyze token-count variability under repeated sampling across problems to assess adaptive compute allocation. While we observe higher variability in some RL-tuned models than in their SFT counterparts, we see strong consistency in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone. We believe this token-allocation variability reveals the spread of plausible on-policy reasoning, highlighting which models exhibit stable policies versus those that are under-determined, potentially non-identifiable solution behaviour.