GSPO：迈向可扩展的语言模型强化学习
英文标题：GSPO: Towards Scalable Reinforcement Learning for Language Models
针对现有强化学习算法（如GRPO）在长训练中不稳定导致模型崩溃的问题，提出组序列策略优化（GSPO）算法。GSPO通过序列级重要性比率与优化，在30B MoE模型实验中训练效率显著超越GRPO，并实现随算力增加持续提升的可扩展性。该算法消除了对路由重放策略的依赖，简化了基础设施并提升了对精度差异的容忍度。GSPO已成功应用于最新Qwen3系列模型，为大规模强化学习训练奠定了算法基础，对开发者意味着更鲁棒高效的RL方案。
80.4 KB
284.6 KB
72.3 KB
157.2 KB
中文
引言#
强化学习（RL）已成为扩展语言模型并提升其深度推理与问题解决能力的关键范式。要实现RL的规模化，首要前提是保持稳定且稳健的训练动态。然而，我们观察到现有RL算法（例如GRPO）在长时间训练中暴露出严重的不稳定性问题，并导致不可逆的模型崩溃，从而阻碍了通过增加算力来进一步提升性能。
为了实现成功的RL规模化，我们提出组序列策略优化（Group Sequence Policy Optimization，GSPO）算法。与以往的RL算法不同，GSPO基于序列似然定义重要性比率，并进行序列级裁剪、奖励与优化。相较于GRPO，GSPO在以下方面展现出显著优势：
- 高性能与高效：GSPO具有明显更高的训练效率，并能通过增加训练算力实现持续的性能提升；
- 显著稳定：GSPO保持稳定的训练过程，并从根本上解决了大型混合专家（MoE）模型RL训练中的稳定性挑战；
- 基础设施友好：得益于序列级优化，GSPO在根本上对精度差异具有更高的容忍度，这为简化RL基础设施提供了吸引力巨大的潜力。
这些优势为最新Qwen3模型（Instruct、Coder、Thinking）的卓越表现奠定了基础。
序列级优化目标#
设 $x$ 为查询，$\pi_{\theta_\mathrm{old}}$ 为生成回答的旧策略，${y_i}{i=1}^G$ 为采样得到的回答组，$\widehat{A}{i}$ 为每个回答的组内相对优势，$\pi_\theta$ 为待优化的当前策略。GSPO采用如下优化目标：
$$ \mathcal{J}\text{GSPO} (\theta) =, \mathbb{E}{ x \sim \mathcal{D},, {y_i}{i=1}^G \sim \pi{\theta_\mathrm{old}}( \cdot | x) } \left[ \frac{1}{G} \sum_{i=1}^{G} \min \left( s_{i}(\theta) \widehat{A}{i}, , \mathrm{clip} \left( s{i}(\theta), 1 - {\varepsilon}, 1 + {\varepsilon} \right) \widehat{A}_{i} \right) \right], $$
其中
$$ s_{i}(\theta) = \left( \frac{ \pi_{\theta} (y_i | x) }{ \pi_{\theta_\text{old}} (y_i | x)} \right)^{\frac{1}{|y_i|}} = \exp \left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{ \pi_{\theta} (y_{i,t} | x, y_{i,<t}) }{ \pi_{\theta_\text{old}} (y_{i,t} | x,y_{i,<t})} \right). $$
这里，$s_i(\theta)$ 是GSPO中基于序列似然定义的重要性比率，我们在此进行了长度归一化，以降低方差并统一 $s_i(\theta)$ 的数值范围。
训练效率与性能#
我们基于从Qwen3-30B-A3B-Base微调而来的冷启动模型进行实验，并报告其在AIME'24、LiveCodeBench和CodeForces基准上的训练奖励曲线及性能曲线。我们以GRPO作为基线进行对比。请注意，GRPO需要借助路由重放（Routing Replay）训练策略才能使MoE模型的RL正常收敛（我们稍后讨论），而GSPO已完全不需要这一策略。
实验结果
如上图所示，GSPO展现出显著高于GRPO的训练效率，在相同的训练成本下取得了更好的性能。尤其值得注意的是，我们发现GSPO可以通过增加训练算力、定期更新查询集并延长生成长度，实现持续的性能提升——这正是我们期望算法具备的可扩展性。最终，我们成功将GSPO应用于最新Qwen3模型的大规模RL训练中，进一步释放了RL规模化的潜能！
一个有趣的观察是，GSPO中被裁剪的token比例比GRPO高出两个数量级（如下图所示），但GSPO依然实现了更高的训练效率。这进一步表明，GRPO的token级优化目标噪声较大且效率低下，而GSPO的序列级方法提供了更可靠、更有效的学习信号。
裁剪token比例
MoE强化学习与基础设施的益处#
我们发现，在采用GRPO算法时，MoE模型的专家激活波动会导致RL训练无法正常收敛。为解决这一难题，我们此前采用了**路由重放（Routing Replay）**训练策略，即缓存 $\pi_{\theta_\text{old}}$ 中激活的专家，并在计算重要性比率时将这一路由模式在 $\pi_\theta$ 中“重放”。如下图所示，路由重放对于GRPO在MoE模型上训练的正常收敛至关重要。然而，路由重放策略会引入额外的内存和通信开销，并可能限制MoE模型的实际容量。
GRPO训练MoE模型时路由重放的效果
GSPO的显著优势在于完全消除了对路由重放的依赖。其核心洞察在于，GSPO仅关注序列级似然（即 $\pi_\theta(y_i|x)$），而对单个token的似然（即 $\pi_\theta(y_{i,t}|x,y_{i,<t})$）不敏感。因此，它无需像路由重放这样对基础设施要求繁重的变通方案，既简化并稳定了训练过程，又能让模型充分发挥其容量。
此外，由于GSPO在优化中仅使用序列级似然而非token级似然，直观上前者对精度差异的容忍度要高得多。因此，GSPO使得直接使用推理引擎返回的似然进行优化成为可能，免去了使用训练引擎重新计算的必要。这在部分展开（partial rollout）、多轮RL以及训练-推理分离架构等场景下尤其有益。
结论#
我们提出了组序列策略优化（GSPO），一种用于训练语言模型的新型RL算法。GSPO在训练稳定性、效率和性能方面均显著优于GRPO，并在MoE模型的大规模RL训练中表现出特别的有效性，为最新Qwen3模型的卓越提升奠定了基础。以GSPO作为我们的算法基石，我们将继续推进RL规模化的边界，并期待由此带来的智能领域的根本性进展。
引用#
如果您觉得我们的工作有帮助，欢迎引用。
@article{gspo,
title={Group Sequence Policy Optimization},
author={
Chujie Zheng and Shixuan Liu and Mingze Li and Xiong-Hui Chen and Bowen Yu and
Chang Gao and Kai Dang and Yuqiong Liu and Rui Men and An Yang and Jingren Zhou and
Junyang Lin
},
journal={arXiv preprint arXiv:2507.18071},
year={2025}
}
English / 原文
Introduction#
Reinforcement Learning (RL) has emerged as a pivotal paradigm for scaling language models and enhancing their deep reasoning and problem-solving capabilities. To scale RL, the foremost prerequisite is maintaining stable and robust training dynamics. However, we observe that existing RL algorithms (such as GRPO) exhibit severe instability issues during long training and lead to irreversible model collapse, hindering further performance improvements with increased compute.
To enable successful RL scaling, we propose the Group Sequence Policy Optimization (GSPO) algorithm. Unlike previous RL algorithms, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. Compared to GRPO, GSPO demonstrates remarkable advantages in the following aspects:
- Performant and Efficient : GSPO possesses significantly higher training efficiency and can achieve continuous performance improvements through increasing training compute;
- Notably Stable : GSPO maintains stable training processes and inherently resolves the stability challenges in the RL training of large Mixture-of-Experts (MoE) models;
- Infrastructure-Friendly : Due to sequence-level optimization, GSPO is fundamentally more tolerant to precision discrepancies, offering attractive potential for simplifying RL infrastructure.
These merits have contributed to the exceptional performance of the latest Qwen3 models (Instruct, Coder, Thinking).
Sequence-Level Optimization Objective#
Let $x$ be a query, $\pi_{\theta_\mathrm{old}}$ be the old policy that generates responses, $\{y_i\}{i=1}^G$ be the sampled response group, $\widehat{A}{i}$ be the group relative advantage of each response, and $\pi_\theta$ be the current policy to be optimized. GSPO adopts the following optimization objective:
$$ \mathcal{J}\text{GSPO} (\theta) =, \mathbb{E}{ x \sim \mathcal{D},, \{y_i\}{i=1}^G \sim \pi{\theta_\mathrm{old}}( \cdot | x) } \left[ \frac{1}{G} \sum_{i=1}^{G} \min \left( s_{i}(\theta) \widehat{A}{i}, , \mathrm{clip} \left( s{i}(\theta), 1 - {\varepsilon}, 1 + {\varepsilon} \right) \widehat{A}_{i} \right) \right], $$
where
$$ s_{i}(\theta) = \left( \frac{ \pi_{\theta} (y_i | x) }{ \pi_{\theta_\text{old}} (y_i | x)} \right)^{\frac{1}{|y_i|}} = \exp \left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{ \pi_{\theta} (y_{i,t} | x, y_{i,<t}) }{ \pi_{\theta_\text{old}} (y_{i,t} | x,y_{i,<t})} \right). $$
Here, $s_i(\theta)$ is the importance ratio defined based on sequence likelihood in GSPO, where we perform length normalization to reduce variance and unify the numerical range of $s_i(\theta)$.
Training Efficiency and Performance#
We experiment with a cold-start model fine-tuned from Qwen3-30B-A3B-Base and report its training reward curves as well as performance curves on the AIME'24, LiveCodeBench, and CodeForces benchmarks. We compare against GRPO as the baseline. Note that GRPO necessitates the Routing Replay training strategy for the normal convergence of MoE RL (which we will discuss later), while GSPO has obviated the need for this strategy.
Experimental results
As shown in the figure above, GSPO demonstrates significantly higher training efficiency than GRPO, achieving better performance under the same training cost. Particularly, we observe that GSPO can deliver continuous performance improvement through increasing the training compute, regularly updating the query set, and extending the generation length — this is exactly the scalability we expect from an algorithm. Ultimately, we successfully applied GSPO to the large-scale RL training of the latest Qwen3 models, further unleashing the potential of RL scaling!
An interesting observation is that the fraction of tokens clipped in GSPO is two orders of magnitude higher than that in GRPO (as shown in the figure below), while GSPO still achieves higher training efficiency. This further demonstrates that GRPO’s token-level optimization objective is noisy and inefficient, while GSPO’s sequence-level approach provides a more reliable and effective learning signal.
Fractions of clipped tokens
Benefits for MoE RL and Infrastructure#
We found that when adopting the GRPO algorithm, the expert activation volatility of MoE models prevents RL training from converging properly. To address this challenge, we previously employed the Routing Replay training strategy, which caches the activated experts in $\pi_{\theta_\text{old}}$ and “replays” these routing patterns in $\pi_\theta$ when computing importance ratios. As shown in the figure below, Routing Replay is crucial for normal convergence of GRPO training on MoE models. However, the Routing Replay strategy incurs additional memory and communication overhead and may limit the actual capacity of MoE models.
Effect of Routing Replay in the GRPO training of MoE models
The notable advantage of GSPO lies in completely eliminating the dependency on Routing Replay. The key insight is that GSPO only focuses on sequence-level likelihood (i.e., $\pi_\theta(y_i|x)$) and is not sensitive to individual token likelihood (i.e., $\pi_\theta(y_{i,t}|x,y_{i,<t})$). Therefore, it does not require infrastructure-heavy workarounds like Routing Replay, both simplifying and stabilizing the training process while allowing models to maximize their capacity.
Additionally, since GSPO uses only sequence-level rather than token-level likelihoods for optimization, intuitively the former is much more tolerant of precision discrepancies. Therefore, GSPO makes it possible to directly use likelihoods returned by inference engines for optimization, eliminating the need for recomputation with training engines. This is particularly beneficial in scenarios such as partial rollout, multi-turn RL, and training-inference disaggregated frameworks.
Conclusion#
We propose Group Sequence Policy Optimization (GSPO), a new RL algorithm for training language models. GSPO demonstrates notably superior training stability, efficiency, and performance compared to GRPO and exhibits particular efficacy for the large-scale RL training of MoE models, laying the foundation for the exceptional improvements in the latest Qwen3 models. With GSPO as our algorithmic cornerstone, we will continue to push the boundaries of RL scaling and look forward to the resulting fundamental advances in intelligence.
Citation#
If you find our work helpful, feel free to give us a citation.
@article{gspo,
title={Group Sequence Policy Optimization},
author={
Chujie Zheng and Shixuan Liu and Mingze Li and Xiong-Hui Chen and Bowen Yu and
Chang Gao and Kai Dang and Yuqiong Liu and Rui Men and An Yang and Jingren Zhou and
Junyang Lin
},
journal={arXiv preprint arXiv:2507.18071},
year={2025}
}