DAPO：字节跳动 Seed 与清华 AIR 开源的大规模 LLM 强化学习系统
英文标题：🔶 DAPO: An Open-source RL System from ByteDance Seed and Tsinghua AIR
字节跳动 Seed 与清华 AIR 开源了 DAPO 大规模 LLM 强化学习系统，覆盖算法、训练基础设施、数据集、模型权重和复现脚本。基于 Qwen2.5-32B 训练的 DAPO-Qwen-32B 在 AIME 2024 上取得 50 分以上成绩，并称以约 50% 的训练步数超过此前的 DeepSeek-R1-Zero-Qwen-32B。项目提供了 Hugging Face 模型、DAPO-Math-17k 数据集、Ray Serve 与 vLLM 推理及 AIME 2024 评测流程。对开发者和研究社区而言，该项目降低了大规模 LLM RL 训练、评测和复现的门槛，并提供了可验证的 wandb 训练记录。
1.9 KB
1.6 KB
4.5 KB
4.5 KB
907.9 KB
174.7 KB
50.2 KB
中文
重要说明
🔥 最新消息！！！
- [2025/05] 我们更新了完整 DAPO 的 wandb 训练记录，以及在 AIME 2024 上取得 50% 以上成绩的检查点。我们还提供了在 AIME 2024 上进行评测的说明。
- [2025/03] 我们在 wandb 发布了 DAPO 早期版本（不含 Token-level PG Loss 和 Dynamic Sampling）的训练记录；该版本在 AIME 2024 上取得了 44% 的成绩。
我们发布了一套完全开源的大规模 LLM RL 系统，涵盖算法、代码基础设施和数据集。该系统实现了最先进的大规模 LLM RL 性能。我们提出了 Decoupled Clip 和 Dynamic sAmpling Policy Optimization（DAPO）算法。通过开源，我们为更广泛的研究社区和社会提供可实践、可扩展的强化学习能力，让所有人都能从这些进展中受益。我们的系统基于优秀的 verl 框架构建，感谢他们出色的工作！
🤗 如果你对我们的论文有任何问题，欢迎提交 issue，我们可以在那里讨论。谢谢！
🚀 基于 Qwen2.5-32B 基座模型，DAPO 在 AIME 2024 上取得了 50 分，以 50% 的训练步数超越此前的 SOTA 模型 DeepSeek-R1-Zero-Qwen-32B。
训练期间的指标监控
-
长度的稳定性与增长：响应长度的稳定增长能够带来更充分的探索，帮助模型学习更复杂的推理行为，最终提升训练稳定性和性能。
-
奖励分数稳定性：奖励信号的稳定增长表明模型正在成功拟合训练分布，确保学习过程稳健且一致，不会出现显著波动。
-
熵与平均概率趋势：熵在初始下降后得到受控增长，可确保探索与利用之间保持健康平衡，避免过拟合或过度随机等问题，并推动模型性能持续提升。
我们提供了 DAPO-Qwen-32B 的模型权重。该模型基于 Qwen2.5-32B，并使用 DAPO 算法训练。
我们建议使用 conda 配置环境：
conda create -n dapo python=3.10
conda activate dapo
pip3 install -r requirements.txt
我们在此提供模型推理代码：
import torch
from transformers import AutoTokenizer
from vllm import SamplingParams, LLM
examples = [
{
"question": "Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\nFind the largest possible real part of \\[(75+117i)z+\\frac{96+144i}{z}\\]where $z$ is a complex number with $|z|=4$.\n\nRemember to put your answer on its own line after \"Answer:\".",
"answer": "540"
},
{
"question": "Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\nEvery morning Aya goes for a $9$-kilometer-long walk and stops at a coffee shop afterwards. When she walks at a constant speed of $s$ kilometers per hour, the walk takes her 4 hours, including $t$ minutes spent in the coffee shop. When she walks $s+2$ kilometers per hour, the walk takes her 2 hours and 24 minutes, including $t$ minutes spent in the coffee shop. Suppose Aya walks at $s+\\frac{1}{2}$ kilometers per hour. Find the number of minutes the walk takes her, including the $t$ minutes spent in the coffee shop.\n\nRemember to put your answer on its own line after \"Answer:\".",
"answer": "204"
},
{
"question": "Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\nLet $\\mathcal{B}$ be the set of rectangular boxes with surface area $54$ and volume $23$. Let $r$ be the radius of the smallest sphere that can contain each of the rectangular boxes that are elements of $\\mathcal{B}$. The value of $r^2$ can be written as $\\frac{p}{q}$, where $p$ and $q$ are relatively prime positive integers. Find $p+q$.\n\nRemember to put your answer on its own line after \"Answer:\".",
"answer": "721"
}
]
def main():
model = "BytedTsinghua-SIA/DAPO-Qwen-32B"
tokenzier = AutoTokenizer.from_pretrained(model)
llm = LLM(
model=model,
dtype=torch.bfloat16,
tensor_parallel_size=8,
gpu_memory_utilization=0.95
)
sampling_params = SamplingParams(
temperature=1.0,
top_p=0.7,
max_tokens=20480
)
for example in examples:
question = example["question"]
answer = example["answer"]
output = llm.generate(
prompts=tokenzier.apply_chat_template(conversation=[{"content": question, "role": "user"}],
add_generation_prompt=True,
tokenize=False),
sampling_params=sampling_params
)
print(f"***QUESTION***:\n{question}\n***GROUND TRUTH***:\n{answer}\n***MODEL OUTPUT***:\n{output[0].outputs[0].text}\n")
print("-"*100)
if __name__ == "__main__":
main()
如需在 AIME 2024 上评测模型，我们使用 Ray Serve 和 vLLM 部署 DAPO-Qwen-32B。
从 Hugging Face 加载模型：
serve run eval.llm:build_app model=BytedTsinghua-SIA/DAPO-Qwen-32B tensor-parallel-size=8
# open another terminal
python eval/eval_aime24.py --temperature 1.0 --top_p 0.7 --max_tokens 20480 --model BytedTsinghua-SIA/DAPO-Qwen-32B --test_file eval/aime-2024.parquet
从本地路径加载模型：
serve run eval.llm:build_app model=aaa/bbb/ccc tensor-parallel-size=8
# open another terminal
python eval/eval_aime24.py --temperature 1.0 --top_p 0.7 --max_tokens 20480 --model ccc --test_file eval/aime-2024.parquet
为了让更广泛的研究社区受益，我们完整开源了 RL 训练方案，包括算法细节、数据集和基础设施。
我们提供了用于 DAPO 训练的训练集和验证集。
训练集：DAPO-Math-17k，这是一个经过精心筛选和处理的数学数据集。验证集：AIME 2024。
我们提供了可直接使用的 DAPO 训练复现脚本。快速开始说明和核心代码见 README。这些脚本包括：
注意：
-
DAPO w/o Token-level PG Loss & Dynamic Sampling -- AIME 44脚本已在当前版本的 verl 上验证，可在 AIME 2024 上取得 44 分；其训练记录可在 wandb 查看。 -
DAPO Full -- AIME 50脚本也已在最新版 verl 上验证，在 AIME 2024 上取得 50 分。你可以在 wandb 查看对应的训练记录。
我们感谢 verl 提供出色的开源 RL 基础设施。
我们的开源实验在火山引擎机器学习平台上完成。后续我们将在火山引擎平台提供完整的复现指南，帮助用户复现实验。
English / 原文
Important
🔥 News!!!
- [2025/05] We update the wandb training record of full DAPO and the checkpoint which achieved 50%+ on AIME 2024. We also provide instructions for evaluation on AIME 2024.
- [2025/03] We release the training record of an early version of DAPO (w/o Token-level PG Loss & Dynamic Sampling), achieving 44% on AIME 2024, in wandb.
We release a fully open-sourced system for large-scale LLM RL, including algorithm, code infrastructure, and dataset. The system achieves state-of-the-art large-scale LLM RL performance. We propose the D ecoupled Clip and D ynamic sA mpling P olicy O ptimization (DAPO) algorithm. Through open-sourcing, we provide the broader research community and society with practical access to scalable reinforcement learning, enabling all to benefit from these advancements. Our system is based on the awesome verl framework. Thanks for their great work!
🤗 If you have any questions about our paper, issues are welcomed and we could discuss there. Thank you!
🚀 DAPO achieves 50 points on AIME 2024 based on the Qwen2.5-32B base model, outperforming the previous SoTA DeepSeek-R1-Zero-Qwen-32B with 50% training steps.
Metric Supervision during Training
-
Length stability and growth : The steady increase in response length allows for greater exploration, facilitating the model’s ability to learn more complex reasoning behaviors, ultimately contributing to training stability and performance improvement.
-
Reward score stability : A stable increase in the reward signal indicates that the model is successfully fitting the training distribution, ensuring that the learning process remains robust and consistent without significant fluctuations.
-
Entropy and mean probability trend : A controlled increase in entropy, after an initial decrease, ensures a healthy balance between exploration and exploitation, avoiding issues such as overfitting or excessive randomness, and promoting sustained model performance.
We provide the model weights of DAPO-Qwen-32B, which is trained based on Qwen2.5-32B using the DAPO algorithm.
We recommend using conda to setup the environment:
conda create -n dapo python=3.10
conda activate dapo
pip3 install -r requirements.txt
We provide the model inference code here:
import torch
from transformers import AutoTokenizer
from vllm import SamplingParams, LLM
examples = [
{
"question": "Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\nFind the largest possible real part of \\[(75+117i)z+\\frac{96+144i}{z}\\]where $z$ is a complex number with $|z|=4$.\n\nRemember to put your answer on its own line after \"Answer:\".",
"answer": "540"
},
{
"question": "Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\nEvery morning Aya goes for a $9$-kilometer-long walk and stops at a coffee shop afterwards. When she walks at a constant speed of $s$ kilometers per hour, the walk takes her 4 hours, including $t$ minutes spent in the coffee shop. When she walks $s+2$ kilometers per hour, the walk takes her 2 hours and 24 minutes, including $t$ minutes spent in the coffee shop. Suppose Aya walks at $s+\\frac{1}{2}$ kilometers per hour. Find the number of minutes the walk takes her, including the $t$ minutes spent in the coffee shop.\n\nRemember to put your answer on its own line after \"Answer:\".",
"answer": "204"
},
{
"question": "Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\nLet $\\mathcal{B}$ be the set of rectangular boxes with surface area $54$ and volume $23$. Let $r$ be the radius of the smallest sphere that can contain each of the rectangular boxes that are elements of $\\mathcal{B}$. The value of $r^2$ can be written as $\\frac{p}{q}$, where $p$ and $q$ are relatively prime positive integers. Find $p+q$.\n\nRemember to put your answer on its own line after \"Answer:\".",
"answer": "721"
}
]
def main():
model = "BytedTsinghua-SIA/DAPO-Qwen-32B"
tokenzier = AutoTokenizer.from_pretrained(model)
llm = LLM(
model=model,
dtype=torch.bfloat16,
tensor_parallel_size=8,
gpu_memory_utilization=0.95
)
sampling_params = SamplingParams(
temperature=1.0,
top_p=0.7,
max_tokens=20480
)
for example in examples:
question = example["question"]
answer = example["answer"]
output = llm.generate(
prompts=tokenzier.apply_chat_template(conversation=[{"content": question, "role": "user"}],
add_generation_prompt=True,
tokenize=False),
sampling_params=sampling_params
)
print(f"***QUESTION***:\n{question}\n***GROUND TRUTH***:\n{answer}\n***MODEL OUTPUT***:\n{output[0].outputs[0].text}\n")
print("-"*100)
if __name__ == "__main__":
main()
To evaluate the model on AIME 2024, we deploy DAPO-Qwen-32B with Ray Serve and vLLM.
To load the model from Huggingface:
serve run eval.llm:build_app model=BytedTsinghua-SIA/DAPO-Qwen-32B tensor-parallel-size=8
# open another terminal
python eval/eval_aime24.py --temperature 1.0 --top_p 0.7 --max_tokens 20480 --model BytedTsinghua-SIA/DAPO-Qwen-32B --test_file eval/aime-2024.parquet
To load the model from local path:
serve run eval.llm:build_app model=aaa/bbb/ccc tensor-parallel-size=8
# open another terminal
python eval/eval_aime24.py --temperature 1.0 --top_p 0.7 --max_tokens 20480 --model ccc --test_file eval/aime-2024.parquet
To benefit the broader research community, we fully open-source the recipe of our RL training, including algorithm details, dataset, and infrastructures.
We provide training and validation datasets for DAPO training.
Training: DAPO-Math-17k, a carefully curated and processed math dataset. Validation: AIME 2024.
We provide the out-of-the-box script for DAPO training reproduction. Quickstart and core code are mentioned in README. These are scripts for:
Note:
-
The
DAPO w/o Token-level PG Loss & Dynamic Sampling -- AIME 44script has been verified on the current verl and achieves 44 points on AIME 2024, whose training record can be accessed in wandb. -
The
DAPO Full -- AIME 50script has also been validated on the latest verl version. It scores 50 points on AIME 2024. You can view the corresponding training record on wandb.
We thank the verl for providing the awesome open-source RL infrastructure.
Our open-sourced experiments were conducted on the Volcano Engine Machine Learning Platform. We will provide a full reproduction guideline later on the Volcano Engine platform to help users replicate our experiments.