[提交于2026年7月6日]

融合跨幕记忆与策略蒸馏的自审查强化学习（SRRL）

查看题为《融合跨幕记忆与策略蒸馏的自审查强化学习（SRRL）》的 PDF，作者：Muhammad Zain Amin 和 Kibele Sebnem Yildirim

查看 PDF HTML（实验版）

摘要：强化学习常被用来根据环境反馈训练大语言模型。在实际应用中，环境通常提供稀疏或延迟的反馈。这使得模型难以准确定位其推理过程中哪些动作导致了成功或失败。因此，从这些信号中有效学习很困难，因为模型必须判断每次失败应如何指导后续迭代中有意义的行为修正。我们引入了一个训练框架——自审查强化学习（Self-Review Reinforcement Learning），该框架在每个RL环节中嵌入一个显式的自我审查步骤。当首次响应失败时，模型生成自我审查以识别问题所在，并以此为基础进行改进的二次尝试。与 Reflexion 等仅在推理时反思的方法不同，该框架利用策略梯度优化自我审查，并通过选择性蒸馏将改进内化到基础策略中，确保它们在未来的环节中持久生效。一个跨幕记忆保存成功的自我审查，以便在训练期间遇到类似任务时复用。我们在 GSM8K 基准上，使用 GRPO 优化器，在两个语言模型（Qwen 3-4B 和 OLMo-3-7B）上评估 SRRL 与标准 RLVR 基线。SRRL 在最终奖励性能上持续优于 RLVR，并通过成功地将反馈转化为行为改进，实现了更高的学习效率。

提交历史

来自：Kibele Sebnem Yildirim [ 查看邮件 ] [v1] 2026年7月6日周一 18:24:16 UTC（3,968 KB）

[Submitted on 6 Jul 2026]

Title:Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation

View a PDF of the paper titled Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation, by Muhammad Zain Amin and Kibele Sebnem Yildirim

View PDF HTML (experimental)

Abstract:Reinforcement Learning is commonly used to train large language models using environmental feedback. In applied settings, the environment usually provides sparse or delayed feedback. This makes it difficult for the model to pinpoint which actions in its reasoning led to success or failure. So, learning effectively from these signals is hard because the model must determine how each failure should inform meaningful behavioral corrections in subsequent iterations. We introduce a training framework, Self-Review Reinforcement Learning, that embeds an explicit self-review step into each RL episode. When a first-pass response fails, the model generates a self-review to identify what went wrong, which conditions an improved second attempt. Unlike inference-time reflection approaches, such as Reflexion, the framework optimizes self-review with policy gradients and internalizes improvements into the base policy via selective distillation, ensuring they persist across future episodes. A cross-episode memory keeps successful self-reviews for reuse when encountering similar tasks in future episodes during training. We evaluate SRRL against a standard RLVR baseline using the GRPO optimizer across two language models, Qwen 3-4B and OLMo-3- 7B, on GSM8K benchmark. SRRL consistently outperforms the RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement.

Submission history

From: Kibele Sebnem Yildirim [

view email

]

[v1]

Mon, 6 Jul 2026 18:24:16 UTC (3,968 KB)