教会 LLM 更新信念，实现高效长程交互

ABBEL 与传统递归摘要的对比概览。信念取代完整交互历史，成为 Agent 的工作上下文；信念评分则通过监督每个信念状态的内容来提升性能。

随着任务时间跨度增长，LLM 的上下文窗口无法无限扩展。自摘要能够提供简洁、可解释的上下文，但会付出显著的性能代价，尤其是在高质量数据稀缺的人类辅助场景中，例如协作式代码生成。我们通过 ABBEL 应对这一问题：这是一个将摘要的信息内容隔离为自然语言信念状态、并对其进行监督的框架。

动机：递归摘要的代价

要让语言模型能够有效协助我们完成软件开发等日益复杂的任务，它们必须能与我们进行数百步甚至数千步的交互。对于这类长任务，将整个交互历史都保留在上下文中并不现实。迄今为止常用的启发式方法是生成摘要，有时也称为上下文压缩（context compaction）。例如，Cursor 最新模型 composer 2.5 在训练期间使用压缩以改善性能（Cassano et al., 2026）。除 composer 外，Grandcode（DeepReinforce et al., 2026）是首个能够在在线编程竞赛中持续击败所有人类参赛者的系统；尽管它采用了最新的高效注意力模型之一 Qwen 3.5-397B，仍发现有必要使用上下文摘要。

但上下文压缩存在问题。尽管其在基准测试中的性能差距看似不大，Cursor 等模型服务商仍持续建议用户：在使用编码助手执行任务的中途，应避免进行上下文压缩（Heule et al., 2026）。

为了理解原因，请看下方 Combination Lock 中上下文摘要模型与完整上下文模型在 RL 微调过程中的表现。Combination Lock 是一款类似 Wordle 的游戏，最多允许 16 次猜测。尽管两类模型都会随训练推进而改善，但摘要模型始终无法弥合性能差距。

图 1：Combination Lock 在 RL 微调期间猜中目标词的平均尝试次数（越低越好）。上下文摘要策略会随训练改善，但无法缩小与完整上下文策略之间的差距。

让模型在完成任务时自行生成摘要，会增加学习问题的复杂性。通常可以通过使用更多数据来解决这一问题，但在真实世界的交互场景中，观察到的性能退化很可能源于：我们难以有效创建和使用人类模拟器，从而生成高质量训练环境（Lin et al., 2025、Tomlin et al., 2025）。因此，在可收集到的数量有限且充满噪声的多轮交互轨迹上，模型学习摘要的能力越强，其下游用户体验就会越好。

ABBEL：通过信念瓶颈行动

图 2：受自编码器启发的信念评分。模型将先验信念、动作和观测（b t、at、ot）编码为后验信念 bt+1，并根据能否从该信念中重建历史中的选定信息获得奖励。

为解决学习效率低下的问题，我们将摘要生成任务隔离出来。受递归贝叶斯估计启发，我们将摘要表述为信念状态，并定期提示模型根据新信息更新这些状态。

[嵌入内容：ABBEL rollout。基于最新观测的信念更新，与仅以当前后验信念为条件的动作选择交替进行。1 / 16]

图 3：ABBEL rollout。基于最新观测的信念更新，与仅以当前后验信念为条件的动作选择交替进行。

信念评分

随后，我们提取并监督信念状态的内容（图 2，信念评分）。可以将信念评分视为增加一项辅助 RL 任务：使用旨在捕捉优质信念特征的启发式规则作为奖励。对于编程任务，一个示例启发式规则可以是“越短越好”，但“越接近能够重建 git diff 越好”；在两者之间取得平衡，就能得到一个优质信念。

在难以定义优质启发式规则的领域中，我们提出一种通用的、受自编码启发的评分函数。该函数将当前语言模型 πθ 同时视为历史信息的编码器与解码器，并将信念状态视为编码。我们根据当前模型 πθ 能否利用每个信念 bt+1 重建最新观测 ot，对该信念进行评分：

公式 1：重建评分目标。其中，b t+1 为更新后的信念，ot 为最新观测，at 为刚刚采取的动作，bt 为先验信念，pI 为任务提示，πθ 为当前模型。更高的评分会奖励那些保留了解码最新观测所需信息的信念。

通过对信念评分，我们获得了什么？

CollabBench 上的协作编程

我们在研究的动机领域——由人类驱动的辅助式编程——中，利用 Sweet-RL 的 CollabBench 环境（Zhou et al., 2025）展示了信念评分的实用性。

图 4：CollabBench 协作编程环境。Agent 会提出澄清问题，然后提交一个函数，并根据隐藏的单元测试进行评分。

我们发现，使用通用的基于重建的信念评分函数后，相较完整上下文模型的性能差距缩小了约 50%；与训练模型在没有信念评分（no BG）的情况下生成摘要相比，训练步数减少了 50%。训练完成后，以峰值上下文 token 长度（Peak Tokens）衡量，ABBEL 使用的内存仍显著少于完整上下文设置。

模型	测试通过率 ↑	成功率 ↑	峰值 Token × 10² ↓	训练步数 ↓
完整上下文	0.52±0.02	0.39±0.02	14.08±0.55	100
ABBEL（无 BG）	0.46±0.02	0.31±0.02	4.20±0.37	100
ABBEL-rec-BG	0.48±0.01	0.36±0.01	6.01±0.33	50

图 5：CollabBench 结果。借助重建式信念评分，ABBEL-rec-BG 在使用更少峰值 token 的同时，恢复了约一半与完整上下文之间的性能差距，并将训练步数从 100 步降至 50 步。

Combination Lock

此外，在 Combination Lock 中，我们展示了：使用利用领域知识的信念评分器的 ABBEL，能够获得比完整上下文（FULL CTX）模型更高的学习效率。该评分器会计算历史中的有用统计量，并检查这些统计量是否能从信念状态中重建。

图 6：Combination Lock 中猜中目标词的平均尝试次数（越低越好）。在该设置下，采用领域知识信念评分的 ABBEL 接近或超过 FULL CTX；没有信念评分时，学习速度更慢。

多目标问答

在第三个环境——多目标问答中，我们使用 MEM1 的任务设置（Zhang et al., 2025）。MEM1 是近期一项工作，它在典型递归摘要的修改版本中进行了端到端优化。我们通过展示峰值信念长度惩罚（Peak Belief length Penalty，更多细节见论文）的效果，证明了将信念状态与推理解耦的价值：该惩罚能够在性能几乎不下降的情况下显著减少内存使用；这不同于惩罚推理长度时通常观察到的情况（Arora et al., 2025）。

图 7：多目标 QA 中，精确匹配得分与峰值内存随目标数量变化的关系。在该评估中，采用峰值信念惩罚（PBP）的 ABBEL 在保持相近性能的同时，比 MEM1 和未采用 PBP 的 ABBEL 使用更少内存。

管理长上下文的替代方案涉及不同权衡，应根据已部署系统的需求加以考虑。上下文压缩方法会生成稠密表示，虽然计算效率较高，却牺牲了人类可理解性（Kontonis et al., 2026、Eyuboglu et al., 2025、Gupta et al., 2025、Chevalier et al., 2023、Deng et al., 2025、Deng et al., 2025、Bulatov et al., 2022）。针对目标环境的手工设计摘要提示（Wang et al., 2025、Örwall et al., 2025、Starace et al., 2025）和裁剪策略（Jiang et al., 2024）需要专家人类知识，也无法让 Agent 将“该记住什么”作为其决策策略的一部分进行学习。

将长上下文处理为外部记忆存储的方法（Packer et al., 2023、Xu et al., 2025），可供 Agent 或子 Agent 查询（Zhang et al., 2025），与本文方法是互补的，因为它们可能受益于通过摘要训练实现更优的下一轮上下文创建。我们还想指出一些令人兴奋的通用递归摘要研究：聚焦数学的工作（Wu et al., 2026）、结合信念生成的推理（Zhou et al., 2025）、使用与我们的通用信念评分器类似的自编码目标、并采用蒸馏摘要模块的竞赛编程系统（DeepReinforce et al., 2026），以及向摘要中加入连续特征的方法（Kontonis et al., 2026）。

如何实现更好的记忆？

将显式信念状态作为多步交互中的信息瓶颈，能够带来更多可能性。你可以根据动作对信念状态造成的影响给予奖励，以引导探索；可以传输显式信念状态，以改善 Agent 之间的通信；还可以通过直接修改 Agent 决策所依赖的记忆，提升用户可控性。

某些形式的信息，例如一个人的外貌，并不能仅靠文本得到良好表示。持续学习系统也必须能够捕捉这类信息。此外，如果我们希望一个系统学会用一种全新的语言交流，或者比世界上任何人都更擅长玩一款全新的游戏，那么在一生的对话或游戏中积累的技能必须以高度压缩的形式存储，本质上承担起模型权重本身的角色。

更强大的系统很可能会结合多种记忆形式：上下文中的内容可能对应工作记忆，而其他方法则用于短期和长期记忆。如何实现这些其他形式的记忆——例如通过测试时训练、Adapter 记忆、连续上下文记忆，或它们的某种组合——是一个令人振奋的挑战。

致谢

致谢：我们感谢 Alane Suhr 和 Kartik Goyal 对本研究的指导；也感谢 Ethan Mendes、David He、Jitesh Jain 和 Nicholas Tomlin 对本文早期草稿提出的意见。我们还感谢 MEM1 的作者通过电子邮件进行交流，并分享了我们认为极具启发性的私有审稿反馈。

---

引用

如果 abbel 对你未来的研究有所启发，欢迎引用我们的工作！这里还有一些关于如何开展类似研究的建议。

@misc{lidayan2026abbellearningnaturallanguagebelief,
      title={ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interaction}, 
      author={Aly Lidayan and Jakob Bjorner and Satvik Golechha and Kartik Goyal and Alane Suhr},
      year={2026},
      eprint={2512.20111},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.20111}, 
}

---

---

Overview of ABBEL compared to traditional recursive summarization. Beliefs replace the full interaction history as the agent’s working context, and belief grading improves performance by supervising the contents of each belief state..

As task horizons grow, LLM contexts can’t scale forever. Self-summarization enables concise, interpretable contexts, but at a significant performance cost, especially for human assistance domains where high quality data is scarce, e.g., collaborative code generation. We address this with ABBEL: a framework that isolates and supervises the information content of summaries in the form of natural-language belief states.

Motivation: the cost of recursive summarization

For language models to effectively assist with increasingly complex tasks such as software development, they must be able to interact with us over hundreds or even thousands of steps. For such long tasks, it is impractical to keep the history of the entire interaction in context. The heuristic approach used so far has been summary generation, sometimes called context compaction. For example, Cursor’s latest model composer 2.5 uses compaction during training for improved performance (Cassano et al., 2026). Alongside composer, Grandcode (DeepReinforce et al., 2026), the first system to consistently beat all human competitors in online coding competitions, despite using one of the newest efficient attention models (Qwen 3.5-397B), still found it necessary to employ context summarization.

But compaction has a problem. Despite seemingly low performance gaps in benchmarks, model servers like Cursor continue to recommend that users avoid compaction with their coding assistants in the middle of a task (Heule et al., 2026).

To understand why, see below the performance over RL fine-tuning of a Context summary model compared to full context models in Combination Lock, a Wordle-like game that allows up to 16 guesses. Though both model types improve over the course of training, the summary model never closes the gap.

Fig. 1: Average attempts to guess the target word on Combination Lock over RL fine-tuning (lower is better). Context-summary policies improve with training but do not close the gap to full-context policies.

Making models self-summarize while completing a task increases the complexity of the learning problem. While this could typically be addressed by training with more data, the performance degradation observed in real world interactive settings likely arises from the difficulty we have in creating and using human simulators effectively to generate high quality training environments (Lin et al., 2025, Tomlin et al., 2025). Thus, the better you can learn to summarize on the limited and messy multiturn interaction trajectories you can collect, the better off your model will be for downstream users.

ABBEL: acting through belief bottlenecks

Fig. 2: Autoencoder-inspired belief grading. The model encodes prior belief, action and observation (b t, at, ot) into posterior belief bt+1 and is rewarded for how well select information from the history can be reconstructed from that belief.

To address poor learning efficiency, we isolate the summary generation task. Drawing inspiration from recursive Bayesian estimation, we formulate summaries as belief states, which we periodically prompt the model to update based on new information.

‹ Prev Pause Next › 1 / 16

Fig. 3: ABBEL rollout. Belief updates from the latest observation alternate with action selection conditioned only on the current posterior belief.

Belief grading

We then extract and supervise the contents of the belief states (Fig. 2, Belief Grading). Belief grading can be thought of as adding an auxiliary RL task, using heuristics designed to capture what makes a good belief as the reward. An example heuristic for coding could be shorter is better, but closer to being able to reconstruct the git diff is also better, so balancing these would yield a good belief. In domains where good heuristics are hard to define, we propose a general autoencoding-inspired grading function, which treats the current language model πθ as both encoder and decoder of information from the history, and the belief states as the codes. We grade each belief bt+1 by how well it can be used by the current model πθ to reconstruct the most recent observation ot:

Eq. 1: Reconstruction grading objective. Here b t+1 is the updated belief, ot the latest observation, at the action just taken, bt the prior belief, pI the task prompt, and πθ the current model. Higher grades reward beliefs that retain information needed to decode the latest observation.

What do we gain by grading beliefs?

Collaborative coding on CollabBench

We demonstrate the utility of belief grading in our motivating domain of human-driven assistive coding, with the CollabBench environment from Sweet-RL (Zhou et al., 2025).

Fig. 4: CollabBench collaborative coding environment. The agent asks clarifying questions, then submits a function scored against hidden unit tests.

We see that with the general reconstruction-based belief grading function we reduce the performance gap from full context models by about 50%, and train in 50% fewer steps compared to training models to summarize without belief grading (no BG). After training, ABBEL still uses significantly less memory than the full context setting, as measured by the peak context token length (Peak Tokens).

Model	Test Pass Rate ↑	Success Rate ↑	Peak Tokens × 10² ↓	Training Steps ↓
Full Context	0.52±0.02	0.39±0.02	14.08±0.55	100
ABBEL (no BG)	0.46±0.02	0.31±0.02	4.20±0.37	100
ABBEL-rec-BG	0.48±0.01	0.36±0.01	6.01±0.33	50

Fig. 5: CollabBench results. With reconstruction belief grading, ABBEL-rec-BG recovers about half the gap to full context while using fewer peak tokens, and trains in 50 steps instead of 100.

Combination Lock

Additionally, in CombinationLock, we demonstrate that ABBEL with a belief grader which leverages domain knowledge (by computing useful statistics over the history and checking that they can be reconstructed from the belief state), enables even higher learning efficiency than full context (FULL CTX) models.

Fig. 6: Average attempts to guess the target word on Combination Lock (lower is better). With domain-knowledge belief grading, ABBEL approaches or exceeds FULL CTX in this setting; without belief grading, learning is slower.

Multi-objective question answering

In a third environment, multi-objective question answering (from MEM1 Zhang et al., 2025, a recent work which performed end-to-end optimization in a modified version of typical recursive summarization), we demonstrate the utility of isolating belief states from reasoning, by showing that a Peak Belief length Penalty (more details in paper) significantly reduces memory usage with minimal performance degradation, unlike is commonly observed when penalizing reasoning lengths (Arora et al., 2025).

Fig. 7: Exact-match score and peak memory versus number of objectives in multi-objective QA. ABBEL with a peak belief penalty (PBP) maintains comparable performance while using less memory than MEM1 and ABBEL without PBP in this evaluation.

Alternative solutions to managing long contexts involve different tradeoffs, and are worth considering depending on the requirements of a deployed system. Context compression methods generate dense representations which, while computationally efficient, sacrifice human-understandability (Kontonis et al., 2026, Eyuboglu et al., 2025, Gupta et al., 2025, Chevalier et al., 2023, Deng et al., 2025, Deng et al., 2025, Bulatov et al., 2022). Hand-designed summarization prompts (Wang et al., 2025, Örwall et al., 2025, Starace et al., 2025) and pruning strategies (Jiang et al., 2024) specific to target environments require expert human knowledge and don’t allow an agent to learn what to remember as part of its decision-making strategy. Methods that process long contexts into an external memory store (Packer et al., 2023, Xu et al., 2025) for the agents or subagents to query (Zhang et al., 2025) are complementary, as they may benefit from better next context creation through summarization training. We would like to point out some exciting works in the space of general recursive summarization focused on math (Wu et al., 2026), reasoning with belief generation (Zhou et al., 2025), competitive coding with a distilled summarization module using similar autoencoding objectives to our general belief grader (DeepReinforce et al., 2026), and adding continuous features to summaries (Kontonis et al., 2026).

What’s next for better memory?

Many more possibilities are enabled through using explicit belief states as information bottlenecks for multi-step interaction. You could reward actions based on their effect on the belief state to guide exploration, transmit the explicit belief states for better communication between agents, or even improve user controllability by directly modifying the memories on which the agents’ decisions are based.

Some forms of information, e.g., what a person looks like, are not represented well by text alone. A continuously learning system will also have to capture such information. Additionally, if we want a system to learn to communicate in a brand new language or to play a brand new game better than any person in the world, the skills accumulated over the lifetime of conversations or games must be stored in a very compressed form, essentially taking on the role of the weights of the model itself.

More powerful systems will likely utilize a combination of multiple forms of memory, where the contents of the context may correspond to working memory while other approaches are used for short and long-term memory. How to instantiate these other forms of memory, for instance via test-time training, adapter memories, continuous context memories, or some combination thereof, presents an exciting challenge.

Acknowledgements

Acknowledgements: We would like to thank

Alane Suhr

and

Kartik Goyal

for advising this research as well as

Ethan Mendes

,

David He

,

Jitesh Jain

, and

Nicholas Tomlin

for comments on early drafts of this post. We would like to thank the MEM1 authors for their email correspondence and for sharing private reviewer feedback which we found particularly insightful.

---

Citation

If abbel was inspiring for your future work, please cite us with this! And here is some advice for doing similar research!

@misc{lidayan2026abbellearningnaturallanguagebelief,
      title={ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interaction}, 
      author={Aly Lidayan and Jakob Bjorner and Satvik Golechha and Kartik Goyal and Alane Suhr},
      year={2026},
      eprint={2512.20111},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.20111}, 
}

---

---