多 Agent 自监督

通过集体智能实现递归自我改进

为了超越人类积累的知识，AI 本身（即_被优化者_）或许也是目前最合适的优化者和评估者。MASS 反复优化多 Agent 工作流，然后从多 Agent 系统中提炼集体智能，将其作为递归自我改进（RSI）的自监督信号。在研究基准测试中，它的每输出 token 得分高于基础模型。我们希望 MASS 能为 AI 的“奥德赛时刻”作出贡献：在被优化者同时担任评估者、优化者同时也是被优化者的环境中持续自主学习。

预计阅读时间：15 分钟

† 该工作在 Sakana AI 实习期间完成。

图 1. 从共同工作到共同知识。 这幅版画展示了排字、印刷和校对如何构成一个协同流程。书籍将成果带出作坊，让其他人能够从中学习并在此基础上继续创造。MASS 借鉴了类似的理念：团队的共同工作可以为其成员共享的模型提供训练样本。Jan Collaert I 根据 Stradanus 的作品创作，《印刷术的发明》，约 1600 年。大都会艺术博物馆，Harris Brisbane Dick 基金，1934 年。公共领域（CC0）。

我们希望前沿 AI 模型能带领我们超越人类世代积累的知识。在这一前沿领域，即便是人类专家，也可能难以评估结果并指导进一步的发展，尤其是在不可验证的任务上。例如，评估一项拟议的发现可能需要稀缺的专业知识、代价高昂的证据，或耗时远超提出该建议的实验。届时，模型本身可能不可避免地成为最有能力的可用监督者，并需要自行找出改进的方法。这促使我们研究同质递归自我改进（RSI）：被改进的模型（被优化者）同时充当自身的优化者和评估者。核心问题是，如何在这种同质循环中获得有用的自监督信号。

人类社会为解决这一问题提供了有益的类比。从采集社群、农业聚落、工业社会到全球信息网络，人们通过分工、汇集想法和传承所学，不断拓展自身能力。集体成就成为进一步学习的起点。在《科学革命的结构》中，托马斯·库恩强调了科学家借以学习识别和解决问题的共同范例。因此，科学共同体的成就也成为其成员学习开展新工作的基础。

受这一模式启发，我们提出一个问题：模型能否从自己团队的集体智能中学习？这种集体智能是单独工作的 Agent 可能无法稳定展现出来的。

我们提出多 Agent 自监督（MASS），将这一想法转化为学习循环。MASS 使用同一个模型担任执行者、优化者和评估者。优化者首先根据评估者的反馈搜索最佳团队工作流，然后使用自评最佳团队的执行过程训练执行者。更新后的模型重新承担这三个角色，流程不断重复。因此，MASS 通过将团队的集体经验提炼进共享模型，为 RSI 提供自监督信号。

同质 RSI

我们研究的同质 RSI 中，同一个共享语言模型分别担任任务求解 Agent、工作流优化器和评估者：

Agent（见下图）

任务求解者，即_被优化者_。生成包含结果、图表和研究报告的工作区。

工作流优化器（见下图）

利用评估者的反馈，调整 Agent 分工、交换结果和相互检查工作的方式。

评估者（见下图）

比较两个工作区，选择哪个更符合任务要求，并解释判断理由。

我们使用不可验证的研究任务作为训练数据。这里的_不可验证_是指不存在基于规则的自动化信号，整个学习循环仅由自我评估信号引导，而该信号的正确性未知。

为了衡量进展，我们另外请 GPT-5.5 和 Claude Opus 4.8 评估已完成的工作。MASS 无法获得它们的判断。这为一个监督信号来自模型自身的流程提供了外部改进度量。

MASS 的工作原理

MASS 利用工作流优化激发集体智能，再将团队经验提炼进共享模型。内循环在模型权重保持不变的情况下优化团队工作流；模型自行评估并判断哪个工作流最优。随后，外循环通过学习优化后工作流的执行过程来更新这些权重。下方交互图展示了这两个循环。

← 滑动查看完整图示 →

图 2. MASS。 工作流搜索改变 Agent 的协作方式，使用选出的执行过程进行训练则将其经验带入共享权重。更新后的模型随后重新承担三个角色。该动画展示了整个流程。拖动进度条可查看动画的任意部分，包括暂停时；将鼠标悬停在组件上可查看详情，或选中组件以跳转至相应说明。正文中带下划线的术语链接至图示。

0. 准备：将工作流作为提示词（见上图）

_工作流_是一份说明多 Agent 团队应如何协作的书面计划。编排者向子 Agent分配工作并整合结果。该计划会加入任务提示词，使模型能够查看和修改团队的组织方式。它包含四个组成部分：

• **角色：**分配给每个子 Agent 的职责，例如准备数据。
• **指令：**该子 Agent 应执行的工作。
• **契约：**它必须返回的信息和产物，包括任何必要的检查。
• **步骤顺序（Hop）：**子 Agent 的调用顺序，包括可能返回较早步骤的情况。

例如，一个经过演化的金融工作流将数据准备、特征构建、建模和回测依次排列，之后进行审计。其数据契约要求明确检查数据的时间顺序：

经过演化的金融工作流，摘编自论文

text Call order Data → Features → Modeling → Backtest → Audit → Reconciliation → Report

Data specialist's output contract validation_report: no_future_leakage: bool train_end_before_val_start: bool val_end_before_test_start: bool

如果回测允许历史决策使用来自未来的信息，结果可能看起来很有说服力。该契约要求数据专家检查这类错误，并报告时间划分是否有效。它还告诉后续 Agent，在继续工作之前应获得哪些证据。因此，研究计划的一部分体现在 Agent 必须确认并相互传递的内容中。

1. 内循环：搜索更好的工作流（见上图）

对于每项训练任务，MASS 都会保留一个_冠军工作流_：即根据当前模型的判断，迄今发现的最佳工作流。每次迭代包含三个步骤：

1. **执行。**团队遵循候选工作流，生成一个工作区。
2. **评估。**评估者将该工作区与冠军工作流的工作区进行比较，并解释哪个更好。获胜的候选工作流成为新冠军。
3. **修订。**优化者利用历史反馈提出下一个工作流。

评估者评判已完成的工作；优化者利用这一判断修订产出该工作的流程。优化者的系统提示词强调契约和步骤顺序，将工作流搜索的重点引向 Agent 交换什么信息，以及何时收到这些信息。

2. 外循环：向选出的团队学习（见上图）

即便使用了优化后的工作流，长周期研究工作的质量仍可能因运行而异。因此，MASS 使用多个随机种子执行自评最优工作流（也称冠军工作流，即迄今发现的最优工作流），并使用当前模型对生成的工作区排序，保留排名最高的轨迹。每条_轨迹_都记录了编排者及其子 Agent 的消息、工具调用和工具结果。

1. 对每项任务运行冠军工作流 18 次。
2. 使用当前模型比较并排列生成的工作区。Bradley–Terry 模型将两两比较偏好转换为每个工作区的分数。
3. 每项任务最多保留 16 条轨迹：15 条用于训练，1 条用于验证。
4. 构建训练样本时，从编排者的初始提示词中移除工作流文本。
5. 使用 LoRA 对共享模型进行微调，训练数据包括编排者和子 Agent 的对话。模型输出是训练目标，任务提示词和工具结果提供上下文。

从编排者的训练提示词中移除工作流，有助于促使模型根据任务本身学习如何组织工作。

结果

我们在 qwen-code 编码 Agent 环境中，以 Qwen3.6-27B 为起点运行了两轮完整的 MASS。我们的合成研究任务集包含金融、机器人和药学领域的 12 项任务，每项任务都要求提供代码、定量结果和研究报告。其中九项任务指定用于训练，另有三项任务——每个领域各一项——在后训练阶段留出。最终有八项任务提供训练数据：一项训练任务被排除，因为工作流搜索未能找到优于其参考工作流的方案。任务描述由 GPT-5.5 在实验前生成。在 MASS 内部，当前 Qwen 模型提供工作流反馈并选择训练数据。三个测试任务未用于微调、验证和检查点选择，不过研究人员也在这些任务上评估了工作流搜索。

我们比较三个代际：基础模型 (\mathcal{L}^{(0)})、第一轮模型 (\mathcal{L}^{(1)}) 和第二轮模型 (\mathcal{L}^{(2)})。我们考察它们的工作质量、组织和评判工作的能力，以及在没有提供工作流时运用协作的情况。

更好的任务表现

在三个合成测试任务上，与基础模型相比的胜率从第一轮后的 53.9% 升至第二轮后的 69.9%（表 1）。第二轮模型在与第一轮模型的比较中也取得了 60.8% 的胜率。

评估任务	第一轮对基础模型	第二轮对基础模型	第二轮对第一轮
训练任务	65.6%	82.5%	68.1%
测试任务	53.9%	69.9%	60.8%
表 1. 合成任务表现。 外部评判者在八项训练任务和三项后训练留出任务上的胜率判断。所有模型收到的任务提示词均不包含优化后的工作流。

为了测试能否迁移到合成研究任务集之外，我们评估了公开基准。经过两轮后，在 MLR-Bench、DSBench、ScienceAgentBench 和 AstaBench 上，每输出 token 的得分达到基础模型的 1.2 至 1.6 倍（图 3）。

图 3. 公开基准表现。 每个箭头连接基础模型（蓝色）和经过两轮 MASS 后的模型（绿色）。标签以基础值的倍数表示每输出 token 得分的变化。误差线表示各次试验的标准差。转载自论文图 2；附录 C.5 给出了基准测试流程和试验次数。

提升在与训练任务相似的四项研究基准上最为明显。Terminal-Bench 2.0 的每输出 token 得分为基础值的 0.96 倍，SWE-bench Verified 为 0.94 倍，表明迁移效果仍不均衡。我们推测，在训练数据中加入软件工程任务，可以较容易地解决这一泛化问题。

观察 1：RSI 中的角色泛化

我们观察到，MASS 能够实现 RSI 中的角色泛化。**我们仅使用任务求解轨迹训练模型，却观察到工作流优化器和工作区评估者的能力也有所提升。**这使 RSI 循环能够持续进行。

图 4. 角色泛化。 后续代际更早找到成功的工作流。每条曲线都使用同一代模型担任全部三个角色。当六次外部判断全部认为某个候选工作流的输出优于未使用工作流的基础模型输出时，该任务计入统计。曲线记录截至每次迭代是否已找到一次成功结果；之后的候选方案表现仍可能更差。

图 4 显示，随着循环推进，优化器能够更快找到成功的工作流。

事实上，严格来说，图 4 并不是十分公平的比较，因为 (L^{(0)}) 表示基础模型（优化者）更新基础模型（被优化者）的情形，而 (L^{(1)}) 表示第一轮模型（优化者）更新第一轮模型（被优化者）的情形。为了进行公平比较，我们将第一轮模型的工作流交给……

Multi-Agent Self-Supervision

Recursive self-improvement through collective intelligence

To advance beyond humanity's accumulated knowledge, AI itself (the optimizee) may also be its best available optimizer and evaluator. MASS repeatedly optimizes multi-agent workflows, then distills collective intelligence from multi-agent systems and uses it as a self-supervision signal for RSI. It improves score per output token over the base model on research benchmarks. We hope MASS contributes to AI’s Odyssean moment: continual self-learning in a setting where the optimizee is the evaluator, and the optimizer is the optimizee.

Estimated reading time: 15 min

† Work done during an internship at Sakana AI.

Figure 1. From shared work to shared knowledge. This engraving shows typesetting, printing, and proofreading as parts of a coordinated process. Books carry the result beyond the workshop, where others can learn from it and build on it. MASS draws on a related idea: a team's collective work can provide training examples for the model its members share. Jan Collaert I, after Stradanus, The Invention of Book Printing , c. 1600. The Metropolitan Museum of Art, Harris Brisbane Dick Fund, 1934. Public domain (CC0).

We want frontier AI models to take us beyond the body of knowledge humanity has built over generations. At that frontier, even human experts may struggle to assess results and guide further progress, particularly on non-verifiable tasks. For example, evaluating a proposed discovery may require scarce expertise, costly evidence, or experiments that take far longer than generating the proposal. The model itself may then inevitably be the most capable supervisor available and need to figure out for itself how to improve. This motivates homogeneous recursive self-improvement (RSI) , in which the model being improved (optimizee) also serves as its own optimizer and evaluator. The central question is how to obtain useful self-supervision within this homogeneous loop.

Human societies offer a useful analogy for addressing this question. Across foraging communities, farming settlements, industrial societies, and global information networks, people have expanded their capabilities in part by dividing work, combining ideas, and passing on what they learn. Collective achievements become a starting point for further learning. In The Structure of Scientific Revolutions , Thomas Kuhn emphasized the shared examples through which scientists learn to recognize and solve problems. A scientific community's achievements thus become part of how its members learn to do new work.

Inspired by this pattern, we ask whether a model can learn from its own team's collective intelligence , which an agent working alone may not reliably exhibit.

We introduce Multi-Agent Self-Supervision (MASS) to turn this idea into a learning loop. MASS uses the same model as the executor, optimizer, and evaluator. The optimizer first searches for the best team workflow using feedback from the evaluator, then trains the executor on executions from the self-judged best team. The updated model then fills all three roles, and the process repeats. Therefore, MASS provides a self-supervision signal for RSI by distilling the team's collective experience into the shared model.

Homogeneous RSI

We study homogeneous RSI, in which one shared language model serves as the task-solving agent, workflow optimizer, and evaluator:

Agent↓ in figure

The task solver, or optimizee. Produces a workspace containing results, plots, and a research report.

Workflow optimizer↓ in figure

Uses the feedback from the evaluator to revise how agents divide the task, exchange results, and check one another's work.

Evaluator↓ in figure

Compares two workspaces, chooses which better satisfies the task, and explains the judgment.

We use non-verifiable research tasks as training data. We use non-verifiable to mean that an automatic rule-based signal is not available and that only a self-evaluation signal (whose correctness is unknown) guides the overall learning loop.

To measure progress, we separately ask GPT-5.5 and Claude Opus 4.8 to assess the completed work. Their judgments are withheld from MASS. This gives us an outside measure of improvement in a process whose supervision comes from the model itself.

How MASS works

MASS uses workflow optimization to elicit collective intelligence, then distills the team's experience into the shared model. The inner loop optimizes the team's workflow while the model weights remain fixed. The model evaluates and judges for itself which workflow is optimal. The outer loop then updates those weights by learning from executions of the optimized workflow. The interactive diagram below follows both loops.

← swipe to see the whole figure →

Figure 2. MASS. Workflow search changes how the agents coordinate. Training on selected executions carries their experience into the shared weights. The updated model then resumes all three roles. This animation illustrates the procedure. Drag the progress bar to explore any part of the animation, including while paused. Hover over a component for details, or select it to jump to the explanation. Underlined terms in the text link to the diagram.

0. Setup: workflow as a prompt↑ in figure

A workflow is a written plan for how a multi-agent team should work together. The orchestrator assigns work to subagents and integrates their results. The plan is added to the task prompt, making the team's organization available for the model to inspect and revise. It has four components:

• Role: the responsibility assigned to each subagent, such as preparing data.
• Instruction: the work that subagent should carry out.
• Contract: the information and artifacts it must return, including any required checks.
• Hop: the order of subagent calls, including any return to an earlier step.

For example, an evolved finance workflow arranges data preparation, feature construction, modeling, and backtesting into a sequence, followed by an audit. Its data contract calls for explicit checks on the timing of the data:

Evolved finance workflow, condensed from the paper

Call order
  Data → Features → Modeling → Backtest
       → Audit → Reconciliation → Report

Data specialist's output contract
  validation_report:
    no_future_leakage:           bool
    train_end_before_val_start:  bool
    val_end_before_test_start:   bool

A backtest can look convincing if it lets historical decisions use information from the future. This contract requires the data specialist to check for that error and report whether the time splits are valid. It also tells later agents what evidence they should receive before proceeding. Part of the research plan is thus expressed in what agents must establish and pass on to one another.

1. Inner loop: search for a better workflow↑ in figure

For each training task, MASS retains a champion : the best workflow found so far according to the current model's judgments. Each iteration has three steps:

1. 1Execute. The team follows the candidate workflow and produces a workspace.
2. 2Evaluate. The evaluator compares this workspace with the champion's workspace and explains which is better. A winning candidate becomes the new champion.
3. 3Revise. The optimizer uses the feedback history to propose the next workflow.

The evaluator judges the completed work; the optimizer uses that judgment to revise the process that produced it. Optimizer's system prompt emphasizes contracts and hop, directing workflow search toward what information agents exchange and when they receive it.

2. Outer loop: learn from the selected team↑ in figure

Even with an optimized workflow, the quality of long-horizon research work can vary from run to run. MASS therefore executes self-judged optimal workflows (or champion workflows, meaning the optimal workflows found so far) with multiple random seeds and uses the current model to rank the resulting workspaces, retaining the highest-ranked trajectories. Each trajectory records the messages, tool calls, and tool results of the orchestrator and its subagents.

1. 1Run the champion workflow 18 times for each task.
2. 2Use the current model to compare and rank the resulting workspacesA Bradley–Terry model converts the pairwise preferences into a score for each workspace..
3. 3Retain up to 16 trajectories per task: 15 for training and one for validation.
4. 4Remove the workflow text from the orchestrator's initial prompt when constructing the training examples.
5. 5Fine-tune the shared model with LoRA on both orchestrator and subagent conversations. The model's outputs are the training targets; task prompts and tool results provide context.

Removing the workflow from the orchestrator's training prompt encourages the model to learn how to organize the work from the task itself.

Results

We run two complete rounds of MASS starting from Qwen3.6-27B in the qwen-code coding-agent environment. Our synthetic research suite contains 12 tasks in finance, robotics, and pharmacy, each requiring code, quantitative results, and a research report. Nine tasks are designated for training and three, one per domain, are held out from post-training. Eight ultimately supply training dataOne training task is excluded because workflow search finds no workflow that beats its reference. The task descriptions were generated by GPT-5.5 before the experiment. Within MASS, the current Qwen model supplies workflow feedback and training-data selection. The three test tasks are excluded from fine-tuning, validation, and checkpoint selection, although workflow search is also evaluated on them..

We compare three generations: the base model \(\mathcal{L}^{(0)}\), the first-round model \(\mathcal{L}^{(1)}\), and the second-round model \(\mathcal{L}^{(2)}\). We examine the quality of their work, their ability to organize and judge it, and their use of coordination without a supplied workflow.

Better task performance

On the three synthetic test tasks, the win rate against the base rises from 53.9% after one round to 69.9% after two rounds (Table 1). The second-round model also wins 60.8% of comparisons against the first-round model.

Evaluation tasks	Round 1 vs. base	Round 2 vs. base	Round 2 vs. round 1
Training tasks	65.6%	82.5%	68.1%
Test tasks	53.9%	69.9%	60.8%
Table 1. Synthetic task performance. External-judge win rates across eight training tasks and three tasks held out from post-training. All models receive task prompts without an optimized workflow.

To test transfer beyond the synthetic research suite, we evaluate public benchmarks. After two rounds, score per output token reaches 1.2–1.6× the base model's level across MLR-Bench, DSBench, ScienceAgentBench, and AstaBench (Figure 3).

Figure 3. Public benchmark performance. Each arrow connects the base model (blue) to the model after two MASS rounds (green). Labels report the change in score per output token as a multiple of the base value. Error bars show standard deviations across trials. Reproduced from Figure 2 of the paper; Appendix C.5 gives benchmark protocols and trial counts.

The gains are strongest on four research benchmarks resembling the training tasks. Score per output token is 0.96× the base value on Terminal-Bench 2.0 and 0.94× on SWE-bench Verified, showing that transfer remains uneven. We suspect this generalization can be easily addressed by introducing software engineering tasks into the training data.

Observation 1. Role generalization in RSI

We observe that MASS enables role generalization in RSI. We train the model only on task-solving trajectories and observe that the capabilities of the workflow optimizer and workspace evaluator also improve. This enables RSI loop to be sustainable.

Figure 4. Role generalization. Later generations find successful workflows earlier. Each curve uses one model generation in all three roles. A task is counted once all six external judgments prefer a candidate's output to the base model's output without a workflow. The curves record whether a success has been found by each iteration; later candidates can still perform worse.

Figure 4 shows optimizer can find successful workflows sooner as cycle goes by.

In fact, Figure 4 is technically not a very fair comparison, since \( L^{(0)}\) denotes a setting in which the base model (optimizer) updates the base model (optimizee), and \( L^{(1)}\) denotes a setting in which the first-round model (optimizer) updates the first-round model (optimizee). For a fair comparison, we give the first-round model's workflows to the base model (Table 2). We observe that the base model with first-round search workflows wins 64% of comparisons against the base model with base-search workflow. Each set of workflows was discovered using its generation's executor, evaluator, and optimizer. The transfer test holds execution weights fixed during evaluation. It does not isolate the optimizer from the other roles during search. Table 2 clearly shows that optimizer's capabilty have increased.

Executor	Workflow source	Compared with	Win rate
Round 1	Round-1 search	Base executing base-search workflows	80%
Base	Round-1 search	Base executing base-search workflows	64%
Base	Round-1 search	Base without a workflow	63%
Table 2. Workflow transfer. In the highlighted comparison, the same base model executes both sets of workflows. External judges assess the resulting workspaces.

The evalutor also move closer to those of the strong external model judges (more details in paper). Agreement with those external judges rises from 73% to 93% across the three cycles. Overall, these results suggest that task-solving examples can also improve capabilities used to produce and select the next round's training data.

Ablation 1: Optimal workflow maximizes information flow

What really makes an optimal multi-agent workflow effective?

We analyze the roles, instructions, contracts, and hops of the champion workflows (the best workflows found so far). For each component, we estimate how much its text reveals about the task. In the finance example above, a role such as “careful data scientist” could apply to many tasks. A contract specifying leakage checks for a trading backtest is much more distinctiveWe estimate mutual information between embeddings of workflow components and task descriptions, using text-embedding-3-large and all-mpnet-base-v2 with unadjusted and permutation-adjusted estimators. These statistics describe workflow text, rather than messages exchanged during execution. The samples cover only 8–9 tasks; agents within a workflow are not independent task replications..

Figure 5. Task information shifts toward contracts and call order. Curves show changes from iteration 3. By iteration 11, estimated task information falls in roles and instructions and rises in contracts and hops, across both embedding models and both estimators. Intermediate changes need not be monotonic, and vertical scales differ.

Figure 5 shows that between the first and final iterations of workflow optimization, an optimal multi-agent workflow tends to encode task information more strongly in contracts and hops and less in roles and instructions. We do not show this here, but mutual dependencies among the components generally decrease too (interesting!). This pattern suggests that the LLM optimizer evolves the workflow components to be mutually independent but optimizes the information flow via contracts and hops (seethis blog post for deep, intuitive explanations). I would say that workflow optimization over discrete components resembles discovering a basis vector for that harness space.

Observation 2: Multi-agent trajectories provide a dense RSI signal

We observe that trajectories from multi-agent systems can be more effective than trajectories from single agents in RSI. Multi-agent execution also changes the structure of the training data. The orchestrator's conversation shows how to divide a broad task into focused assignments and coordinate their execution. A subagent's conversation shows how to carry out an individual assignment. Specifically, we observe that jointly learning coordination from the orchestrator and bounded execution from subagents provides more effective supervision per training token than learning from long single-agent trajectories (Figure 6).

We compare students trained independently from the same base model on single-agent trajectories (S , S +, S ++), multi-agent trajectories (M , M +), and a mixture (X). A plus sign denotes a larger training set; M + is the first-round MASS model. All points below report results pooled across four training tasks and one test task.

Figure 6. Performance per supervised training token. Multi-agent students lie above the line joining the observed single-agent results. Scores pool four shared training tasks and one test task. Token counts include supervised targets processed up to the selected checkpoint, including repeated sampling; they exclude data generation, search, and evaluation.

Against the base model, M + reaches a 68.3% win rate after processing 28.8M supervised training tokens, while S ++ reaches 64.0% after 42M tokens. The MASS model thus achieves the higher score against this reference with 31% fewer supervised tokensThe first-round dataset stores 17.2M supervised tokens. Sampling with replacement and balancing orchestrator and subagent examples produce 28.8M processed supervised tokens through the selected checkpoint. This measures exposure to training targets, including repeated examples. It excludes later updates, input tokens, validation, data generation, search, and evaluation..

Higher-quality trajectories may contribute to this advantage. In a separate analysis, strong external judges prefer the multi-agent systems' work in 85.1% of comparisons with single-agent outputs. Team runs also generate about 2.4× as many tokens on averageAmong ten pairs with similar generated trace lengths, the preference for team outputs remains 80%. This comparison matches observed lengths, not generation budgets. The groups also differ in workflow and verification instructions, so the comparison does not isolate the effect of delegation. These external judgments are used for this analysis only and do not guide MASS.. But when we compare the supervised tokens used during SFT, we find that multi-agent trajectories can provide a denser learning signal than single-agent trajectories.

These results suggest that trajectories from multi-agent systems can be more effective than trajectories from single agents in RSI, extending their value beyond test-time scaling.

Observation 3. The RSI bottleneck can be the optimizer, not the evaluator

We observed that the bottleneck in RSI can be the optimizer, not the evaluator. We kept the base model as the task solver and swapped in a stronger external model (GPT-5.5) for the evaluator only, or for both the evaluator and the optimizer.

Figure 7. Workflow-search ablations. Number of tasks where the evolved workflow beats the bare base model, by search iteration. The executor is always the base model \(\mathcal{L}^{(0)}\). Upgrading both the evaluator and optimizer to GPT-5.5 (orange, dashed) finds winning workflows for all 12 tasks within 5 iterations. Upgrading only the evaluator (green) helps much less. The fully homogeneous loop (blue) is slower but still reaches 11 of 12 tasks.

Figure 7 shows that upgrading only the evaluator helps a little, but upgrading the optimizer as well helps a lot. The strictly homogeneous self-improvement loop does eventually find winning workflows for 11 of 12 tasks; it just takes longer. Our interpretation is that the strong evaluator may provide high-quality feedback, but the weak optimizer is not capable enough to digest it. This suggests that, under a limited budget, one may have to prioritize the optimizer over the evaluator within an RSI loop unit.

Figure 8. Workflow-search stability. Win rate of the base model executing each candidate workflow against the same model without a workflow, by search iteration. Colors match the configurations in Figure 7. Strong iterations can be followed by sharp performance regressions.

Figure 8 also shows that workflow search is noisy. In text-space optimization, reaching a good workflow and improving on it stably are different problems. This suggests that the LLM optimizer may need an equivalent of a learning-rate scheduler to stabilize the search in text space.

Conclusion

In Tennyson's Ulysses , Odysseus imagines another voyage in pursuit of knowledge “Beyond the utmost bound of human thought.” For AI, we imagine an Odyssean moment when continued learning must proceed without a stronger supervisor. The challenge concerns both who can supervise further learning and how progress should be judged. Human experts may be able to state a goal without specifying an objective function that fully captures it.

Writing is a good example: expert writers can recognize strong writing, yet struggle to express their judgment in a rubric that applies reliably across contexts. For RSI, the research question is how AI can turn incomplete human guidance into useful criteria for learning, and how to test whether those criteria continue to capture the intended goal as its capabilities grow. This becomes especially challenging when the model being improved also serves as its own optimizer and evaluator: the judgments that guide improvement must themselves remain open to correction.

Figure 9. Illustration for Tennyson's Ulysses (1857). Drawing by Clarkson Stanfield; engraving by W. J. Linton. Scan by George P. Landow, The Victorian Web.

Keeping a system's self-evaluation open to correction is also essential for safety. In a homogeneous loop, a shared blind spot could allow a flawed result to be produced, accepted as an improvement, and incorporated into the next model. If humans also struggle to recognize the flaw, successive updates may reinforce it. The safety concern is that the same weakness can compromise both the system's behavior and the mechanism meant to correct it.

In this sense, shared blind spots may be an Achilles' heel of recursive self-improvement. Understanding them matters for controlling a more capable system because it reveals where reliance on the system's own judgments is least justified. That knowledge could guide independent checks, constraints on autonomous actions, and conditions for intervention. Identifying a weakness alone does not establish control; we must also show that we can detect its consequences and intervene effectively as the system changes.

OpenAI's research on chain-of-thought monitorability illustrates one part of this challenge: reasoning traces can help reveal misbehavior, but they provide incomplete evidence. Recent analysis of a possible intelligence explosion gives further reason to study these conditions early. This is a central safety question for AI's Odyssean moment: how to preserve meaningful oversight when no stronger supervisor is available. ⛵