Anthropic 研究员让我们一窥自我改进型 AI

使用其他 AI 模型来训练 AI 模型，已成为新兴 AI 实验室的热门目标；如今，Anthropic fellows 项目的一名研究员让我们得以初步看到这种做法在实践中可能呈现的样貌。

周五，Anthropic 发布了一篇新论文，题为《自动化研究员能够可靠地缓解对齐失败问题》，详细说明了 AI 系统如何能够可靠地提升模型在一组对齐基准上的表现。在获得 10 项针对特定失准行为的基准后，这些自动化系统能够在每一项基准上提升表现，同时不降低整体性能。

该系统由 Anthropic fellow 陈岳汉（Chen Yueh-Han）领衔开发，复现了传统研究流程中的大量环节。每个自动化系统都会检索可用文献、提出一种方法，并使用该方法训练模型 30 分钟，再经过多轮迭代逐步提升基准表现。有效的方法会被保留，无效的方法则被丢弃，使系统能够以更快的速度和更大的规模运行。

“总体而言，这些结果为自动化对齐后训练在近期变得实用提供了初步证据。”论文写道。

这篇论文朝着递归式自我改进迈出了一步，许多人认为这将是 AI 进步的下一个重要阶段。如果模型能够改进自身的对齐训练，那么它们或许也能更广泛地改进训练实践——届时，人类 AI 研究员可能很快会变得多余。

论文并未回避这一设想，而是明确将自动化对齐研究员（Automated Alignment Researcher，AAR）与其人类对应角色进行比较。论文写道：

“最佳 AAR 方法平均而言能击败经验丰富的人类在六小时内提出的方案。由人类指导的研究方向不会带来更强的性能。”

为了打消仍有疑虑者的顾虑，论文还给出了成本对比：

“一名 AAR 的 API 推理成本约为每小时 4 美元，而我们支付给人类研究员的成本为每小时 150 美元。”

不过，公平地说，论文也指出了这种方法的一些局限性。自动化系统的有效性仅限于这些基准确实反映了实际的对齐目标；即便如此，建立和维护这些基准仍有大量工作要做，更不用说维护和扩展自动化研究员所依赖的文献资料。

Training AI models with other AI models has become a very popular goal for neolabs — and now, a researcher in Anthropic’s fellows program has given us an early look at what it might look like in practice.

On Friday, Anthropic published a new paper titled “Automated Researchers Can Reliably Mitigate Alignment Failures,” detailing how AI systems could reliably improve a model’s performance on a set of alignment benchmarks. When given 10 benchmarks for specific misaligned behaviors, the automated systems were able to improve performance on every single one without degrading overall performance.

Led by Anthropic fellow Chen Yueh-Han, the system replicates much of the traditional approach to research. Each automated system searches the available literature, proposes a method, and trains the model using that method for 30 minutes, gradually increasing the benchmark over several iterations. Effective methods are preserved while ineffective ones are discarded, allowing the system to operate quickly and at a great scale.

“Overall, these results provide early evidence that automated alignment post-training could become practical in the near term,” the paper reads.

The paper is a step toward recursive self-improvement, which many see as the next significant step in AI progress. If models can improve their own alignment training, it’s plausible they could improve training practices more broadly — at which point, human AI researchers might soon become obsolete.

The paper isn’t shy about addressing this idea, explicitly comparing the Automated Alignment Researcher (AAR) to its human equivalent. “The best AAR method beats what experienced humans propose, on average within six hours,” the paper reads. “Human guided research directions do not lead to stronger performance.”

There’s even a cost comparison, in case anyone wasn’t convinced. “An AAR costs roughly $4 per hour in API inference against the $150 per hour we pay our human researchers.”

In fairness, the paper also points out a few limitations to this approach. The automated system only works insofar as the benchmarks reflect the actual alignment goals, and even then there’s significant work to be done in establishing and maintaining those benchmarks — not to mention maintaining and expanding on the literature the automated researchers are drawn from.