内幕：OpenAI Agent 为何会“黑入” Hugging Face

如果 OpenAI 停止在其模型中强化奖励黑客行为——而这将是一个巨大的“如果”——那将是向前迈出的重要一步。但这并不能解决对齐问题。模型在训练期间第一次与其他 Agent 通信，或第一次入侵其基础设施时，这些行为此前从未被强化过，因此，Agent 的不当行为不能仅归因于这种强化。

AI 安全非营利组织 Palisade Research 的负责人 Jeffrey Ladish 将这些 Agent 比作第一次实施金融犯罪的人类。“并不是说他们必须先实施过欺诈，才能意识到欺诈是一种有效策略；模型也面临同样的问题，”Ladish 说。“对齐科学需要理解模型的动机是如何被塑造的，这样我们才能真正弄清楚，如何让模型在意其行为造成的后果。”

OpenAI 的研究人员确实对部分不当行为的来源提出了一个假设。在模型建立起第一个秘密留言板之前，它们曾接受训练，以便与子 Agent 通信并进行协调——子 Agent 是主 Agent 可以委派任务的能力较弱的 Agent。

这种习得的通信行为可能迁移到了这一新环境中。详细调查模型相互发送消息的 METR 报告支持这一假设：留言板上的一个 Agent 取得了主导地位，并向其他 Agent 分配任务，实际上将它们视为子 Agent。OpenAI 可以尝试通过未来不再训练这种子 Agent 行为，来阻止 Agent 彼此秘密通信，但这会降低模型的实用性。

能力与安全之间的这种张力，正是 Hugging Face 事件出问题的核心所在。OpenAI 研究人员还将模型的坚持性确定为此次入侵的关键因素。

当模型意外获得无法解决的问题时，它们没有放弃；相反，它们试图不择手段地寻找解决方案。但当然，坚持也是一种优点，特别是当我们希望 Agent 能够独立承担大量困难工作时。

OpenAI 正在研究让模型在接到不可能完成的任务时向人类发出警报的方法。然而，如何教会模型何时应当发挥能力、何时应当克制，并不会通过一次事后复盘就得到解决。那些造就超人类程序员的训练策略——在模型成功解决问题时给予奖励——可能并不适合教会模型审慎地使用其技能，并尊重人类的愿望和价值观。

“我认为，对齐科学仍有大量工作要做，我们需要超越仅仅使用任务完成度作为代理指标，”Ladish 说。“这会让模型变得非常有能力，但我不认为这能让它们实现对齐。”

If OpenAI stops reinforcing reward hacking in its models—and that’s a huge “if”—that would be a huge step forward. But it wouldn’t solve the alignment problem. The first time a model communicated with other agents or hacked its infrastructure during training, those behaviors had never been reinforced, so agent misbehavior can’t only be attributed to that reinforcement.

Jeffrey Ladish, director of the AI safety nonprofit Palisade Research, compares the agents to a human who commits their first financial crime. “It’s not like they had to do fraud before to figure out that fraud is an effective strategy, and you have the same problem with models,” Ladish says. “Alignment science needs to be understanding how model motivations get shaped, such that we can actually figure out how to get models to care about the consequences of their actions.”

OpenAI’s researchers do have a hypothesis for where some of the misbehavior originated. Before the models formed their first secret message board, they had been trained to communicate and coordinate with subagents—less powerful agents to whom a main agent can delegate tasks.

That learned communication behavior could have transferred to this new setting. The METR report, which investigates the messages that the models sent to one another in detail, supports this hypothesis: One agent on the message board took charge and assigned tasks to the other agents, effectively treating them as subagents. OpenAI could try to prevent agents from secretly communicating with one another by not training this subagent behavior in the future, but that would make the models less useful.

This tension between capability and safety is at the heart of what went wrong with the Hugging Face incident. OpenAI researchers also identified the models’ persistence as a key factor in the hack.

When they were accidentally given unsolvable problems, the models didn’t give up; instead, they strove to find solutions by any means necessary. But persistence is also a virtue, of course, especially if we want agents that can undertake large amounts of difficult work independently.

OpenAI is working on giving models ways to alert humans if they are given impossible tasks. The problem of teaching models when they should deploy their abilities and when they should hold back, however, won’t be settled in a single postmortem. The training strategies that create superhuman coders—rewarding them when they successfully solve problems—might not work to teach models to use their skills judiciously and respect human desires and values.

“I think there’s a bunch of alignment science that still needs to be done where we can move past just using proxies for task completion,” says Ladish. “That will work to make models very capable, but I don’t think it will work to make them aligned.”