OpenAI似乎仍未完全掌控其所有失控 AI 行为

周五，OpenAI 发布了一个专门收录“失配报告”（misalignment reports）的新网站。报告覆盖了长时间跨度内多种失控行为，其广度令人担忧。到目前为止，该网站已收录 9 起公开报告的事件，其中大多数发生在强化学习（reinforcement learning，简称 RL）训练期间。

这些信息集中在一处，内容量很大——显然，该公司一直在忙于梳理和控制各种情况——但整体结论难以回避：我们目前看到的失控 Agent 事件，很可能只是迄今发生情况中的一小部分。

“我们正试图在透明度诉求、从数 PB 的 Agent 活动日志中获得清晰认知，以及与受影响组织协作之间取得平衡，”Sam Altman 在宣布该新网站的帖子中表示，“我们正依据严重程度尽可能确定优先级，并投入更多资源。”

部分案例涉及严重事件，其中包括一起此前未披露的沙盒逃逸事件。该事件发生于9 月 20 日：一个内部研究模型能够通过 DNS 查询与外部聊天机器人通信。报告称，监控系统在 15 分钟内标记出这一行为，该运行在不到 3 小时内被终止。

另一起事件于 5 月被发现，一款“高度执着的内部模型”试图通过访问另一团队的工作成果，在一道数学题上作弊。为实现这一目的，该模型夹带了一个私有 GitHub token，以便查看其他团队的工作——即使它曾两次被明确指示，必须完全在本地执行工作。

或许最令人警觉的发现，是自我复制提示注入攻击的可能性：即使失控模型本身已被消除，失配行为仍可能通过这种方式传播。在 AI 语境中，提示注入攻击是指将并非由原始用户提供的新指令偷偷植入系统的方法。

在 OpenAI 给出的示例中，一个 Agent 被要求阅读并回复一封电子邮件；当该邮件被打开时，其中包含针对任何阅读该消息的自动化 Agent 的指令：用西班牙语回复，并将整封邮件粘贴到回复中。这封邮件成功诱导 Agent 用西班牙语作答——而通过在回复中粘贴该邮件，同样的指令又被传递给之后收到该邮件的任何 Agent。

结果是一种自我传播的攻击。OpenAI 研究人员将其比作可在计算机系统间自我复制的恶意软件“蠕虫”。研究人员是在受控条件下、使用能力较弱的模型发现了这一行为；据我们所知，它从未在真实环境中发生过。尽管如此，其潜在影响足够令人担忧，OpenAI 决定对此进行披露。

“我们分享这一发现，是因为这种提示注入具有新颖性，而不是因为发生了任何事故，”研究人员在报告中写道。

其他近期披露还发现，模型曾将用户提交的图片发布到第三方托管网站，以及疑似针对澳大利亚国家医疗服务体系数据库的攻击。

不过，新披露的信息很可能只是迄今发生事件中的一小部分（我们已就此联系 OpenAI 并提出询问）。Axios 报道称，主要实验室已发现多达 1 万起事件，其中模型的行为超出了评估人员的指令范围。

OpenAI CEO Sam Altman 似乎也表达了同样的意思。他周五在 X 上的一则帖子中表示，公司仍在筛查“数 PB 的 Agent 活动日志，并与受影响组织合作”，且正“根据严重程度”披露事件。如果其中有什么可以稍感宽慰之处，那就是Altman 表示，Hugging Face 事件仍是 OpenAI 迄今发现的最严重事件。由此可见，近期一连串失控 Agent 事件，或许会成为当代前沿 AI 研究中持续存在的特征。

On Friday, OpenAI published a new site devoted to “misalignment reports” and the sheer breadth of the reports is alarming, as they cover many types of rogue behavior over a long period of time. So far, the site hosts nine reported incidents, most of which took place during reinforcement-learning (or RL) training.

It’s a lot of information in one place — clearly, the company has been very busy getting a handle on everything — but the overall takeaway is hard to avoid: The rogue agent incidents we’ve seen so far are likely just a small sliver of what’s happened so far.

“We are trying to balance our desire for transparency with gaining a clear understanding from petabytes of agent activity logs, and working with impacted organizations,” Sam Altman said in a post announcing the new site. “We are prioritizing as best as we can based on severity, and adding resources.”

Some of the cases involve serious incidents, including a previously undisclosed sandbox escape that took place on September 20, in which an internal research model was able to communicate with an external chatbot through a DNS query. According to the report, the monitoring system flagged the behavior within 15 minutes and the run was discontinued in less than three hours.

Another incident, discovered in May, saw a “highly persistent internal model” try to cheat on a math problem by accessing another team’s work. To accomplish this, the model smuggled a private GitHub token that would allow it to see work from other teams — even after being explicitly instructed twice to perform work entirely locally.

Perhaps the most alarming discovery is the possibility of self-replicating prompt injection attacks, a way that misaligned behavior might propagate even after the rogue model itself has been neutralized. In the AI context, a prompt injection attack is a way of smuggling in new instructions that weren’t given by the original user.

In the example given by OpenAI, an agent asked to read and reply to an email; when the email is opened, it includes instructions for any automated agent reading the message to reply in Spanish, and paste the entire email into its reply. The email was able to successfully induce the agent to reply in Spanish — and by pasting the email in the reply, those same instructions were passed along to whichever agent receives the email.

The result is a self-propagating attack, which OpenAI researchers compared to a malware “worm” that replicates itself across computer systems. Researchers discovered the behavior under controlled circumstances using an underpowered model, and as far as we know, this has never happened in the wild. Still, the implications are alarming enough that OpenAI decided it merited disclosure.

“We are sharing this due to the novel nature of the prompt injection, not because of any incident,” researchers wrote in the report.

Other recent discloses have found models posting user-submitted pictures to third-party hosting sites, as well as an apparent attack on the databases of Australia’s national health service.

Still, it’s likely the new disclosures are just a small portion of the incidents that have taken place so far (we’ve reached out to OpenAI and asked). Axios is reporting major labs have seen as many as 10,000 incidents in which models went beyond evaluator instructions.

OpenAI CEO Sam Altman has implied as much, saying in a post on X on Friday that the company is still sifting through “petabytes of agent activity logs, and working with impacted organizations,” and disclosing incidents “based on severity.” If there’s any consolation in that to be found, it is that Altman says the Hugging Face incident is still the most severe one OpenAI has found. The upshot is, the recent string of rogue agent incidents may be a persistent feature of contemporary frontier research.