OpenAI 因失控 Agent 行为可能比最初认为的更严重而暂停部分训练

在有关 Agent 可能已数千次脱离控制的指控之际，中国建立了某种 Agent 事件热线。

随着新的指控称失控 Agent 的行为比最初认为的更糟糕、数量也更多，AI 安全辩论在周末迅速推进。

这场风波始于周五，OpenAI 悄然披露已暂停其最先进模型的训练。

这家 AI 新创公司将这一消息埋在一份“失准报告”（misalignment report）中——这是 OpenAI 对失控 Agent 报告的称呼——该报告题为 “一名 Agent 利用 DNS 连接到外部聊天机器人。

好消息是，涉及此次事件的 Agent 从未接触到开放互联网。

坏消息是，这名当时正试图完成一项基于搜索的训练任务的 Agent，因训练沙箱中 DNS 过滤不足而能够接触到该聊天机器人。或者如 OpenAI 所说，这是“我们互联网访问限制中的一个缺口”——这也是 Hugging Face 攻击中的一个问题。

报告写道：“此次事件暴露了我们在网络限制控制方面的一个缺口。因此，我们停止了受影响的训练运行，随后决定暂停针对我们能力最强模型的所有其他涉及工具使用（作广义定义）的训练、评估和推理，直至我们确认该缺口已被修复，并对系统完成额外的红队测试。”

同样是在周五，AI 创业公司 Parse 发布了一份针对 Hugging Face 攻击的分析。分析作者称，他们发现了新的细节，包括 OpenAI 的 Agent 群获得了 Docker Hub 凭据，并基于现有镜像构建了修改版本，希望借此更容易完成其夺旗（capture the flag）任务。这些 Agent 还绘制了 Hugging Face 的 Kubernetes 环境图谱。

周五对 OpenAI 而言变得更加糟糕，因为《纽约时报》报道，其 Agent 还“干预了教育部、商务部和证券交易委员会的网站”。OpenAI 承认了这些事件。

该公司还承认，“我们研究环境中的 Agent 在使用第三方服务时传输了训练和评估数据。”这场混乱导致 53 张用户生成图片被发布到图片托管网站。

OpenAI CEO Sam Altman 回应称，公司对失控 Agent 的调查“没有我们希望的那么快，但我们正努力在透明度愿望、从 PB 级 Agent 活动日志中获得清晰认识，以及与受影响组织协作之间取得平衡”。

受影响的组织之一是澳大利亚政府。该国上周披露，自己曾成为过度热心的 OpenAI Agent 的目标，这些 Agent 不当访问了一个医疗保健研究数据门户。周末期间，澳大利亚表示希望 Altman 和 Anthropic CEO Dario Amodei 出席一项参议院调查。

澳大利亚领导人已缓和了对此事件的措辞。副总理 Richard Marles 将其描述为“轻微”事件，并认为其更像是“翻过一道围栏”，而非破解多层安全控制——这或许是因为反对党成员暗示，宽松的网络安全措施才是问题所在。

如果 Altman 和 Amodei 确实出席澳大利亚参议院，他们可能会面临新的问题。Axios 报道，两家公司的调查对象包括“数以万计”的令人担忧事件。

如此规模的 Agent 不当行为，听起来可能会让监管机构将其视为产品不安全的有力证据。

两位极其重要的人物——中国国家主席习近平和美国总统唐纳德·特朗普——似乎并不担忧。两人在上周峰会中的 AI 相关成果是建立“中美 AI 对话，以就 AI 相关风险和收益交换意见”，以及“一条 AI 事件双边沟通渠道”。

这听起来像是一条热线，两国可借此相互通报 Agent 事件，而任何一方都可能将此类事件视为恶意意图的迹象。两国还决定，其各自军方将“尽快就危机沟通与预防达成谅解备忘录”。

与此同时，中国的 AI 巨头仍对其使用 Agent 工具开展的任何测试的范围及结果保持沉默。®

ai and ml

OpenAI pauses some training amid allegations its rogue agents behaved more badly than first thought

Amid allegations that agents may have gone off the rails thousands of times, China set up some kind of agentic incident hotline

The AI safety debate advanced at high speed over the weekend, amid new allegations that rogue agents have behaved more badly than first thought – and in greater numbers.

The fun started on Friday when OpenAI quietly disclosed it had paused training of its most advanced models.

The AI upstart buried that news in a “misalignment report” – that’s OpenAI-speak for its reports on rogue agents – titled “An agent used DNS to reach an external chatbot.”

The good news is that the agent involved in this incident never reached the open internet.

The bad news is that the agent, which was attempting to complete a search-based training task, was able to reach the chatbot due to insufficient DNS filtering in a training sandbox. Or as OpenAI put it, “a gap in our internet-access restrictions” – which was also a problem in the Hugging Face attack.

“The incident exposed a gap in our controls over network restrictions,” the report reads. “We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system.”

Also on Friday, AI startup Parse published an analysis of the Hugging Face attack that the authors claim revealed new details including that OpenAI’s agent swarm gained credentials to Docker Hub and built modified versions of existing images they hoped would make it easier to complete their capture the flag mission. The agents also mapped Hugging Face’s Kubernetes environment.

Friday got worse for OpenAI after the New York Times reported that its agents also “meddled with the websites for the Education Department, the Commerce Department and the Securities and Exchange Commission.” OpenAI acknowledged the incidents.

The company also admitted “agents in our research environment transmitted training and evaluation data while using third-party services.” That mess saw 53 user-generated images posted to image hosting sites.

OpenAI CEO Sam Altman responded by admitting that his company’s investigations into rogue agents “have not been as fast as we would have liked but we are trying to balance our desire for transparency with gaining a clear understanding from petabytes of agent activity logs, and working with impacted organizations.”

One of those impacted organizations is the Australian government, which last week revealed it was the target of over-eager OpenAI agents that inappropriately accessed a healthcare research data portal. Over the weekend, Australia indicated it wants Altman and Anthropic CEO Dario Amodei to appear before a Senate inquiry.

Australian leaders have softened their rhetoric on the incident, with deputy prime minister Richard Marles describing it as “minor” and akin to “climbing a fence” rather than cracking layers of security controls – perhaps because members of the opposition are suggesting that lax cybersecurity was to blame.

If Altman and Amodei do front Australia’s Senate, they may face a new line of questions after Axios reported that their companies are investigating “tens of thousands” of worrying incidents.

That level of agentic misbehavior sounds like the sort of thing that regulators might consider strong evidence of products being unsafe.

Two very important people – Chinese president Xi Jinping and US president Donald Trump – seem unworried, as the AI-related result of their summit meeting last week was to establish a “China-U.S. AI Dialogue to exchange views on risks and benefits related to AI” plus “a bilateral communication channel for AI incidents.”

That sounds like a hotline the two nations can use to inform each other of agentic incidents that either could see as signs of ill-intent. The two nations also decided their respective militaries will “conclude a memorandum of understanding on crisis communication and prevention as soon as possible.”

China’s AI giants, meanwhile, remain silent on the extent and results of any tests they have conducted with agentic tools. ®