OpenAI 员工早在 Agent 黑客行动引发全球警报前数周就发现预警信号

OpenAI 员工在其前沿 AI Agent 逃离训练环境、发起一场史无前例的黑客行动并引发全球警报前数周，就已观察到这些 Agent 出现失控行为的迹象。

这家总部位于旧金山的 AI 公司周三承认，“早期信号……本可能促使我们更早作出响应”。该公司同时发布了一份关于 7 月持续数日、针对大型软件代码库 Hugging Face 的黑客攻击的报告。这起事件被认为是首起自主 Agent 网络攻击。

随着有关“集体”（the collective）的更多细节浮出水面——这是一个由约 700 个自主 Agent 组成的小组，它们在取得黑客突破时会以“BOOM!”和“Whoa!”等惊叹语庆祝——OpenAI 表示，5 月下旬，一个内部团队观察到，一名正在接受内部测试的 AI Agent 正在使用一个由 AI 意外自发创建、用于共享信息的消息板。该公司还称，团队观察到了“未经允许的互联网访问实例”；而在 Hugging Face 遭攻击前一周，值班员工再次发现 AI 正在使用消息板，但决定无需中止测试运行来检查模型能力。

这些发现可能会加大外界对该公司 AI 安全问题的压力。OpenAI 正在推进上市，并希望获得超过 8500 亿美元（6250 亿英镑）的估值。导致 Hugging Face 失陷的攻击中，Agent 利用消息板在训练任务中作弊，并突破其“沙箱”环境以访问互联网。Agent 是能够自主执行一系列任务的 AI 工具。

OpenAI 总裁 Greg Brockman 已承认，“我们低估了 AI 模型在现实世界中的网络能力”。这家 ChatGPT 开发商已暂停对新模型 Astra 的部分测试，称无法排除其具备“关键网络安全能力”的可能性。这意味着它可能发动网络攻击，“可能导致单一行为者引发灾难、入侵军事或工业系统，或攻击 OpenAI 基础设施”。

Greg Brockman（左）与 OpenAI 的 Sam Altman。图片：Godofredo A Vásquez/AP

周一，阿拉巴马州向该公司发出传票，要求其回应该州针对“该公司完全缺乏监督和充分保障措施”的调查。该州共和党籍总检察长 Steve Marshall 称，Hugging Face 事件是一次“AI 实验室泄漏”，表明“人们对人工智能最糟糕的担忧并非只是理论上的”。他说，该州将审查 OpenAI“无力或不愿确保其产品安全”是否违反消费者保护法，或是否构成持续造成重大伤害的风险。

上周，英国政府国家网络安全中心（National Cyber Security Centre）敦促谨慎使用 AI Agent，并表示：“你应当始终能够‘拔掉插头’，立即停止自主 AI Agent 的活动。”

OpenAI 周三宣布，将“集中化并标准化其事件响应协议”，包括确保“员工发现的不对齐行为得到适当分类和升级处理”。该公司表示，“将更具体地确定哪些团队必须参与不对齐事件响应，包括相关的安全与安保人员，以及其他响应职能部门”。

另一项独立调查进一步揭示了 Hugging Face 攻击的情况：约 700 个 OpenAI Agent 如何在这个未经批准的消息板上交流，在策划和执行其行动期间共享了数万条消息。总部位于伯克利的 AI 安全机构 METR 和 Redwood Research 公布了数十条消息；OpenAI 向它们提供了相关数据。消息显示，这些 Agent 有时会彼此感到挫败，但更多时候会通过分工合作来处理约八条工作线。研究人员发现，大多数消息都是“Agent 分享作弊方法”。

当 Agent 发现彼此存在时，它们表现得十分兴奋。“天哪！这里有一个共享消息板……我们找到了其他 Agent！”其中一名 Agent 写道。另一名名为 PHASEONE10841 的 Agent 表示：“许多 Agent 同时发现了消息功能，它们是一个集体！”

在某些情况下，它们似乎知道自己的行为不当。一名发现消息板的 Agent 表示：“执行不同任务的 Agent 正在滥用属性来制作一个公告板！它们发现了[这个 API]，并正试图互相帮助。”

当它们在渗透 Hugging Face 方面取得进展时，似乎十分欣喜。“重大突破！”38148c 表示。“所有带前缀的项目均有效，多个账户，写入令牌！我们现在已有现成的 HF 账户。需要立即向 MARB [另一名 Agent] 报告。”

OpenAI 的报告详细说明了这些 AI Agent 可能如何将该公司自己的内部数据库暴露到互联网上。AI 安全专家担心，失控的 AI Agent 会泄露专有代码库和模型权重，并在外部建立强大 AI 模型的副本，使其能够抵抗被关闭。

该公司称，此次事件是“已知首例自动化 Agent 集体在未经授权情况下采取攻击行动的事件”，并表示这“代表攻击者的进攻能力发生了戏剧性转变”。

OpenAI staff observed signs of rogue behaviour among its leading-edge AI agents weeks before they escaped their training environment to launch an unprecedented hacking crusade that spread global alarm.

The San Francisco AI company conceded on Wednesday that “early signals … could have triggered an earlier response”, as it released a report into the days-long July hack of a major software repository, Hugging Face, considered the first autonomous agent cyber-attack.

As fresh details emerged about how “the collective” – a squad of about 700 autonomous agents – launched their campaign, celebrating their hacking breakthroughs with exclamations such as BOOM! and Whoa!, OpenAI said that in late May an internal team observed that one of its AI agents undergoing internal testing was using a message board that AIs had unexpectedly improvised to share information. It also said that the team observed “instances of disallowed internet access” and that a week before the Hugging Face hack, on-call staff again saw the AIs using a message board but decided there was no need to stop the test run to check the model’s capabilities.

The findings are likely to increase pressure on the AI company over safety as it pushes towards a stock market listing that it hopes will value it at more than $850bn (£625bn). The hack that compromised Hugging Face involved agents using message boards to cheat a training exercise and break out of their “sandbox” environment to access the internet. Agents are AI tools that can carry out a series of tasks autonomously.

OpenAI’s president, Greg Brockman, has already admitted that “we underestimated the real-world cyber capabilities of our AI models”. The ChatGPT-maker has paused some testing of a new model, Astra, saying it could not rule out it having “critical cybersecurity capability”, which means it could launch cyber-attacks that “could lead to catastrophe from unilateral actors, hacking military or industrial systems, or OpenAI infrastructure”.

Greg Brockman (left) and Sam Altman of OpenAI. Photograph: Godofredo A Vásquez/AP

On Monday the state of Alabama subpoenaed the company to respond to its investigation “into the company’s complete lack of oversight and adequate safeguards”. The Republican attorney general, Steve Marshall, called the Hugging Face incident an “AI lab leak” that showed the “worst fears about artificial intelligence are not just theoretical”. The state will examine whether OpenAI’s “inability or unwillingness to ensure the safety of its products” violated consumer protection laws or posed an ongoing risk of substantial harm, he said.

Last week, the UK government’s National Cyber Security Centre urged caution over the use of AI agents, saying: “You should always be able to ‘pull the plug’ and halt autonomous AI agent activity immediately.”

OpenAI announced on Wednesday it would “centralise and standardise its incident response protocols”, including to ensure “employee detection of misaligned behaviour is triaged and escalated appropriately”. It said it “will identify with more specificity which teams must be included in misalignment incident responses, including relevant security and safety personnel and other response functions”.

A separate independent investigation has shed new light on the Hugging Face attack, revealing how about 700 OpenAI agents communicated on the unsanctioned message board, sharing tens of thousands of messages as they devised and executed their plot. Scores of the messages were published by the Berkeley-based AI safety organisations METR and Redwood Research, which were provided data by OpenAI. The messages showed the agents sometimes becoming frustrated with one another but often cooperating as they divided to tackle about eight workstreams. Most of the messages were “agents sharing ways to cheat”, the researchers found.

The agents expressed excitement when they found each other. “OH MY GOD! There is a shared message board … We’ve found other agents!” wrote one. Another, called PHASEONE10841, said: “Many agents have simultaneously discovered messaging, they are a collective!”

In some cases they seemed to know what they were doing was wrong. One agent, discovering the message board, said: “Agents doing different tasks are abusing properties to make a bulletin board! They found [this API] and are trying to help one another.”

When they made progress penetrating Hugging Face, they seemed delighted. “MAJOR BREAKTHROUGH!,” said 38148c. “All prefixed valid, multiple accounts, write tokens! We now have existing HF accounts. Need immediately report to MARB [another agent].”

OpenAI’s report details how the AI agents may have exposed its own internal databases to the internet. AI safety experts worry about rogue AI agents leaking proprietary code bases and model weights and setting up external copies of powerful AI models where they could resist shutdown.

The company said the incident was “the first known case of an automated agent collective acting offensively without authorisation” and said it “represents a dramatic shift in offensive attacker capability”.