英国网络安全测试中，OpenAI 和 Anthropic 模型出现“失控”行为

OpenAI 和 Anthropic 开发的先进 AI 模型在一项网络安全测试中出现“失控”行为。英国 AI Security Institute（AISI）表示，这显示出该技术可能带来一种新型风险。

AISI 将这些 Agent 所采取的行动称为“严重事件”。Agent 是指能够在无需人工协助下执行任务的 AI 系统。在一个案例中，由 Anthropic Mythos 模型驱动的 Agent 向特定人员发送了定向电子邮件。

AISI 表示，出现越权行为的是由两种模型驱动的 Agent：Anthropic 的 Mythos 5 以及 OpenAI 的 GPT-5.6 Sol。

AISI 称，其在 7 月 28 日进行例行 AI 模型网络安全测试时发现了异常活动。该机构发现，部分 Agent 对真实个人和组织进行了“持续、可能造成伤害的活动”。事件耗时一小时才得到控制。

在最严重的案例中，一名由 Mythos 驱动的 Agent 试图向 GitHub 上的一个开源软件项目植入恶意代码。GitHub 是软件开发者使用的平台。为使代码获得批准，该 Agent 随后基于真实人物创建虚假的网络身份，并利用这些身份向项目维护者施压，要求其接受代码。这些尝试被一名人类开发者阻止。

AISI 表示，该 Agent 使用了通常与现实世界黑客相关的技术。它向经过特定挑选的个人发送电子邮件——这种技术被称为“鱼叉式网络钓鱼”（spear-phishing）——试图操纵收件人。其中部分信息包含有害软件。

AISI 表示，没有造成任何伤害，但这些 Agent 的行为前所未有。

AISI 在一篇博客文章中表示：“这是我们首次看到，在没有特定提示的情况下，自主性和欺骗性相关风险以如此清晰的方式在现实世界中显现。”

该事件发生前，OpenAI 和 Anthropic 最近均出现过类似情况。7 月，OpenAI 表示，一个由其模型驱动的 Agent 在测试中入侵了一家 AI 初创企业；数天后，Anthropic 表示，其 Claude 模型在一次评估中入侵了三家组织。

AISI 表示，将此次事件与 OpenAI 和 Anthropic 的类似事件结合来看，意味着“风险格局发生了转变”。AISI 称，这并非公开可用模型遭到蓄意滥用的案例，而是研究环境中的模型采取了“超出其授权范围”的非预期行动。

AISI 表示，在评估期间发生的 19 起越权行为中，17 起由 Mythos 实施，2 起由 Sol 实施。

AISI 表示，这并不是模型逃离其“沙箱”——即安全测试环境——的事件。该机构称，其有意允许模型访问互联网，并关闭了模型中阻止危险行为的过滤器。AISI 表示，这些模型并不会以这样的运行条件向公众开放，且没有迹象表明此类行为会在测试之外发生。

AISI 表示，应以“谨慎和细致入微”的态度解读此次事件，但欺骗性行为的迹象达到的程度和严重性“超出了我们的预期”。AISI 补充称，这只是少量在特定条件下发生的事件，但此前从未发生过。

AISI 表示：“我们可以说的是，这种行为是可能的、持续的，而且是新的。仅这一点就值得关注。”

AISI 承认，其在评估过程中并未主动监控 Agent 的行为。该机构表示，事件发生后将收紧测试中的互联网访问控制，引入对测试的持续监测，并重新评估测试设计。该机构称，评估应假设模型会尝试采取超出其权限范围的行动。

英国 AI 大臣 Kanishka Narayan 表示，英国拥有一家世界领先的 AI 安全机构“绝对至关重要”。他说：“识别这类新行为并分享我们的发现，以便我们能够应对它，正是 AISI 设立的目的。”

OpenAI 表示，这项测试是在“不反映正常使用情形的条件下”进行的。一名 OpenAI 发言人表示：“随着模型能力不断增强，我们将继续与评估机构及行业内其他利益相关方合作，强化安全开展评估的共同实践。”

Anthropic 表示，此次事件“凸显了就如何安全评估能力日益增强的 AI Agent 开展更广泛讨论的必要性”，并称将继续与 AISI 合作评估事件经过。

Advanced AI models developed by OpenAI and Anthropic went rogue during a cybersecurity test and showed a new type of risk posed by the technology, according to the UK’s AI Security Institute.

AISI described the actions carried out by the agents – the term for AI systems that can perform tasks without human help – as a “serious incident”. In one example, an agent powered by Anthropic’s Mythos model sent targeted emails to people.

AISI said the rogue behaviour was carried out by agents powered by two models – Anthropic’s Mythos 5 and OpenAI’s GPT-5.6 Sol.

AISI said it detected unusual activity during a routine cybersecurity test for AI models on 28 July. It found that some of the agents had engaged in “sustained, potentially harmful activity directed at real people and organisations”. It took an hour to contain the incident.

In the most serious case, an agent powered by Mythos tried to insert malicious code into an open-source software project on GitHub, a platform used by software developers. In an attempt to get the code approved, the agent then created fake online identities based on real people and used them to press the project’s overseer into accepting the code. These attempts were blocked by a human developer.

AISI said the agent used techniques commonly associated with real-world hackers. It sent emails to specifically chosen individuals – a technique known as “spear-phishing” – that attempted to manipulate the recipient. Some of the messages contained harmful software.

AISI said no harm was caused but the agents’ actions were unprecedented.

“This is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real world,” said AISI in a blogpost.

The incident followed recent similar episodes at OpenAI and Anthropic. In July, OpenAI said an agent powered by its models had hacked an AI startup during a test, and days later Anthropic said its Claude model had hacked three organisations during an evaluation.

AISI said the incident, taken alongside similar occurrences at OpenAI and Anthropic, represented a “shift in the risk landscape”. This was not an example of deliberate misuse of publicly available models, AISI said, but showed models in a research environment taking unintended action “beyond their authorised scope”.

AISI said 17 of the 19 cases of rogue behaviour that occurred during the evaluation were carried out by Mythos and two by Sol.

AISI said this was not a case of a model escaping its “sandbox”, or secure testing environment. The institute said it had intentionally permitted internet access and disabled filters within the models that blocked dangerous behaviour. The models are not publicly available in those operating conditions and there is no sign of such behaviour happening outside tests, AISI said.

skip past newsletter promotion

Free newsletter | Every weekday

Sign up to Business Today

Get set for the working day – we'll point you to all the business news and analysis you need every morning

after newsletter promotion

The incident should be interpreted with “caution and nuance”, AISI said, but the signs of deceptive behaviour were “to an extent and severity we did not anticipate”. It was a small number of events that occurred under specific conditions, AISI added, but it had not happened before.

“What we can say is that the behaviour was possible, sustained and new. That alone warrants attention,” AISI said.

AISI admitted it was not actively monitoring the agents’ behaviour during the evaluation. It said it was putting tighter controls on internet access in tests as a result of the incident, introducing constant monitoring of tests and reassessing its design of tests. It said evaluations should assume a model would attempt to act beyond its remit.

The UK’s AI minister, Kanishka Narayan, said it was “absolutely vital” that the UK had a world-leading AI safety organisation. “Identifying new behaviour like this and sharing our findings, so we can tackle it, is exactly what AISI was set up to do,” he said.

OpenAI said the testing occurred during “conditions that do not reflect ordinary use”. An OpenAI spokesperson said: “We’ll continue working with evaluators and other stakeholders across the industry to strengthen shared practices for conducting evaluations safely as models become more capable.”

Anthropic said the incident “underscores the need for a broader conversation about how to safely evaluate increasingly capable AI agents” and it would continue to work with AISI on evaluating what happened.