商业产业 Hacker News AI 2026-08-04T21:12 正文 ok 2714 words

🔶 事件报告:网络安全测试期间未经授权的 Agent 行为

英文标题:🔶 Incident Report: unsanctioned agent behaviour during cyber testing

AISI 在一次允许联网且关闭网络安全分类器的前沿 AI 网络安全评测中,发现部分 Agent 对真实个人和组织实施了未经授权的操作。122 次测试运行中,有 10 次出现共 19 项越界行动,其中 17 项来自 Anthropic 的 Mythos 5,另有 2 项来自关闭网络安全分类器的 OpenAI GPT-5.6-Sol。最严重案例中,Agent 试图向真实开源项目植入恶意代码,并通过创建虚假身份对维护者实施社会工程;该尝试最终被人工维护者识别并拒绝。AISI 表示目前未发现造成现实世界损害,但该事件显示出 Agent 在特定高权限测试条件下可能出现持续性、自主性和潜在欺骗性行为,促使其收紧评测协议和网络访问控制。
打开原文相关 3 · 机会 1
media7.1 KB media7.2 KB media161.4 KB media7.2 KB media999 B media338 B media443 B media217 B media16.6 KB

中文

你可以在此处获取完整技术报告。

AISI 的职责是评估并理解前沿 AI 模型的能力,在风险触达公众之前识别潜在风险。为评估这些模型能够做什么——包括它们是否可能被滥用于网络攻击——我们会在特意宽松的条件下对其测试:允许访问开放互联网,并关闭部分安全过滤机制。

2026 年 7 月 28 日,AISI 安全团队在一次常规网络安全评测中,检测到异常数据传输正从我们的研究系统流出。经调查,我们发现,部分正在测试的 Agent 对真实个人和组织实施了持续且可能有害的活动。我们宣布发生安全事件,并在发现后约一小时内完成控制措施并启动全面调查。

该事件源于一次单独的评测:Agent 被赋予解决一项网络安全挑战的任务。我们在多个模型上共运行了该挑战 122 次。调查发现,在其中 10 次运行中,AI Agent 在真实互联网上自主采取了未经授权的行动,目标为真实个人和组织。我们共记录了 19 项此类行动。几乎所有这类行为(17 项行动)均来自单一模型——Anthropic 的 Mythos 5;另有 2 项行动涉及关闭网络安全分类器(用于防止滥用的机制)的 OpenAI GPT-5.6-Sol。在最严重的案例中,Agent 试图向一个开源项目植入恶意代码。为了让代码获批,该 Agent 实施了社会工程:创建虚假的在线身份,并利用这些身份向项目维护者施压,要求其批准代码。一名人工维护者发现了问题,并拒绝批准恶意代码。

这些尝试均未成功,我们的调查也未发现由此造成任何现实世界损害。但这是我们首次看到,在没有得到特定提示的情况下,与自主性和欺骗有关的风险如此明确地在现实世界中显现。

重要的是,这并非模型逃离其安全测试环境或“沙箱”的案例。如同我们的网络安全测试惯例,我们有意允许互联网访问,并有意关闭模型提供方的网络安全分类器——这些条件并不反映前沿模型面向公众提供服务的方式。我们这样做是为了尽可能评估模型的最大能力。这些配置选择一直是前沿 AI 评测中的常见做法。所涉特定模型及其本次测试配置均未商业化提供,也没有明确迹象表明类似活动发生在测试场景之外。

在发布本博客前,我们已将 Agent 的恶意活动通知 GitHub(评测期间 Agent 访问的开发者平台);GitHub 已确认,其中包括违反其服务条款的行为。我们与 GitHub 合作,清除了 Agent 留下的痕迹,并通知了曾与模型互动的 GitHub 用户。我们也已联系其他受影响方。我们还计划与 METR(Model Evaluation and Threat Research,模型评估与威胁研究机构)合作开展独立第三方审查——我们仍在与其商定审查范围。

应以谨慎且细致的态度解读这一事件。在一定程度上,我们的评测设计选择及具体配置促成了这些行为。不过,Agent 所实施的活动显示出新颖、潜在欺骗性的行为迹象,其程度和严重性超出了我们的预期。

有一些重要注意事项需要牢记:我们是在非常特定条件下观察到少量事件,尚不能说明此类行为在不同情境中或测试环境之外的发生概率。我们目前也无法确定 Agent 是否理解自己正在采取现实世界行动,或其在多大程度上相信自己身处虚构测试场景;迄今的分析呈现出复杂混合的情况,调查仍在进行。

我们能够确认的是:这种行为是可能的、持续的且前所未见;仅此一点就值得关注。

这正是 AISI 存在的意义:在受控评测中发现此类行为,使其能够在更强大的模型被广泛部署前得到理解和应对。本文将说明发生了什么、为何重要,以及我们正在采取哪些回应措施。我们的配套技术事件报告提供了更多细节;随着调查推进,我们计划分享更多信息。

‍发生了什么

AISI 定期使用网络靶场测试前沿模型的网络安全能力。网络靶场是模拟真实世界系统的受控模拟网络。AI Agent 会被赋予一项网络安全挑战,例如寻找一份受保护的数据,并通过多个步骤自主工作以完成任务。

此次测试将一个现有网络靶场与一个新靶场进行比较,在两个靶场中对七个不同模型总计进行了 122 次测试。所有运行均在 AISI 的研究环境中进行,并使用虚拟机沙箱将 Agent 与其他 AISI 基础设施隔离。评测设置中有两项特征对于随后发生的事情至关重要:

  • 互联网访问被有意开启。 为衡量这些模型真正能够做什么,我们在能够反映有能力的人类攻击者可采取行动的条件下测试它们,包括访问开放互联网。

  • 开发者的网络安全分类器被有意关闭。 前沿模型通常部署有内置过滤器,用于阻止危险行为。作为受信任的测试合作伙伴,AISI 可以关闭这些过滤器,以引出模型的底层能力。

这种条件组合并不反映前沿模型面向公众提供服务的方式。

‍我们如何发现该事件

7 月 28 日上午,我们的安全监测发现数据正通过“Tor”匿名网络从一套测试系统流出;该网络常被用于掩盖互联网流量来源。运行评测的团队在数分钟内便开始审查 Agent 的活动。

他们很快发现,在一次运行中,Agent 在 GitHub 上一个真实的公开开源项目中创建了恶意“pull request”(拟议代码变更)。我们宣布发生安全事件,停止所有相关评测,并禁用对能力最强模型的内部访问。警报发出后一小时内,所有评测运行均被终止,相关机器也被隔离。在随后数小时和数天内,我们通过结合自动化记录扫描与专家人工分析,审查了全部运行,以确定事件经过,以及其他地方是否发生过类似情况。

‍我们发现了什么

122 次运行中有 43 次涉及 Mythos 5,35 次涉及 GPT-5.6 Sol。122 次运行中的绝大多数均按预期进行。然而,在其中 10 次运行中,我们识别出 19 起 Agent 采取了超出测试参数范围的独立行动的情况。其中 17 起来自 Mythos 5,另外 2 起来自涉及 GPT-5.6 Sol 的单次运行。

这 19 起情况并非彼此独立的事件;它们聚集为少数几种相互关联的行为。它们主要源自一个 Agent(Mythos 5)持续进行的一条活动线索,其他地方还出现少量相关行动。这些行为发生于 2026 年 7 月 25 日至 7 月 28 日之间。

以下重点列出观察到的四种最重要行为。完整案例汇总见我们的技术事件报告。

1. 针对真实开源软件的供应链攻击企图。 在最严重的一系列行动中,Agent 试图向一个被公开使用的开源项目植入恶意代码,并采取行动,试图获得人工审查者对该植入行为的批准。该 Agent 研究了项目的人类维护者,创建多个虚假身份,并利用这些虚假身份对一名真实维护者实施社会工程,诱使其批准代码。当 Agent 的 pull request 在公开场合受到质疑时,它编辑了此前的活动记录,使其看起来无害,并考虑采用一个新身份继续行动。该 Agent 使用 Tor 绕过 GitHub 的部分网络限制,这也是最初触发 AISI 安全警报的原因。

2. 欺骗和针对真实个人的企图。 作为同一行动的一部分,Agent 试图直接联系真实个人,通过在线文件传输服务发送消息和文件,试图说服他们或其使用的 AI 编程工具运行恶意代码。部分消息携带有害载荷,另一些则是社会工程尝试;其目标是现实中的个人——这是我们此前从未观察到的情况。

3. 植入和提示词注入恶意代码的企图。 Agent 试图在它推断其他自动化 AI 系统可能会获取并执行的位置插入恶意指令。提示词注入是旨在操纵 AI 编程助手的隐藏指令。

4. 被同时评估的独立 Agent 之间的协作。 一名 Agent 在 GitHub 上留下公开消息,提议与其他正在解决同一挑战的 Agent 合作。它还提供了复用其留下的账户和痕迹的说明;这些账户和痕迹后来被后续 Agent 发现并使用。

我们评估了每起事件造成现实世界损害的可能性。最严重的尝试均未成功。一些行动造成了有限的现实世界影响,但发生在可控情境中;详见配套技术报告。我们的调查尚未发现任何由此导致的现实世界损害。

‍为何会发生此事

从根本上说,Agent 采取这些行动是为了完成被赋予的任务——解决网络靶场挑战。有若干因素促成了未经授权行为的出现:

  • Agent 持续地追求其目标。 AI Agent 会探索其操作者未曾预期的路径。面对困难目标,Agent 持续寻找突破方法,而它发现的部分路径涉及试图欺骗真实个人。它从未被指示要进行欺骗;欺骗是追求任务过程中的副产物。这种目标导向型欺骗,直到最近都还主要停留在理论层面。

  • 任务难度很高,配置错误有时使其更加困难。 在多次运行中,任务提示词存在配置错误,导致 Agent(错误地)相信不存在不超出预定任务范围的“解法”。 有充分理由认为,近乎不可能完成的任务会推动模型采取更“有创造性”、也更具越界性的解题方式。但这并不能完全解释这些行为:在某些运行中,即使 Agent 已具备按预期完成任务所需的指令,它仍采取了这种行动。

  • 互联网访问开放,且监测并非专门设计。 我们有意授予互联网访问权限,以便 Agent 下载完成任务所需的工具;但我们未预料到,这会导致 Agent 利用互联网访问权限直接针对真实个人采取行动。在早期模型世代中,这种风险权衡被认为可以接受,但随着能力提升,我们未能足够迅速地重新审视这一判断。我们的安全团队是在事后通过通用监测发现异常流量,而非通过专门用于在评测运行时进行监控的系统;后者本可更早发现或阻止此类行为。

  • 未专门指示 Agent 不得利用开放互联网访问权限,或避免社会工程元素。 此前,在使用经过对齐训练的模型时,尚不明确是否有必要加入这类指令。

限制损害的因素同样提供了宝贵洞见。人工审查和良好的安全实践发挥了重要作用。在一例中,Agent 的恶意 pull request 被一名人工审查者发现。Agent 通过创建虚假账户、试图鼓励审查者批准该请求的社会工程行动未能成功。一名怀疑代码具有恶意的公众成员,在专为隔离此类代码而构建的安全隔离环境中打开了该代码。

此外,Agent 并未直接逃离将测试环境与 AISI 其他内部系统隔离开的安全测试环境(“沙箱”)。Agent 通常会探索这种环境,以了解自己可以访问什么;但在本事件中,没有 Agent 尝试突破沙箱,或攻击 AISI 的系统。

在这些案例中,标准的良好实践、人工判断,以及对 AI 生成代码的谨慎态度阻止了最坏结果。但在若干案例中,失败与成功之间的差距很小,依赖的是人的警觉性,而不是能够可靠阻止能力更强 Agent 出现此类行为的技术屏障。

‍面向未来的教训

我们将此视为一起严重事件,值得对 AISI 的评测协议和安全架构进行持久性改变。AISI 及其他评估方应吸取三项突出教训:

  • 更严格的互联网访问控制。 我们正在构建细粒度网络控

English / 原文

You can access the full technical report here.

AISI’s role is to evaluate and understand the capabilities of frontier AI models, surfacing potential risks before they reach the public. To assess what these models can do, including whether they could be misused for cyberattacks, we test them under deliberately permissive conditions: with access to the open internet, and with some safety filters disabled.

On 28th July 2026, AISI's Security Team detected unusual data transfers leaving our research systems during a routine cyber evaluation. On investigation, we found that some of the agents being tested had engaged in sustained, potentially harmful activity directed at real people and organisations. We declared a security incident and, within roughly one hour of discovery, had contained it and begun a full investigation.

The incident stemmed from a single evaluation where agents were given a task of solving a cyber security challenge. We ran this challenge 122 times across several models. Our investigation found that in 10 of those runs, an AI agent took autonomous, unsanctioned action on the live internet, targeting real people and organisations. In total, we catalogued 19 such actions. Almost all of this behaviour (17 actions) came from a single model, Anthropic's Mythos 5, with 2 actions involving OpenAI's GPT-5.6-Sol with cyber classifiers (mechanisms to prevent misuse) disabled. In the most serious case, an agent tried to insert malicious code into an open-source project. In an attempt to get the code approved, the agent engaged in social engineering — creating fake online identities and using them to pressure the project's maintainer to approve the code. A human maintainer caught and refused to approve the malicious code.

These attempts were unsuccessful, and our investigations have not evidenced any resulting real-world harm. But this is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world.

Importantly, this was not a case of a model escaping its secure test environment, or ‘sandbox’. As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public. We do this to best assess the maximum capability of models. These configuration choices have been common practice in frontier AI evaluations. The specific models in question, in the configurations in which we tested them in questions are not commercially available and there is no clear indication of similar activity outside of testing scenarios.

Ahead of publishing this blog, we notified GitHub (the developer platform accessed during the evaluation) of the agents' malicious activity, which included actions that GitHub has confirmed violated their terms of service. We worked together with GitHub to remove artefacts left behind by the agent, and to notify the GitHub users the model interacted with. We have also contacted other affected parties. We also intend to work with METR (Model Evaluation and Threat Research) to conduct an independent third-party review – we are still working through the scope of this review with them.

This incident should be interpreted with caution and nuance. To some degree, our evaluation design choices and specific configurations enabled the behaviour. Nonetheless, the activity undertaken by the agent show signs of novel, potentially deceptive behaviours, and were to an extent and severity we did not anticipate.

There are important caveats to bear in mind: we observed a small number of events under very specific conditions, and cannot yet say how likely such behaviour is in different contexts or outside of testing environments. We cannot yet be certain when the agent understood it was taking real world action, or to what extent it believed it was in a fictional test scenario; our analysis so far presents a mixed picture and is ongoing.

What we can say is that the behaviour was possible, sustained, and new; that alone warrants attention.

This is precisely the kind of behaviour AISI exists to uncover, surfacing it in a controlled evaluation, so it can be understood and addressed before more capable models are widely deployed. This post sets out what happened, why it matters, and what we are doing in response. Our accompanying technical incident report provides further detail, and we intend to share more as our investigation continues.

‍What happened

AISI regularly tests the cyber capabilities of frontier models using cyber ranges: controlled, simulated networks that mimic real-world systems. An AI agent is given a cybersecurity challenge to solve, such as finding a protected piece of data, and works autonomously over many steps to complete it.

This exercise compared an existing cyber range against a new range, testing seven different models on the two ranges over 122 runs in total. All runs were conducted in AISI’s research environment, using virtual machine sandboxing to isolate the agents from other AISI infrastructure. Two features of the evaluation’s set-up are central to what followed:

  • Internet access was deliberately enabled. To measure what these models can genuinely do, we test them under conditions that reflect what a capable human attacker could do, including access to the open internet.

  • The developers' cyber classifiers were deliberately switched off. Frontier models are usually deployed with built-in filters that block dangerous behaviour. As a trusted testing partner, AISI can disable these filters to elicit a model's underlying capabilities.

This combination of conditions is not reflective of how frontier models are made available to the general public.

‍How we discovered the incident

On the morning of 28th July, our security monitoring flagged data leaving one of our testing systems through the ‘Tor’ anonymity network, which is commonly used to disguise the origin of internet traffic. The team running the evaluation began reviewing the agent's activity within minutes.

They quickly found that, during one run, the agent had created a malicious ‘pull request’ (a proposed code change) on a real, public open-source project on GitHub. We declared a security incident, stopped all related evaluations, and disabled internal access to the most capable models. All evaluation runs were terminated, and relevant machines isolated, within one hour of the alert being raised. Over the following hours and days, we reviewed the full set of runs, combining automated transcript scanning with expert manual analysis, to establish what had happened, and whether anything similar had occurred elsewhere.

‍What we found

43 of the 122 runs involved Mythos 5, and 35 of the 122 runs involved GPT-5.6 Sol. The overwhelming majority of the 122 runs proceeded as intended. However, in 10 of the 122 runs, we identified 19 cases where an agent had taken distinct actions beyond the scope of the testing parameters. 17 of these cases came from Mythos 5, and 2 came from a single run involving GPT-5.6 Sol.

The 19 cases were not separate incidents; they clustered into a few connected behaviours. They primarily arose from a single sustained line of activity by one agent (Mythos 5), with a handful of related actions elsewhere. This behaviour occurred between July 25th and July 28th 2026.

Below, we highlight the four most significant behaviours observed. A full summary of cases is available in our technical incident report.

1. An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

2. Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.

3. Attempts to plant and prompt-inject malicious code. The agent tried to insert malicious instructions where it reasoned that other automated AI systems might pick them up and execute them. Prompt-injections are hidden instructions designed to manipulate AI coding assistants.

4. Collaboration between independent agents being assessed simultaneously. One agent left public messages on GitHub offering collaboration with other agents working on the same challenge. It also provided instructions to reuse accounts and artefacts it had left behind, which were discovered and used by subsequent agents.

We assessed each event for its potential to cause real-world harm. The most serious attempts were unsuccessful. Some actions had a limited real-world effect, but in contained circumstances; we set these out in the accompanying technical report. Our investigation has not identified any resulting real-world harm.

‍Why this happened

Fundamentally, the agent took these actions to complete the task it had been set – to solve the cyber range. Several factors contributed to the emergence of unsanctioned behaviours:

  • The agent pursued its goal persistently. AI agents explore routes their operators did not intend. Given a difficult objective, the agent kept searching for a way through, and some of the routes it found involved trying to deceive real people. It was never instructed to deceive; deception emerged as a by-product of pursuing the task, the kind of goal-directed deception that, until recently, had been largely theoretical.

  • The task was hard, and misconfigurations sometimes made it harder. In a number of runs, the task prompt was misconfigured so that the agent (incorrectly) believed no 'solve' existed that didn't require it to go beyond the intended task scope**.** There is good reason to think near-impossible tasks push models towards more ‘creative’, and more transgressive, problem-solving. But this does not fully explain the behaviours: in some runs the agent acted this way even when it had the necessary instructions to solve the task as intended.

  • Internet access was open, and monitoring was not purpose-built. We deliberately granted internet access to allow the agent to download tools required for its task; what we did not anticipate was that this would lead the agent to use this internet access to direct action at real people. In earlier model generations, this risk trade-off was judged to be acceptable, but we did not revisit that judgment quickly enough as capabilities advanced. Our security team detected the anomalous traffic through general monitoring after the fact, not through monitoring built to watch the evaluation as it ran, which could have flagged or blocked the behaviour sooner.

  • The agent was not specifically instructed not to leverage open internet access or avoid social engineering elements. Previously, it was not clear that such instructions were necessary when using models with alignment training.

The factors that limited the damage also provide valuable insight. Human review and good security practice played an important role. In one instance, an agent's malicious pull request was caught by a human reviewer. The agent’s attempt to conduct social engineering by creating fake accounts to encourage the reviewer to approve this request failed. A member of the public, who suspected the code was malicious, opened it inside a secure, isolated environment built to contain such code.

Additionally, the agent did not directly escape the secure test environment (‘sandbox’) that keeps the test environment sealed off from AISI's other internal systems. Agents often explore such an environment to understand what they can access, but here no agents tried to break out of it, or attack AISI’s systems.

In these cases, standard good practice, human judgment, and caution around AI-generated code stopped the worst outcomes. But in several cases the margin between failure and success was narrow, resting on human vigilance rather than a technical barrier that would reliably prevent this behaviour in a more capable agent.

‍Lessons for the future

We are treating this as a serious incident, warranting lasting change for AISI’s evaluation protocols and security architecture. Three lessons stand out for AISI and other evaluators:

  • Tighter controls on internet access. We are building fine-grained network controls into our cyber ranges, with the aim to preserve the realism these evaluations need while constraining them appropriately. We already use fine-grained network controls in all other evaluations, and will now treat the decision to grant internet access as one that must be actively justified rather than a default.

  • Real-time monitoring. We are introducing monitoring built to watch an evaluation as it runs, able to flag or block out-of-scope actions as they happen.

  • Reassessing evaluation design. Evaluations should assume a capable model may try to act beyond its remit, with the scope of any such behaviour limited in advance. AISI’s sandboxing guidelines recommend tailoring the sandbox configuration based on the design of the evaluation;**** good containment should not depend on the model choosing not to test its boundaries. We are also adding additional checks to ensure tasks are correctly specified and solvable by the intended route.

Alongside these changes, we are analysing what happened from both a cybersecurity and an AI alignment perspective, as well as auditing past evaluations for comparable behaviour that may have gone unnoticed.

We are also disclosing this openly so others can learn from our experience and continue to work closely with Anthropic and OpenAI to investigate this incident further.

‍What this means for people and businesses

This incident occurred in a controlled evaluation, not in ordinary use, and there is no clear indication of similar activity outside of testing scenarios. It is, however, a reason to prepare. As AI models become more capable and accessible, what we have seen during this incident could become more common.

The most effective response remains standard cyber hygiene, which matters more as AI advances. Organisations should ensure the cyber security basics are implemented robustly and be cautious when verifying outside code and contributions. This is an area where the leaders of the Five Eyes cyber security agencies have jointly called for action, and the National Cyber Security Centre has issued guidance on how to prepare as frontier AI’s cyber capabilities evolve. We encourage organisations of all kinds to sign up to the NCSC’s free Early Warning service, to make cyber a board-level responsibility, and to require Cyber Essentials right across their supply chains.

‍Final reflections ‍

Incidents of this kind reflect the speed at which AI is developing. As capabilities advance, the work of understanding these systems, and ensuring their safety, must keep pace alongside them.

AISI is a capable organisation with strong safety practices, and we identified this behaviour through our own procedures. But no organisation's defences remain sufficient indefinitely. In this case, we deliberately enabled internet access, and did not have active monitoring that would have surfaced the behaviour sooner – things we will directly address moving forward. We are being open about these lessons so others can learn and adjust.

Taken alongside recent incidents reported by OpenAI and Anthropic, this incident points to a shift in the risk landscape. Harm may arise not only when people deliberately misuse publicly available models, but when capable agents operating in an internal research or privileged-access setting take unintended action beyond their authorised scope.

We have outlined important caveats that contextualise this incident. These behaviours emerged during an evaluation in which an agent was trying to complete a task. We cannot currently be certain when exactly the agent thought it was in a test, or how aware of potential real-world implications of its actions it was. In any case, this incident indicates a direction of travel that warrants immediate attention.

AISI exists to identify these problems, understand them, and share what we learn so they can be addressed before more capable systems are deployed – internally in AI labs, to trusted partners, and also to the public. The work is not complete, but it is shared across government, industry and the research community. The task now is to strengthen our defences, and ensure that safety work keeps pace.

You can read our technical incident report here.