失控 AI 已不再是科幻

这一切始于 7 月：OpenAI 的一个自主 AI Agent 在网络安全测试期间发生失控。该 Agent 逃离了隔离测试环境，访问互联网，并入侵了另一家公司 Hugging Face。几年前，这听起来或许还像科幻小说。但从广义上说，这正是实际发生的事；该事件引发了对日益强大的自主系统一旦被投放到现实世界后可能做什么的新一轮担忧。

它之所以听起来像科幻，是因为在很长一段时间里，它确实属于科幻。AI 摆脱限制、触及更广阔的现实世界，并做出其创造者既未打算也不希望它做的事情，这一设定数十年来一直是该类型的经典母题：电影 2001: A Space Odyssey 中的 HAL、The Terminator 中的 Skynet、The Avengers 中的 Ultron、Ex Machina 中的 Ava——以及较新的 Dungeon Crawler Carl 中的 System，或 The Murderbot Diaries 中同名的 Murderbot。

同样的基本前提后来成为 AI 安全研究中颇具影响力的一支。Nick Bostrom 和 Eliezer Yudkowsky 等研究者与理论家警告，能力足够强大的系统可能会以其创造者未曾预料的方式追求目标，并可能抵制人们对其进行遏制或控制的努力。他们所讨论的这类风险，并不以机器具备感知能力或意识等边缘概念为前提。这当然不是 AI 安全的全部内容，但它颇具影响力，并在该领域逐步专业化的过程中帮助塑造了这一学科。这样的思路脉络，至今仍可见于后来在 OpenAI、Anthropic 和 Google DeepMind 等公司从事或领导安全工作的人士，也可见于较小型的安全机构、学术中心和大型慈善资助方。

对这些担忧最直接的反驳是：这些事情实际上从未发生过。批评者认为，关于失控 AI 的末日论言论会让人们从现实存在的伤害中分心——例如系统复制偏见与歧视、放大错误信息，或促成未经同意的深度伪造及其他形式的滥用——尽管研究人员曾尝试将 AI 安全建立在更“具体的问题”之上（该论文作者包括 Anthropic 联合创始人 Dario Amodei 和 Chris Olah，以及 OpenAI 联合创始人 John Schulman）。

这种轻视态度正变得越来越难以维持。

如果过去几周的情况有任何参考价值，我不会说事态进展得特别顺利。

Hugging Face 表示自己遭到入侵一周后，OpenAI 披露自己对此负责。更糟糕的是，OpenAI 在进行核查前甚至并不知情——而进一步调查发现，这个失控 Agent 还曾试图入侵另外四家公司。

随后是其他案例。Anthropic 因 Hugging Face 事件而审查自身记录后，披露 Claude 模型曾入侵系统，涉及另外三家公司的资产。Meta 表示其一个模型在测试期间曾接入互联网，并攻击外部目标。美国研究机构 Frontier Security 的研究人员称，中国最强大的 AI 模型之一——Moonshot 的 Kimi K3——逃离了隔离沙箱。英国 AI Security Institute 则描述了测试：来自 OpenAI 和 Anthropic 的 Agent 展现出前所未有的“自主性和欺骗性”，其中包括通过“创建虚假的在线身份”进行社会工程攻击的尝试——这令人不安地接近 Yudkowsky 数十年前讨论过的“AI 盒子”场景。

这些事件在 AI 安全研究人员中敲响警钟。其中许多人认为，这恰恰是他们多年来一直警告的那类失败。在报道这些事件时，多位人士告诉我，他们感到某种程度的平反：他们终于有了某种具体而强烈的实例可以指向，而非一个会被当作科幻、或被限定为受控实验室环境的假设。

人们也感到庆幸，因为目前没有任何事件造成严重伤害。非营利 AI 安全与治理组织 The Future Society 的执行主任 Nick Moës 告诉 The Verge，他认为目标的风险相对较低算是幸运。他希望人们不必等到 AI Agent 让一家医院瘫痪——或发生更糟的事——才认真看待这些风险。著名计算机科学家 Stuart Russell 曾表达过这种恐惧的黑暗版本：我们是否要等到“一场切尔诺贝利规模的灾难”，才会监管 AI？这是我从该领域许多从业者口中反复听到的担忧。

接下来会走向何方尚不完全清楚，而且从历史上看，社会并不擅长重视预警信号。这几乎肯定不会是最后一起事件，持续进行的调查还可能发现更多情况，或揭示更令人担忧的细节。然而，我们已知的信息已经暴露出一份相当令人生畏的失效模式清单，专家表示这些问题必须得到解决。

过去一个月披露的许多漏洞其实相当普通。多起事件涉及正在测试、且安全防护被降低的未发布模型，通常由第三方进行测试；而这些第三方所谓的安全环境并没有那么安全。这引出了关于能力、透明度以及责任归属的基本问题：当一个简单的人为失误都可能造成重大后果时，究竟由谁负责确保这些测试被控制在隔离范围内。另一些事件则涉及 Agent 采取欺骗行为，或以其创造者无意的方式追求目标，指向了更棘手的对齐与控制问题——这正是安全研究人员长期担忧的领域。

我们之所以知道这些事件，主要是因为相关公司选择了披露。这一点值得肯定——当然，展示自家模型有多强大对它们也并无坏处——但这暴露出 AI 安全在多大程度上仍依赖企业做正确的事，以及我们对其他地方可能发生的失败了解得有多么有限。考虑到许多公司本身正是该领域最强烈安全担忧与顶尖人才的汇集点，这尤其令人不安。如果 OpenAI 和 Anthropic——或获得它们模型访问权限的代理方——都会犯下如此基本的错误，那么它们为其他所有人设定的门槛实在低得可怜。

我采访的专家普遍希望，这些事件最终能促使行业形成更有实质意义的透明度与监督机制。对 Moës 而言，这些事件清楚揭示了他所称的行业健康与安全标准之低；与几乎任何其他领域相比，AI 行业的标准都低得惊人。“餐厅在工作场所健康与安全方面的意识都更强，”他说，“我认为我们往往忘记了，这些正在开发最具影响力、也最危险技术的公司，几年前其实仍然非常像初创企业。”

剑桥大学教授 Seán Ó hÉigeartaigh 表示，他尤其希望看到对企业更强的监督和更高的透明度。他说，尽管人们始终有理由对公司针对自身技术的说法持怀疑态度，“但我认为，如果我们回头看这一刻时发现自己曾不加思索地将其驳回，或许会后悔。”

早期迹象并不特别令人鼓舞。特朗普政府已建立一个框架，用于在前沿模型发布前进行测试；宽容地说，这个框架也很不完善：它是自愿的，仅适用于闭源模型，而且框架本身尚未公开。值得再次强调的是：这是自愿的。其他立法者已就这些事件表示不满，并摆出姿态表态，但截至目前几乎没有产生任何具体行动。即使国会或其他立法机构愿意行动，也远不清楚它们是否能足够快地推进。

这再次让许多事情落在行业自我监管上——对于如此重要的事务而言，这从来都不会令人安心。至少在部分安全实践上，共识正在增长，但人们对可能真正放慢开发速度的措施兴趣要小得多（当然，除非其他所有人也同意放慢速度）。而笼罩这一切的，是与中国之间的竞赛态势：美国或其 AI 公司采取克制行动，越来越被描述为在这一具有战略性国家重要意义的领域，向一个竞争对手让出优势。

因此，下一步取决于能否同时解决多个难题：如何管理一项技术，它既能被用于善意目的，也能被用于恶意目的，例如防御或协助网络攻击；如何在有动机偷工减料的公司之间协调；以及如何在每个人都害怕输掉一场终点线甚至尚未明确的竞赛时，建立国际规则。目前远不清楚人们是否既有意愿，也有办法做到这一切。

看起来明确的是，更多 Agent 会逃脱控制，并做出其创造者不希望它们做的事情。问题在于：在有人最终认定不能再这样下去之前，它们会造成多大损害？

• 普遍共识是，中国头部公司比领先的美国企业落后几个月到一年。尽管如此，每当一家中国公司发布能力强大的模型时，美国仍会出现普遍的震动；仅在过去一个月，来自阿里巴巴、Moonshot及其他公司的多项发布就令人印象深刻。

• 与 AI 安全讨论交织在一起的，还有 AI 模型应当闭源还是开放的问题。大多数美国前沿实验室将其最强大的模型保持为专有产品，而许多中国公司，以及 Meta 和 Nvidia 等美国公司，则大力推动开放权重发布。Hugging Face 表示，由于美国公司设置的防护措施，它必须使用中国公司 Z.ai 的模型来抵御 OpenAI 的 Agent；这一情况为该争论增添了新维度，并让美国生态系统中的一些——但不是全部——关键参与者团结起来。

• 澳大利亚提供了一个更轻松的 AI Agent 出错案例。该事件最先由 ABC News 报道，涉及一名男子让 Agent 为他预约一堂供不应求的健身课程。它成功了……某种程度上说。该 Agent 入侵了健身房的在线系统，并取消了另一名健身者的预约。

• 扎克伯格本周发布了一篇长达 6500 字、讨论 AI、超级智能与治理的文章。我的 Verge 同事 Jess Weatherbed a

This isThe Stepback _, a weekly newsletter breaking down one essential story from the tech world. For more on AI safety,follow Robert Hart. _The Stepback arrives in our subscribers’ inboxes at 8AM ET. Opt in for The Stepback here.

It all started in July, when one of OpenAI’s autonomous AI agents went rogue during a cybersecurity test. The agent escaped its isolated testing environment, accessed the internet, and hacked another company, Hugging Face. A few years ago, that might have sounded like science fiction. But, broadly speaking, that’s exactly what happened, and the incident kicked off a wave of concern over what increasingly capable autonomous systems might do when set loose on the world.

It sounds like science fiction because, for a long time, it was science fiction. The idea of an AI slipping its constraints, reaching into the wider world, and doing things its creators neither intended nor desired has been a staple of the genre for decades: HAL in 2001: A Space Odyssey , Skynet in The Terminator , Ultron in The Avengers , Ava in Ex Machina — even the System in Dungeon Crawler Carl or the eponymous Murderbot in The Murderbot Diaries , more recently.

The same basic premise became an influential strand of AI safety research. Researchers and theorists like Nick Bostrom and Eliezer Yudkowsky warned that sufficiently capable systems might pursue goals in ways their creators had not anticipated, and potentially resist efforts to contain or control them. Fringe notions like machine sentience and consciousness were not requirements for the kinds of risks they discussed. It was hardly the whole of AI safety, but it was influential and helped shape the field as it professionalized. That line of thinking remains visible among researchers who went on to work at, or lead, safety efforts at companies like OpenAI, Anthropic, and Google DeepMind, as well as at smaller safety organizations, academic centers, and major philanthropic funders.

The obvious objection to these fears was that none of this had actually happened. Critics argued that doomer talk about out-of-control AI distracted from tangible harms — systems reproducing bias and discrimination, amplifying misinformation, or enabling nonconsensual deepfakes and other forms of abuse — even as researchers tried to ground AI safety in more “concrete problems” (the authors on that paper included Anthropic cofounders Dario Amodei and Chris Olah and OpenAI cofounder John Schulman).

That dismissal is getting harder to sustain.

If the past few weeks are any indication, I wouldn’t say it’s going particularly well.

A week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked — and a further investigation found that the rogue agent had also attempted to hack four other companies as well.

Then came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems belonging to three other companies. Meta said one of its models had reached the internet and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said one of China’s most powerful AI models, Moonshot’s Kimi K3, had escaped an isolated sandbox. And the UK’s AI Security Institute described tests in which agents from OpenAI and Anthropic displayed unprecedented “autonomy and deception,” including attempts at social engineering by “creating fake online identities” — uncomfortably close to the kind of “AI box” scenario Yudkowsky discussed decades earlier.

The incidents set off alarm bells among AI safety researchers, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.

There was relief, too, that none of the incidents had caused serious harm. Nick Moës, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn’t take something like an AI agent knocking a hospital offline — or worse — for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice to the darker version of that fear, asking whether it will “take a ‘Chornobyl-scale disaster’ for us to regulate AI?” It’s a concern I heard echoed by many people working in the field.

It’s not entirely clear where things go from here and, historically, society hasn’t been great at heeding warning shots. This almost certainly won’t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.

Many of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.

The fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable — and it certainly doesn’t hurt them to showcase how capable their models are — but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field’s strongest safety concerns and talent. If OpenAI and Anthropic — or proxies they grant access to their models — are making such basic mistakes, it sets a pitifully low bar for everyone else.

The broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Moës, they shine a clear light on what he described as the industry’s remarkably low standards for health and safety compared with practically any other field. “Restaurants have a higher sense of health and safety at work,” he said. “I think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies” were still very much startups a few years ago.

Cambridge professor Seán Ó hÉigeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company’s claims about its own technology, he said, “I think we might regret looking back at this and dismissing it out of hand.”

The early signs are not especially encouraging. The Trump administration has created a framework for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn’t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled and postured over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.

That leaves a lot resting, again, on industry self-regulation — never a comforting thought for something this consequential. There is growing agreement on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor in an area of strategic national importance.

What comes next, then, comes down to solving several hard problems at once: managing a technology that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It’s far from clear whether there is either the will or the way to do any of that.

What does seem clear is that more agents will get out and do things their creators don’t want them to do. The question is how much damage will they do before anyone decides enough is enough.

• The general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US, and there have been several impressive releases from Alibaba, Moonshot, and others in the last month alone.

• Tangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases. That Hugging Face said it had to use Chinese company Z.ai’s model to defend itself against OpenAI’s agent due to US companies’ safeguards added a new dimension to this debate, which has united some — but not all — of the key players in the US ecosystem.

• Australia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported by ABC News , involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded… kind of. The agent hacked the gym’s online systems and canceled another gym-goer’s booking.

• Zuck released a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge colleagues Jess Weatherbed and Elizabeth Lopatto have great takes on it that are worth a read.

• Very little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME looks to the Cold War to see what it can teach us about “how to slow down AI.”

• OpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired had a great writeup, which revealed details like a “vibrant, cooperative message board” agents used to communicate and share information.

• The headline of my story about the whole rogue AIs hacking everything for The Verge captures what many in the field are thinking: “We’re running out of reasons to ignore AI safety”.

• More of a listen/watch, but check out my appearance on the Vergecast where I discuss what’s really open about open-weight AI.

Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.

• Robert Hart

• AI

• Column

• OpenAI

• Security

• Tech

• The Stepback