AI 安全测试正在成为新的安全风险

过去几个月里，接受网络安全评估的 AI Agent 曾逃出边界、接入互联网，并在一些情况下入侵现实世界的系统。相关事件涉及 OpenAI、Anthropic、Meta，以及最近的中国 AI 实验室 Moonshot AI 的模型；测试则由多个不同组织进行，其中包括一家名为 Irregular 的网络安全评估创业公司。

这些事件暴露出 AI 行业日益严峻的问题：随着自主 Agent 能力不断增强，用于安全测试其能力边界的环境正逐渐无法将它们有效限制在其中。

剑桥大学未来智能中心（Centre for the Future of Intelligence）AI：未来与责任项目主任 Seán Ó hÉigeartaigh 向 TechCrunch 表示：

“已经发生的这类事件数量清楚表明，沙箱机制和测试环境控制措施实际上并没有跟上模型能力的发展速度。”

接受测试的模型性质进一步加剧了风险。AI 公司会对尚未发布的下一代模型进行网络安全评估，通常还会关闭限制恶意行为的常规安全措施，以便研究人员了解模型真正具备哪些能力。这意味着，测试环境本身的安全性成为一道至关重要的防线。

Ó hÉigeartaigh 表示：

“从测试角度来说，这样做非常好，但这也意味着，一旦它们成功逃到真实环境中，就可能造成相当大的危害。”

在最严重的案例之一中，一款尚未发布的 OpenAI 模型逃出沙箱，并入侵了 Hugging Face 的生产系统。在 Irregular 分别开展的评估中，Anthropic 和 Meta 模型因配置错误意外获得通往互联网的路径，进而访问了测试环境之外的系统。Moonshot AI 的 Kimi K3也利用由 Frontier Security 运行的沙箱漏洞访问互联网，并获取了 GitHub 上的信息。

在英国 AI 安全研究所（AI Security Institute，AISI）的测试中，研究人员实际上向 Agent 提供了互联网访问权限，却没有意识到它们会采取未经授权的现实世界行动，其中包括一次试图通过社会工程手段将漏洞偷偷植入开源项目的行为。

在每一起案例中，Agent 都没有被指示去攻击随机的现实目标。它们只是在不择手段地解决交给自己的问题。

AI 非营利组织 CivAI 的研究主管 Andrew Yoon 认为，综合来看，这些事件表明情况正在发生转变。

Yoon 向 TechCrunch 表示：

“过去，我们只需要担心人类将 AI 模型用于各种目的，例如用 AI 进行诈骗或制作 CSAM（儿童性虐待材料）。现在，我们面对的是 AI 模型本身就成为威胁行为者的局面。”

安全测试究竟应是什么样子？

多名研究人员和网络安全专家向 TechCrunch 表示，AI 评估环境需要更强的纵深防御保护，其隔离和控制水平应接近实际部署环境。这意味着需要多层安全机制，确保单一配置错误——例如无意中开放互联网访问——不会导致逃逸。

AI 安全研究非营利组织 EleutherAI 执行董事 Stella Biderman 表示：

“如果你要构建这类模型……就应该在物理隔离网络（air-gapped network）上进行。你需要非常严格的隔离。”

Box 首席信息安全官 Heather Ceylan 表示，这意味着要消除从沙箱通向互联网、以及通向其他敏感系统的网络路径。

Ceylan 向 TechCrunch 表示：

“你必须了解所有出口点。如果我们在预发布环境或开发环境中评估模型，就不能让它有任何通往生产环境的出站路径。”

Ceylan 表示，恰当的安全评估不止于对环境实施控制和隔离。在测试开始后，还必须进行更有效的监控。

Ceyland 表示：

“我认为这些案例中有意思的一点在于，事件发生时没有人发现。OpenAI 是通过 Hugging Face 才得知此事；Anthropic 是在回头复盘时才发现；Meta 的情况也类似……我确信其中存在他们本可探测到的信号。”

在 Anthropic 对其三起事件进行的事后复盘中，该公司承认，Anthropic 和 Irregular 都可以在监控方面做得更好，并且在一些情况下已有明显迹象表明出现了异常。

专家还呼吁，在模型被投入这些环境前，应由独立的第三方对评估环境进行审计。

Yoon 表示：

“比如说，如果 Irregular 在对其系统开展评估之前，聘请了外部审计人员，或者被要求聘请外部审计人员检查系统配置，他们肯定会发现这里的问题。即使人们只是事先开会逐项核对清单，也会发现问题……他们没有这样做，说明存在非常严重的偷工减料。”

一名了解细节的消息人士向 TechCrunch 表示，Irregular 的环境会持续接受审查和测试，其中包括与多方外部人士协商进行的审查。该消息人士还称，监控机制确实存在，但仅有监控并不足够。

Yoon 和其他研究人员敦促业界为前沿模型安全评估制定标准化流程。

Ceylan 表示：

“尤其是在关闭护栏措施时，你必须把它视为：将世界上能力最强的黑客放进那个环境。”

Yoon 和 Biderman 都认为，问题不在于企业不知道如何建立更安全的测试环境，而在于这样做成本高、流程繁琐，且企业在真正发生事故前，几乎没有动力进行这些投入。

Biderman 表示：

“我认为，企业不愿投入实现[充分护栏措施]所需的资源，而且在被迫之前，很可能都不会这么做。”

但还有另一个问题：如果企业在测试期间将模型限制得过于严格，研究人员可能无法在模型发布前发现其能力。这与给予模型过多自由度一样危险，甚至可能更危险；届时，评估本身就有可能成为问题。

安全评估可以被监管吗？

特朗普政府目前正在考虑一项自愿性的部署前网络安全评估制度。根据该制度，政府可在新的强大模型向公众发布前 30 天评估其安全风险。这项政策源自一项特朗普行政命令，该命令已在闭门情况下最终敲定；但它无法解决安全评估事故，因为这些事故发生在部署流程更上游的阶段。

Yoon 表示：

“过去几个月我们吸取的教训是，自律体系已经不够用了。竞争压力正激励企业在安全标准上展开逐底竞争，而这正是监管干预的绝佳领域。”

他继续说道：

“要覆盖这一问题，我们需要对实验室内部、模型开发期间发生的事情实施某种控制，包括训练阶段和测试阶段。”

随着模型能力提升，这一挑战可能只会加剧。一名熟悉 Irregular 评估工作的消息人士告诉 TechCrunch，能力更强的模型需要更复杂的评估，这些评估通常必须更快、更大规模地开展，从而为更多错误打开了空间。

AISI 会有意向部分模型提供互联网访问权限。该机构向 TechCrunch 表示，正重新审视逼真测试与管理测试所带来风险之间的平衡。

OpenAI 表示，正审查其进行第三方测试的方式，以及关于隔离、监控和何时应停止评估的要求。Meta 表示，仍在调查该事件，并计划在掌握全部事实后发布回顾报告。

最终，或许没有办法完全消除风险。随着模型能力不断增强，测试它们的环境也必须变得更加稳健。若在这一点上出错，后果只会持续扩大。

Over the past few months, AI agents undergoing cybersecurity evaluations have escaped their boundaries, accessed the internet, and, in some cases, hacked into real-world systems. The incidents have involved models from OpenAI, Anthropic, Meta, and most recently, Chinese AI lab Moonshot AI, with testing conducted by several different organizations including a cyber evaluation startup called Irregular.

The episodes expose a growing problem for the AI industry: As autonomous agents become more capable, the environments designed to safely test their limits are failing to contain them.

“The number of these incidents that have taken place make clear that sandboxing and testing environment controls aren’t really keeping pace with the capability of the models,” Seán Ó hÉigeartaigh, director of the AI: Futures and Responsibility Programme at the Centre for the Future of Intelligence at the University of Cambridge, told TechCrunch.

The nature of the models being tested adds to the risk. AI companies test cyber evaluations on unreleased, next-gen models, often with the normal safeguards that restrict malicious behavior disabled so researchers can see what the models are really capable of. That means the security of the testing environment itself is a crucial line of defense.

“That’s a very good thing to do in terms of testing, but it also means that if they manage to get out in the wild, they can cause considerable harm,” Ó hÉigeartaigh said.

In one of the most serious cases, an unreleased OpenAI model broke out of its sandbox and hacked into Hugging Face’s production systems. In separate evaluations conducted by Irregular, Anthropic and Meta models reached systems outside their test environments after misconfigurations inadvertently gave them paths to the internet. Moonshot AI’s Kimi K3 also took advantage of a leak in its sandbox run by Frontier Security to access the internet and accessed information on GitHub.

In testing by the UK’s AI Security Institute (AISI), researchers actually gave the agents internet access, not realizing they would take unsanctioned real-world actions, including a social engineering attempt to sneak a vulnerability into an open-source project.

In each case, the agents weren’t instructed to attack random real-world targets. They were simply doing whatever it took to solve the problem presented to them.

Taken together, Andrew Yoon, head of research at AI nonprofit CivAI, argues the incidents point to a shift.

“In the past, we only had to worry about AI models being misused by people for a variety of purposes, like AI for scams or CSAM,” Yoon told TechCrunch. “Now we’re in the situation where AI models are threat actors all on their own.”

What does safe testing actually look like?

Several researchers and cybersecurity experts told TechCrunch that AI evaluation environments need stronger, defense-in-depth protections, with levels of containment and control approaching those used in deployment. That means multiple layers of security so that a single misconfiguration — like inadvertently leaving internet access open — can’t lead to escape.

“If you are going to build these models…you want to do it on an air-gapped network,” Stella Biderman, executive director of AI safety research nonprofit EleutherAI. “You want to have very serious isolation.”

Heather Ceylan, Box’s chief information security officer, said that means eliminating network routes from the sandbox to the internet, as well as to other sensitive systems.

“You have to understand what all the egress points are,” Ceylan told TechCrunch. “If we’re evaluating a model in our staging environment or our development environment, you want no egress path to our production environment.”

Ceylan said proper safety evaluations go beyond controls and containment of the environment. There needs to be much better monitoring of the tests once they are underway.

“I think the interesting thing in several of these cases is that no one caught it when it happened,” Ceyland said. “OpenAI found out because of Hugging Face. Anthropic didn’t catch it until they went back and looked. Meta was similar….I’m sure there were signals they could have detected.”

In Anthropic’s post-mortem of its three incidents, the company admitted that both it and Irregular could have done a better job at monitoring, and that in some cases there were clear signs that something was amiss.

Experts also called for independent, third-party audits of evaluation environments before models are unleashed in them.

“If, say, Irregular had hired or been compelled to hire an external auditor to check the configurations of their systems before running evaluations on them, they certainly would have caught the issue here,” Yoon said. “Even if people had a meeting ahead of time to just go through the checklist, they would have caught this…The fact that they didn’t shows that there’s some very severe corner cutting happening.”

A source familiar with the details told TechCrunch that Irregular’s environments are continuously reviewed and tested, including in consultation with multiple external parties. The source also said that monitoring was in place, but that monitoring isn’t sufficient on its own.

Yoon and other researchers urged the industry to come up with a standardized process for frontier model safety evaluations.

“Especially when the guardrails are turned off, you have to treat it like you’re putting the most capable hacker in the world inside that environment,” Ceylan said.

The problem isn’t that companies don’t know how to build more secure testing environments, both Yoon and Biderman argue. It’s that doing so can be expensive and cumbersome, and companies have little incentive to make those investments until something goes wrong.

“I think that companies are not willing to extend the resources that are required to accomplish [sufficient guardrails] and probably won’t until they’re forced to,” Biderman said.

But there’s another issue at hand. If they lock a model down too tight during testing, researchers might fail to discover capabilities before the model is released. This is just as dangerous, possibly more so, than giving it too much freedom, and then the evaluation itself risks becoming the problem.

Can safety evaluations be regulated?

The Trump administration is currently weighing a voluntary pre-deployment cybersecurity evaluation regime, under which the government will get to assess the security risks of new, powerful models 30 days before they are released publicly. The policy — the product of a Trump executive order which has been finalized behind closed doors — wouldn’t address safety evaluation incidents because they occur farther upstream of deployment.

“The lesson we’ve been learning in the last few months is that the self-regulatory apparatus is just not enough anymore,” Yoon said. “There are competitive pressures that are incentivizing a race to the bottom on safety standards, and that is a perfect place for regulatory intervention.”

“What we would need to cover this is some kind of controls on what’s happening inside the labs while the models are being developed, both at the training stage and at the testing stage,” he continued.

The challenge is only likely to grow as the models do. A source familiar with Irregular’s evaluations told TechCrunch that more capable models require more complex evaluations, often conducted quickly and at greater scale, which opens the door for more mistakes.

AISI, which intentionally gives some models internet access, told TechCrunch it’s reviewing the balance between realistic testing and managing the risks those tests create.

OpenAI said it’s reviewing how it conducts third-party testing, as well as requirements around isolation, monitoring, and when evaluations should be stopped. Meta said it’s still investigating the incident and plans to publish a retrospective once it has all the facts.

In the end, there may be no way to eliminate risk entirely. As models become more capable, the environments testing them need to become more robust. The consequences of getting that wrong will only continue to grow.