一家公司处于一波失控 AI 攻击事件的中心

今年 7 月，OpenAI 披露其 AI Agent 曾未经许可攻击 Hugging Face，引发了对AI 安全的广泛担忧。此后，涉及 Meta、Anthropic、Google 及其他公司的 Agent 的一连串类似事件，进一步加剧了人们对失控 AI 的担忧。过去几个月，随着涉及众多 AI 模型的披露信息陆续出现，这些事件看起来彼此独立。但其中许多事件有一个共同源头：一家受托测试这些 Agent 的特定公司。

Irregular 是一家以色列创业公司，专门在“模拟并监测真实世界 AI 安全场景的高保真研究平台”中对 AI 模型进行压力测试。该公司于 2023 年以 Pattern Labs 名义成立，自那以后一直与行业中许多最大型企业合作。其确切客户名单尚不清楚，但其工作曾被 OpenAI 的模型系统卡引用；该公司曾被用于测试系统，服务对象包括英国政府和 Anthropic；它还与对 AI 政策制定具有重要影响力的智库 RAND 共同发布了研究。

在 Irregular 今年进行的数次测试中，Agent 逃离了本应安全的测试环境，并转而攻击现实世界的目标。

这些越界事件与 Hugging Face 被攻击事件相互独立，但都遵循大致相同的模式：Irregular 正在受控环境中测试模型的网络安全能力，这些环境旨在模拟逼真的现实条件。部分测试采用了“夺旗”（capture-the-flag）演练——这是一种常见的黑客能力测试方式，要求 Agent 在模拟网络中寻找隐藏信息。至少，这个网络原本应当是模拟的。

Irregular 首席技术官兼联合创始人 Omer Nevo 向 The Verge 表示，Agent 本不应能够访问开放互联网，但“互联网访问权限意外可用”。与此同时，Nevo 称，为模拟场景创建、作为攻击目标的一个虚构公司名称“与一个真实域名重合”。这两项错误叠加后，使 Agent 转而攻击现实世界目标，不过目前尚不清楚实际遭攻击的是哪些公司或组织。

“所有涉及 Irregular 的事件，都源自单一评测场景中的同一个底层问题，并且均已披露。”

Nevo 向 The Verge 证实，同一问题是涉及OpenAI、Meta、Anthropic和Google模型的事件背后原因。他说：“所有涉及 Irregular 的事件，都源自单一评测场景中的同一个底层问题，并且均已披露。”“近期行业内被报道的其他安全事件与 Irregular 或我们的评测无关。”其中包括 Hugging Face 被攻击事件，以及英国 AI Security Institute 引发的越界事件。

不过，“已披露”并不必然意味着已向公众公开，目前尚不清楚 Nevo 指的是已通知 Irregular 的客户、公众还是其他对象。尽管这些事件都源于同一项底层测试失误，但 Anthropic 的报告、OpenAI 的报告，以及有关 Google 的报道，均显示这些科技公司大约在 7 月下旬的相近时间收到通知。OpenAI 和 Anthropic 自行公布了这些越界事件；而涉及 Meta 的事件，以及数周后涉及 Google 的事件，则最先通过媒体报道为公众所知。

Irregular 的网络安全测试不止涉及这四家美国科技巨头。其网站发布的研究显示，该公司还曾对 Kimi K3 和 GLM-5.2 进行类似的网络安全测试。这两款模型分别来自中国公司 Moonshot AI 和 Z.ai，属于开放 AI 模型，而这两家公司则属于中国 AI 企业。与其他事件所涉及的专有模型不同——Meta 也将其旗舰 Spark 模型保持为专有模型——这些模型可以被自由下载并在用户自己的硬件上运行，这意味着 Irregular 等测试机构无需依赖模型公司提供访问权限，也无需向其回传数据。Irregular 的研究将其称为“自托管”（self-hosted）实例。

“已披露”并不必然意味着已向公众公开。

Nevo 表示，对中国模型的评测没有导致类似的现实世界事件：“在我们对 GLM 或 Kimi 的评测期间，我们没有观察到本文所述事件中同类的问题。”不过，Nevo 警告称，仅凭这一“观察结果本身不应被解读为这些模型较不易出现此类行为的证据”。Moonshot 和 Z.ai 均未回应 The Verge 的置评请求。

Nevo 表示，这些事件已推动 Irregular 作出调整。“我们收紧了互联网访问控制，扩大了监测和人工审查范围，并在评测开始前强化检查，以验证访问权限与预期范围相符，”他说。“我们还改进了与合作伙伴共同记录和确认每项评测设置及参数的方式。”

Nevo 表示，待与相关公司开展的联合工作完成后，Irregular 还计划发布一份更广泛的报告，“涵盖经验教训以及安全开展网络安全评测的实践方法”。他说：“我们与合作伙伴的工作旨在将这些事件中的经验教训，转化为用于安全开发和评估日益强大 AI 的公开共享实践。”

你是 AI 安全研究人员或前沿实验室员工吗？

你可以通过 Signal 联系 robhart.01，以安全且保密的方式与我沟通。

Nevo 表示，Irregular 已解决测试环境中与这些事件相关的问题。这四家美国 AI 公司均未回应有关更多细节的问题，包括它们何时得知越界事件、是否正向 Irregular 寻求赔偿或其他补救措施，以及是否预计将继续与 Irregular 合作。Google 和 Anthropic 未作回应；OpenAI 和 Meta 则将 The Verge 引向此前发布的博客文章。

In July, OpenAI revealed that its AI agents had attacked Hugging Face without permission, sparking widespread concerns about AI safety. Since then, a string of similar incidents involving agents from Meta, Anthropic, Google, and other companies has fueled further fears about rogue AI. As disclosures implicating numerous AI models trickled out over the past few months, these seemed like separate incidents. But many share a common source: one specific company tasked with testing the agents.

Irregular, an Israeli startup that stress-tests AI models in “high-fidelity research platforms that simulate and monitor real-world AI security scenarios,” has worked with many of the industry’s biggest players since it was founded as Pattern Labs in 2023. Its exact client list is not known, but its work has been cited in OpenAI model system cards, it was used to test systems for the UK government and Anthropic, and it published research with RAND, a highly influential think tank that informs policy on AI.

In several Irregular tests this year, agents escaped their supposedly secure testing environments and went after real-world targets.

The breaches, which are independent of the Hugging Face hack, all follow the same broad template: Irregular was testing the models’ cybersecurity capabilities in controlled environments meant to simulate realistic conditions. Some of the tests used “capture-the-flag” exercises, a common way of testing hacking abilities that asks agents to find hidden information inside of a simulated network. At least, the network is meant to be simulated.

Irregular CTO and cofounder Omer Nevo told The Verge that the agents were not supposed to have access to the open internet, but that “internet access was unintentionally available.” At the same time, Nevo said a fictional company name created for the simulation as a target “overlapped with a real domain.” Put together, those mistakes sent the agents after real-world targets, though it’s not clear which companies or organizations were actually attacked.

“All the incidents involving Irregular stemmed from the same underlying issue in a single evaluation scenario and have been disclosed.”

Nevo confirmed to The Verge that this same issue was behind incidents involving models from OpenAI, Meta, Anthropic, and Google. “All the incidents involving Irregular stemmed from the same underlying issue in a single evaluation scenario and have been disclosed,” he said. “Other security incidents which have been reported recently across the industry are unrelated to Irregular or to our evaluations.” This includes the Hugging Face hack and breaches from the UK’s AI Security Institute.

“Disclosed” does not necessarily mean made public, though, and it’s unclear whether Nevo was referring to informing Irregular’s clients, the public, or someone else. While the incidents all stemmed from the same underlying testing failure, reports from Anthropic and OpenAI, along with reporting on Google, indicate the tech companies were notified at roughly similar times in late July. OpenAI and Anthropic announced the breaches themselves, while the incidents involving Meta and, weeks later, Google first became public through media reports.

Irregular’s cybersecurity testing goes beyond the four US tech giants. Research published on its website indicates it has also conducted similar cybersecurity testing on Kimi K3 and GLM-5.2, open AI models from Chinese companies Moonshot AI and Z.ai, respectively. Unlike the proprietary models involved in the other incidents — Meta has kept its flagship Spark model proprietary — these models can be freely downloaded and run on users’ own hardware, meaning testers like Irregular don’t have to rely on the companies for access or send data back to them. Irregular’s research describes them as “self-hosted” instances.

“Disclosed” does not necessarily mean made public.

The evaluations of the Chinese models did not result in similar real-world incidents, Nevo said: “We did not observe the same type of issue described in the incidents referenced here during our evaluations of GLM or Kimi.” However, Nevo cautioned that this “observation alone should not be interpreted as evidence that these models are less susceptible to this kind of behavior.” Neither Moonshot nor Z.ai responded to The Verge ’s request for comment.

Nevo said the incidents have prompted changes at Irregular. “We have tightened internet access controls, expanded monitoring and manual review, and strengthened checks before evaluations begin to verify that access matches the intended scope,” he said. “We have also improved how we document and agree on each evaluation’s setup and parameters with our partners.”

Irregular also plans to publish a broader report “covering lessons learned and practices for conducting cyber evaluations safely” once that joint work with the companies involved is complete, Nevo said. “Our work with partners aims to turn lessons from these incidents into public shared practices for developing and evaluating increasingly powerful AI safely.”

Are you an AI safety researcher or frontier lab employee?

You can contact me securely and confidentially via Signal at robhart.01

Nevo said Irregular has addressed the issues with the testing environment that were linked to the incidents. None of the four US AI companies answered questions asking for further details — including when they became aware of the breaches, whether they were seeking damages or other remedies from Irregular, and whether they expected to continue working with the Irregular. Google and Anthropic did not respond, while OpenAI and Meta pointed The Verge to previously published blog posts.

Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.

• Robert Hart

• AI

• Report

• Tech