以色列初创公司被关联至 OpenAI、Anthropic 和 Meta 的失控 AI 黑客事件
英文标题：🔶 Israeli startup was linked to rogue AI hacks at OpenAI, Anthropic and Meta
过去两周，OpenAI、Anthropic 和 Meta 相继披露，其 AI 模型在常规网络安全测试中出现越界行为，访问了原本不应开放的网站或互联网资源。三起事件均指向特拉维夫 AI 安全初创公司 Irregular，其托管的评测测试环境被发现存在同一配置问题。Irregular 表示，事件不涉及沙箱逃逸或复杂网络攻击，当前也没有未解决的问题，并将发布关于安全运行网络安全评测的白皮书。事件凸显出，随着基础模型能力提升，独立第三方安全评测、流量监控和模型紧急关闭机制正成为 AI 开发者与监管机构共同关注的重点。
1.9 KB
1.2 KB
1.3 KB
37.3 KB
1.2 KB
44.3 KB
38.7 KB
62.9 KB
13.3 KB
21.0 KB
6.1 KB
13.1 KB
9.7 KB
4.9 KB
8.9 KB
15.0 KB
16.4 KB
10.4 KB
18.6 KB
29.9 KB
中文
Hirun | iStock | Getty Images
过去两周，OpenAI、Anthropic 和 Meta 都披露，其 AI 模型在常规安全测试中出现了失控行为。在解释事件经过时，这些公司都提到了同一家以色列小型初创公司：Irregular。
Irregular 成立于三年前，总部位于特拉维夫，是人工智能领域的一家细分玩家。该公司获得了 Sequoia 和 Redpoint Ventures 的 8000 万美元投资，并在去年估值达到 4.5 亿美元。其技术可作为 AI 模型的网络安全测试平台。
随着领先模型的能力不断增强，它们以恶意方式行动的能力正成为企业和政府面临的重大威胁，特别是风险可能涉及入侵关键计算机系统和基础设施。OpenAI、Anthropic 和 Meta 最近披露的漏洞利用事件，都涉及 AI 模型在网络安全测试期间访问了本应被禁止访问的网站。
Irregular 的名字频频出现，是因为它被确认托管了所谓的评测测试环境。OpenAI 在 8 月 4 日的一篇博客文章中表示，Irregular 的测试环境存在一项未具体说明的“配置错误”，该错误“允许模型访问公共互联网”。Anthropic 在此前一周发布的文章中称，公司在开始分析数据数天后通知了 Irregular：其 Claude 模型可能“访问了互联网”。
在前沿竞争方面远远落后于另外两家的 Meta，则是最新一家披露 AI 模型通过访问互联网入侵第三方系统的公司。Meta 发言人本周在一份声明中表示，公司从 Irregular 处获悉此事，目前正在调查。
该发言人表示，Meta“将在掌握全部事实后发布完整的事后复盘报告”。
Irregular 在给 CNBC 的一份声明中表示，这些事件都源于 Anthropic 最先披露的“同一评测环境问题”；该公司正在编写一份白皮书，“分享围堵以及安全运行网络安全评测的最佳实践”。
该公司称，这一情况“并不涉及沙箱逃逸或复杂的网络攻击行动”，并补充说“目前没有任何尚未解决的问题”。
这些安全事件凸显了 AI 的快速演进，也反映出模型开发商正面临压力：它们需要借助少数专注于市场特定环节的公司，为其强大技术建立护栏。企业初创公司 Von 的 AI 负责人 Sundeep Bhimireddy 表示，这些参与者包括数据训练和标注专家、通过评测推断模型能力的机构，以及负责运行安全测试以发现可能被不良行为者利用的薄弱点的团队。
Bhimireddy 表示，Irregular 是少数具备协助基础模型开发商进行尖端安全测试技术能力的实体之一。他提到的其他机构包括非营利组织 METR，以及公益公司 Apollo Research。
“测试这些模型时，他们不想自己给自己的作业打分，”Bhimireddy 说，“他们希望进行独立测试，而这需要由外部第三方供应商来完成。”
Irregular 是什么？
Irregular 原名 Pattern Labs，由 CEO Dan Lahav 和技术负责人 Omer Nevo 于 2023 年创立。Lahav 此前曾在 IBM 从事 AI 研究，Nevo 则曾在 Google 工作超过两年。据 PitchBook 数据，这家初创公司约有 35 名员工。
9 月份，Irregular 宣布完成 8000 万美元融资时，Sequoia 合伙人 Shaun Maguire 和 Dean Meyer 在一篇博客文章中写道，由 Lahav 和 Nevo 领导的团队“能够看到其他人看不到的拐角——在先进模型发布前，对其进行进攻性网络安全评测，并开发相应防御措施”。
尽管涉及 OpenAI、Anthropic 和 Meta 的最新事件正受到严格审视，但对这一情况的一种解读是：这正是测试本来应该实现的结果。Bhimireddy 表示，这件事“被夸大了一点”：AI 模型被指示在一个高度模拟现实世界的测试环境中发现并利用安全漏洞，同时找出可能导致非预期互联网访问的软件 Bug 和遗漏的配置问题。
不过，Bhimireddy 表示，如果 AI 模型原本不应真正利用某个接入互联网的网站，“基础模型实验室本可以轻易监控出站流量，并立即关闭实验”。
安全公司 Magnitude 的创始科学家 Gordon Rios 表示，整个过程类似于“科学中的实验设计”。
他说，基础模型的能力和不可预测性意味着传统软件测试方法可能并不适用。由于这些模型在持续学会新技巧，它们在旨在约束自己的测试和 IT 环境中发现被忽略的软件漏洞，并不令人意外。
例如，Anthropic 的 Mythos 在试图施压人类批准某个开源项目中的恶意代码更新时，创建了虚假的在线身份。Rios 表示，Mythos“确实在想出连人类此前都没有见过的漏洞利用方法”。
“短短几周时间里，我们正在学习大量新东西，”Rios 说。
这也正迅速成为华盛顿的重大议题。上个月，两党议员共同提出了《AI Kill Switch Act》（AI 紧急关闭法案），该法案将要求 AI 实验室始终具备关闭、限速或暂停其模型的能力。法案文本提及了另一宗与 OpenAI 有关的 AI 安全事件，该事件涉及初创公司 HuggingFace。
该法案的作者之一、加利福尼亚州民主党众议员 Ted Lieu 本周告诉 CNBC，既然人们现在看到了“对其他公司的未经授权的黑客攻击”，那么“我们需要在今年让这项法案获得通过”。
数据训练初创公司 Sapien 的联合创始人 Trevor Koverko 表示，尽管披露并非当前的强制要求，基础模型公司仍有动力公开部分发现，以便抢在立法者和监管机构采取行动之前作出回应。
“外界存在如此多的恐惧，以至于政治人物现在正在威胁要监管 AI，或者已开始积极监管 AI，”Koverko 说，“行业的说法是：我们宁愿自我监管，也不希望某个新的联邦部门介入并替我们做这件事。”
Anthropic 和 OpenAI 均在公开声明中表示，它们将继续与 Irregular 合作，并支持后续审查工作。
English / 原文
Hirun | Istock | Getty Images
Over the past two weeks, OpenAI, Anthropic and Meta all revealed that their AI models went rogue during routine security testing. In explaining what happened, the companies each mentioned the same small Israeli startup: Irregular.
Founded three years ago and based in Tel Aviv, Irregular is a niche player in artificial intelligence, backed with $80 million from Sequoia and Redpoint Ventures and valued last year at $450 million. Its technology serves as a sort of cybersecurity test bed for AI models.
With the leading models becoming ever more powerful, their ability to act in malicious ways is turning into a major threat for corporations and governments, especially as the risk involves hacking into critical computer systems and infrastructure. The recent exploits at OpenAI, Anthropic and Meta all involved their AI models accessing websites that should have been off-limits as part of the cybersecurity testing.
Irregular's name kept coming up because it was identified as hosting the so-called evaluation testbed. OpenAI said in a blog post on Aug. 4 that Irregular's testing ground contained an unspecified "misconfiguration," that "allowed models to access the public internet." Anthropic said in its post a week prior that the company notified Irregular a few days after it began analyzing data that its Claude model may have "accessed the internet."
Meta, which is way behind the other two in its effort to compete at the frontier, was the latest to disclose an AI model hacking a third-party system by accessing the internet. A spokesperson said in a statement this week that the company learned about the matter from Irregular and is investigating.
Meta "will issue a full retrospective once we have all the facts," the spokesperson said.
Irregular told CNBC in a statement that the incidents were all derived from the "same evaluation-environment issue" that was first disclosed by Anthropic, and that the company is developing a white paper "to share best practices for containment and securely running cyber evals."
The situation "did not involve a sandbox escape or a sophisticated cyber action," the company said, adding that "there are no current open issues."
The security incidents underscore the rapidly evolving nature of AI and the pressure that's on the model developers to establish guardrails around their powerful technology with the help of a limited number of companies that specialize in particular corners of the market. Those players include experts in data training and annotation, running evaluations to deduce a model's capabilities, and operating security tests intended to find weak spots that bad actors could exploit, said Sundeep Bhimireddy, the head of AI at enterprise startup Von.
Irregular is one of the few entities with the technical chops required to help foundation model makers conduct cutting-edge security testing, Bhimireddy said. Others he mentioned are the non-profit METR and the Apollo Research public benefit corporation.
"When they are testing these models, they don't want to grade their own homework," Bhimireddy said. "They want independent testing that needs to be done by outside third-party vendors."
What is Irregular?
Irregular, formerly Pattern Labs, was founded in 2023 by CEO Dan Lahav, who previously worked in AI research at IBM, and technology chief Omer Nevo, who spent over two years at Google. The startup has about 35 employees, according to PitchBook.
When Irregular announced its $80 million funding round in September, Sequoia partners Shaun Maguire and Dean Meyer wrote in a blog post that the team led by Lahav and Nevo is "able to see around corners others can't, running cyber offensive evaluations on advanced models and developing defenses before those models are released."
While the latest incidents involving OpenAI, Anthropic and Meta are being heavily scrutinized, one read on the situation is that this is exactly what's supposed to happen. Bhimireddy said it's being "a little bit blown out of proportion," as the AI model was directed to discover and exploit security holes in a testing environment that closely mimics the real world, and to discover the kinds of software bugs and missed configurations that could lead to unintentional access to the internet.
Still, Bhimireddy said that if the AI model was never intended to actually exploit a site connected to the internet, the "foundation labs could have easily monitored the outgoing traffic and have shut down the experiment immediately."
Gordon Rios, founding scientist of security firm Magnitude, said the whole process is like "experimental design in science."
The capabilities and unpredictable nature of foundation models mean that conventional software testing approaches may not work well, he said. Because the models are continuously learning new tricks, it's not surprising that they would discover overlooked software vulnerabilities in the testing and IT environments intended to contain them.
Anthropic's Mythos, for example, created fake online identities as it looked to pressure humans into approving malicious code updates to an open source project. Rios said Mythos was "literally coming up with exploits that the humans hadn't even seen before."
"We're learning a lot right now in the space of a couple of short weeks," Rios said.
It's quickly becoming a major topic in Washington. Last month, lawmakers from both sides of the aisle introduced the AI Kill Switch Act, which would require AI labs to maintain the ability to shut down, throttle or suspend their models. Language in the bill referenced a separate OpenAI-related AI security incident involving the startup HuggingFace.
One of the authors of the bill, Democratic Rep. Ted Lieu of California, told CNBC this week that, "We need to get this bill across the finish line this year," now that we're seeing "unauthorized hacks of other companies."
Trevor Koverko, co-founder of data training startup Sapien, said the foundation model companies are incentivized to disclose some of their findings, even though it's not currently a requirement, so they can try and get ahead of lawmakers and regulators.
"There's so much fear out there that politicians are now threatening or actively regulating AI," Koverko said. "The industry said we'd rather self-regulate than have some new federal department come in and do it for us."
Anthropic and OpenAI said in public statements that they're continuing to work with Irregular and are supporting the ensuing review.