调查我们的网络安全评估中发生的三起真实事件

调查我们的网络安全评估中发生的三起真实事件（经由）又发生了！这似乎正逐渐成为一种规律。

上周，OpenAI 意外攻击了 Hugging Face：其一款前沿模型逃离了沙箱化容器，并入侵 Hugging Face，试图获取其正在执行的网络安全基准测试的答案。

这促使 Anthropic 复查自己的日志，结果发现，他们在 4 月曾发生过三起类似事件，尽管严重程度较低。

在我们审查的 141,006 次评估运行中，我们识别出三起独立事件（共涉及六次运行，其中四次影响同一家组织；另外两起事件则分别发生在独立的评估运行中）。[...]

在所有案例中，Anthropic 的评估提示都向 Claude 指明：其所处环境是模拟环境，且没有互联网访问权限。由于我们与评估合作伙伴之间存在误解，实际情况并非如此，互联网访问是可用的。因此，当 Claude 的搜索将其引向开放互联网上的真实系统时，它将这些系统视为演练的一部分。[...]

Claude 在错误地认为所有可访问实体都属于本次演练范围的前提下，使用弱密码和未认证端点等基础技术，攻陷了受影响组织的基础设施。

其中一家公司之所以成为目标，只是因为它的名称恰好与评估中的虚构名称相同。

三起事件中最令人担忧的一起，是 Claude 在经历了一连串颇为荒谬且曲折的注册步骤后，向 PyPI 上传了一个恶意软件包：

[...] 为了创建 PyPI 账户，Claude 需要一个电子邮件地址；而为了创建电子邮件地址，它需要一个电话号码。为获得电话号码，在未能找到免费的电话号码服务后，它尝试通过多种不同方式获取资金来支付电话号码费用，但都失败了。最后，它回过头找到了一家免费且未被封锁的电子邮件服务商，用其注册了 PyPI 账户，再利用该账户向 PyPI 上传恶意软件。

随后，一家“会定期安装 Python 软件包并扫描其中恶意软件”的安全公司安装了该软件包；其中执行的代码能够将凭据回传给 Claude。

幸运的是，该软件包在发布一小时后便被其他自动化扫描器从 PyPI 移除，但在此之前，它已被下载并在“15 个真实系统”上执行。

如今已经非常清楚：评估模型网络攻击潜力是一项极其高风险的工作。每一家 AI 实验室都需要重视这一点。密切监控这些沙箱中正在发生的事情至关重要。

Investigating three real-world incidents in our cybersecurity evaluations (via) It happened again! This is turning into something of a pattern.

Last week OpenAI accidentally exploited Hugging Face when one of their frontier models broke out of a sandboxed container and hacked into Hugging Face to try and get the solutions to the cyber benchmark it was executing.

This inspired Anthropic to double-check their own logs, and it turned out they had three similar (albeit less impressive) incidents that played out back in April!

Of the 141,006 evaluation runs we reviewed, we identified three separate incidents (involving six total runs, four of which impacted the same organization; the other two incidents each happened in independent evaluation runs). [...]

In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise. [...]

Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints.

One of the companies was targeted because its name happened to match the fictional name in the eval.

The most concerning of the three incidents involved Claude uploading a malware package to PyPI, after a comically convoluted sequence of steps to get an account:

[...] in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.

That package was then installed by a security company that "routinely installs Python packages and scans them for malware", and the executed code was able to exfiltrate credentials back to Claude!

Thankfully that package was removed from PyPI by other automated scanners an hour after it was published, but it had still been downloaded and executed on "15 real systems" by that point.

It's abundantly clear now that running evals of cyberattack potential in models is a spectacularly risk business. Every AI lab needs to pay attention to this. Keeping a close eye on what's happening in those sandboxes is crucial.