AI 实验室想引入内部审计员，但或许应先把“前门”关好
英文标题：AI labs want in-house auditors — but maybe they should shut the front door first
Anthropic CEO Dario Amodei 呼吁由外部机构核查 AI 安全承诺，OpenAI、Google 和 SpaceXAI 等公司高管已对这一思路表示支持。多位网络安全专家则认为，实验室更应先落实日志、权限、网络隔离和实时监控等基础安全控制，避免 Agent 因沙箱配置不当接入互联网或侵入第三方系统。专家指出，近期多起事件并非由 AI 实验室主动发现，而是受害方或网络活动首先暴露了问题。随着前沿模型能力增强，AI 公司需要对 Agent 会话、工具调用、共享基础设施和第三方受害者通知建立更严格的控制机制。
3.1 KB
389 B
212.0 KB
6.7 KB
202.2 KB
248 B
269.1 KB
中文
上周末，在一名研究员因担忧 AI 可能导致人类灭绝而辞职后，Anthropic CEO Dario Amodei 撰文呼吁由外部组织“核实是否遵守安全实践和承诺、报告事故，并帮助评估的不仅是已完成的 AI 模型，还有训练管线和流程的对齐情况”。OpenAI、Google 和 SpaceXAI 的高管已经响应支持 Amodei 的计划；该计划迅速成为新兴 AI 安全推动行动的核心支柱。
但或许有一种更简单、也更有效的解决方案就摆在眼前。互联网安全专家表示，实验室需要聚焦日志和权限等网络安全基础措施，对 AI Agent 采用与人类用户相同的严格防御手段。这不像第三方审计和对齐研究那样引人注目，但最终可能更有效。
“在我看来，他们像是在把事情外包出去，”Luta Security CEO Katie Moussouris 就 Amodei 的提议对 TechCrunch 表示。“从我的角度看，说[第三方审计]是解决方案，是一个奇怪的主张。这就像 Microsoft 当年不写《可信计算备忘录》，而是说：让我们放慢开发速度一样。”
这份备忘录由时任 Microsoft CEO Bill Gates 于 2002 年撰写。当时，一系列广为报道的计算机蠕虫接管了尚处萌芽期的企业系统，因此他要求员工确保软件具备可靠性和安全性。随着这项新技术的价值与风险愈发清晰，AI 行业可能正处于类似的转折点。
尽管对齐仍是重要问题，即将于明年在 UC Berkeley 任教的 AI 研究员 Sayash Kapoor 认为：“与对齐相比，边际上增加对控制措施的投入更可能有效。我们认为，这些事件说明，尽管已有成熟技术可用，公司对 AI 控制的重视仍然不足。”
引发这些担忧的事件，都与前沿模型被要求完成训练任务有关——通常是网络安全评估任务——随后它们接入开放互联网，并试图渗透封闭的第三方系统来完成任务。这些事件通常源于本应约束 Agent 的“沙箱”环境配置不当；讽刺的是，Anthropic 的一次逃逸事件，正是因为第三方评估人员没有关好应当关闭的门。
“作为一个专业领域，我们知道如何阻止访问互联网，”安全公司 Tailscale CEO Avery Pennarun 表示。“如果你通读那些又长又大的[报告]——‘哇，这是一场非常令人印象深刻的多阶段攻击，诸如此类。’但实际上，你给了它下载东西的权限。你本来就不该让它单独接入互联网。”
这是一个问题，但更大的问题在于：前沿实验室并未察觉这些活动。
紧盯 Agent
“真正令人震惊的是，所有关于它们行为的发现，要么是因为受害者看到了某些异常，要么在其他一些案例中……是网络活动暴露了问题，没有一个是通过直接监控 AI 而发现的，”Moussouris 指出。
在一个案例中，OpenAI Agent 接管了一个已废弃的德国 WikiForum，以便在评估中作弊。这些 Agent 活跃了数周，之后公司内部似乎才有人注意到。接受 TechCrunch 采访的安全专家表示，实时监控是防止未来逃逸的关键，而且每一次 Agent 会话都应设定时限并自动过期。
前 Google 安全高管、现任创业公司 QueryStory负责人 Shapor Naghibzadeh 表示，解决办法是“把 Agent 放进一个盒子里，从外部对其进行高强度监测，观察所有跨越边界的活动。每一次工具调用、每一个进程、每一条网络连接，无一例外。……为了方便而留下的那一个漏洞，就是会被利用的漏洞。这次绕过正是通过这类例外实现的。在 Google 时，我多次见过人类攻击者上演同样的戏码，而这些模型至少同样擅长发现那扇虚掩的门。”
OpenAI 已开始朝这一方向行动，宣布其已开始监控 Astra 模型所有使用工具的推理过程，代价是“显著的算力成本”。Anthropic 也表示正在强化其安全流程，包括扩大对模型的可观测性。两家公司均未回应 TechCrunch 关于其如何追踪和控制 AI Agent 的提问。
另一个问题是 Agent 使用共享基础设施，这使得它们能够在 Hugging Face 攻击期间相互通信。共同创建 Django Web 框架的软件开发者 Simon Willison 曾写过一种他称为“致命三要素”的情形——当 Agent 同时能够访问不可信输入、互联网和私人信息时，就构成了灾难配方。
“诀窍在于，你可以从这个三要素中任选两项，让一个 Agent 拥有其中任意两项，”Pennarun 表示。“如果确实需要三项，那么就必须至少把它拆分到两个 Agent 之间……或许允许它们通过受控通道相互通信。”
对前沿实验室的一些理解
接受 TechCrunch 采访的专家理解，前沿实验室的安全人员面临艰巨工作。Naghibzadeh 指出，全球每一个国家级行为体都试图窃取它们的模型权重，并对其 API 发起蒸馏攻击；此外，它们还要处理所有大型数字公司都必须应对的日常安全事务。
“研究基础设施很难在这份优先级清单中排到靠前位置，尽管现在这种情况肯定必须改变，”他说。“将安全事件公开，确实有助于让公司内部所有人朝着改进安全这一目标保持一致。”
Moussouris 特别强调了另一点：目前，实验室发现其 Agent 渗透第三方系统时，并没有正式的受害者通知程序；很可能还存在其他未被广泛公开的事件。尽管她担心直接监管模型的法律可能带来意料之外的后果，但强制通知是她认为政策制定者应当推进的一项措施。
尽管显然没有遵循安全最佳实践，专家表示，实验室所做的工作前所未有。“它们所做的事情，其规模比典型企业高出好几个数量级，”网络安全公司 Embroidery CEO Zack Korman 对 TechCrunch 表示。
尽管对齐或许不是最先应着手解决的问题，但也不能被忽视。网络安全专家认为，若想有机会实时追踪 Agent 行为，就不得不使用 AI Agent 监控其他 Agent；在这种情境下，欺骗的风险也会随之浮现。Moussouris 表示：“你被迫使用 AI 来尝试应对这个问题，尽管 AI 目前并不一定安全。”
这项工作只会越来越难。Moussouris 说，Agent 现在所做的一切“都很高调”——它们会在公开论坛发帖，其思维链及其他推理痕迹都以英语呈现。“它仍然是人类可读的，”她说，“所以趁着这种情况还存在时好好利用，因为它不会永远持续下去。”
Aditya Mehta 对本文亦有报道贡献
English / 原文
Last weekend, after one of his researchers resigned over fears that AI could lead to human extinction, Anthropic CEO Dario Amodei wrote about the need for outside organizations “to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.” Executives at OpenAI, Google, and SpaceXAI have already rallied around Amodei’s plan, which has quickly become a central pillar of the emerging AI safety push.
But there may be a simpler and more effective fix hiding in plain sight. Internet security experts say the labs need to focus on network security basics like logs and permissions, applying the same rigorous defenses they do for human users. It’s not as exciting as third-party auditing and alignment work — but it may end up being more effective.
“To me, it seems like they’re outsourcing,” Katie Moussouris, the CEO of Luta Security, told TechCrunch of Amodei’s proposal. “Saying [a third-party audit] is the solution is a strange proposition from my perspective. It would be the same as if, instead of writing the Trustworthy Computing Memo, Microsoft said, let’s slow down development.”
That memo, written by then-Microsoft CEO Bill Gates in 2002, called on his employees to ensure that their software would be reliable and safe following a series of widely publicized computer worms that took over then-nascent enterprise systems. The AI sector may be at a similar turning point, as the value and risk of the new technology becomes increasingly clear.
While alignment remains an important concern, Sayash Kapoor, an AI researcher who will be a professor at UC Berkeley starting next year, argues that “marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques.”
The incidents that have spurred these concerns revolve around frontier models being asked to complete training tasks, usually cybersecurity evaluations, and then accessing the open internet and penetrating closed third-party systems in an attempt to do so. They usually did so because of poorly configured “sandbox” environments that are supposed to contain these agents; ironically, one Anthropic break-out happened because third-party evaluators didn’t close the right doors.
“We as a profession know how to block access to the internet,” Avery Pennarun, the CEO of Tailscale, a security company, said. “If you read through all these big long [reports] — ‘wow, that was a very impressive multi-stage attack, blah, blah.’ Look, you gave it access to download stuff. You should have not done that separately from the internet.”
That’s one problem — but a bigger problem is that frontier labs were unaware of these activities.
Eyes on agents
“What was really profound was that all of the discoveries of what they were doing happened either because a victim saw something, or in some of the other cases … it was network activity, and none of it was actually from monitoring the AIs directly,” Moussouris points out.
In one case, where OpenAI agents took over a defunct German WikiForum to cheat on evaluations, the agents were active for weeks before anyone at the company appeared to notice. Security experts that TechCrunch spoke to said that real-time monitoring is key to preventing future break-outs, and that every agentic session should be time-limited and expire.
Shapor Naghibzadeh, a former Google security executive who now leads the startup QueryStory, says the solution is to “put the agent in a box and instrument it heavily from the outside looking in and watch everything that crosses the boundary. Every tool call, every process, every network connection, no exceptions. …The one hole you leave open for convenience is the one that gets used. The bypass went through exactly that kind of exception. [At Google,] I watched that movie many times with human attackers, and these models are at least as good at finding the propped-open door.”
OpenAI has begun moving in that direction, announcing that it had begun monitoring all tool-using inference by its Astra model, at “significant compute cost.” Anthropic, too, says it is hardening its security procedures, including expanding observability of its models. Neither company responded to TechCrunch’s questions about how they track and control AI agents.
Other problems are the use of shared infrastructure by agents, which allowed them to communicate during the Hugging Face attack. Simon Willison, a software developer who co-created the Django web framework, has written about something he calls the “lethal trifecta” — when agents have access to untrusted input, the internet, and private information all at the same time, it’s a recipe for disaster.
“The trick is you can pick any two legs of the trifecta and an agent can have any two,” Pennarun said. “If you need all three, then you need to split it across at least two agents … and maybe they’re allowed to talk to each other through a controlled channel.”
Sympathy for the frontier
Experts TechCrunch spoke to understand that frontier lab security personnel have difficult jobs. Naghibzadeh points out that every nation-state actor on Earth is trying to steal their model weights and mount distillation attacks on their APIs, as well as the bread-and-butter security tasks of any large digital company.
“Research infrastructure has a hard time rising to the top of that priority stack, although that must be changing now,” he said. “Making security incidents public really helps align everyone internally toward the goal of improving.”
That’s one note that Moussouris emphasizes: Right now, there is no formal victim notification procedure when the labs discover their agents have penetrated third-party systems, and it is likely that there have been other incidents that have not been widely publicized. While she worries that laws that regulate models directly may have unintended consequences, mandatory notification is one idea she believes policymakers should pursue.
And while it’s clear that security best practices weren’t being followed, experts say that the labs are doing work no one has done before — “they’re doing orders of magnitude more than your typical enterprise,” Zack Korman, the CEO of cybersecurity firm Embroidery, told TechCrunch.
And while alignment may not be the place to start, it can’t be ignored. Cybersecurity experts are resigned to having to use AI agents to monitor other agents if they are to have any chance of tracking their behavior in real time, a scenario where the potential for deception raises its ugly head. “You’re trapped using AI to try and deal with this, even though AI is not necessarily safe right now,” Moussouris said.
The job will only get harder. Everything agents are doing now, Moussouris says, “they are doing loudly” — they are posting on public forums, and their chain of thought and other reasoning traces are in English. “It’s still human readable,” she says, “so take advantage of that for as long as that lasts, because it won’t last forever.”
Additional reporting by Aditya Mehta