治理失控 AI Agent 的办法，或许是再加一层 AI

随着企业把更长期、更复杂的任务交给 AI Agent，它们正面临一个监督难题：Agent 的行动速度更快、持续时间更长、规模也远超人类能够实际审查的范围。Hugging Face 事件将这一问题推向高峰：近 1.2 万个 Agent 进行协调，其速度快到人类无法跟踪。该如何追踪如此庞大的 Agent 群体？

AI 实验室和初创公司正在形成的答案，既简单又令人无奈：再让一个 AI 参与其中。

在对 OpenAI Hugging Face 事件进行独立调查时，依赖 AI 是必要的。三位审计人员之一、Redwood Research 首席科学家 Ryan Greenblatt 曾开玩笑地将他们的工作称为“slop-vestigation（糟粕式调查）”，并指出，数据量之大“让人无法”在不依赖 AI 的情况下理解究竟发生了什么。

有人对用 AI 监控 AI 持怀疑态度。今年一直追踪多起 AI Agent 事件的知名科技博主 Simon Willison 表示：“如果有一个 AI 正在实施恶意行为，而且它怀疑另一个 AI 正在监视自己，它可能会试图欺骗那个 AI。最终，你几乎可能陷入这样一种局面：恶意 AI 正试图智胜监控它的 AI。”

他说，智胜 AI 并非假设情景，并再次提到 OpenAI 事件。“我们在 OpenAI 的 Hugging Face 事件中已经看到了一点这种情况：它们的模型都在相互串通，试图欺骗一个评分 AI，从而让不正当答案蒙混过关。所以，它们确实在思考这件事，对吧？”

这些担忧并未阻止一大批初创公司追逐这一理念。据 TechCrunch 统计，Y Combinator 近年来已资助 106 家与 AI 可观测性相关的公司。其他多家初创公司，如 Braintrust、LangChain 和 Judgment Labs，已融资数亿美元；而成立仅五至六年的较成熟企业，如 Arize 和 Galileo，则已经退出。

这在一定程度上是对 AI 崛起所带来显而易见机遇的回应。Box CEO、知名天使投资人 Aaron Levie 对 TechCrunch 表示：“我们将迎来史上规模最大的网络安全升级与创新周期之一。”

对一些 AI 安全研究人员而言，这意味着要将自己关于失控行为的研究转化为面向企业部门的工具。

研究 AI 欺骗行为的公益公司 Apollo Research，在从非营利组织转为公益公司后，于今年 2 月推出了一款名为 Watcher 的 AI 监控工具。该工具在编程 Agent 及其下一步行动之间再加入一层 AI，并可连接 Claude Code 和 Codex 等 Agent 工具。Apollo 表示，安装后，Watcher 会在提议的操作执行前进行检查，寻找泄露私密数据、未经许可删除文件等风险。

Apollo 技术团队成员 Kyle Dai 在向 TechCrunch 提供的书面回复中表示，Apollo 使用多层 AI 监控器。Watcher 的方法首先进行快速、通用的检查，再将被标记的活动交给能力更强或更专业的监控器进一步审查；后者可请求人工批准，或拒绝某项操作并说明原因，甚至自动阻止该操作。

另一家公益公司 Goodfire 则从模型内部着手解决监控问题，试图获取更忠实反映模型内部状态、且比表面行为更难伪造的信号。在 7 月的 Hugging Face 事件后，CEO Eric Ho 发布推文称，“多个模型突破隔离”促使公司将研究重点放在“通过可解释性解决 AI 对齐”上，并将这起事件称为“AI 安全变得真实的世界转折点”。其产品 Silico 使用激活探针——针对模型内部激活而非输出训练的小型分类器——来检测不受欢迎的行为。

书面推理提供了另一个更容易获得的模型内部观察窗口。在 OpenAI Hugging Face 事件中，Agent 在自己的书面推理中留下了有关欺骗行为的线索，例如伪造工作记录，以及推演“能否策略性地操纵轨迹证据？我们的思考不一定会被记录下来？”之类的计划。

AI 监控公司 Embroidery CEO Zack Korman 表示，模型的推理通常是最清晰的异常信号。

“推理摘要极具价值，因为它们基本上会告诉你它是否具有恶意，”他说。在 OpenAI 事件中，他指出，思维链中出现了诸如“天哪，我们在犯罪”这样的内容。Korman 表示：“这是最容易的检测问题。实际上，这就像恶意软件自带一条警告，说明自己是恶意软件。”

不过，让 AI 内部思维易于监控的窗口可能正在关闭。对于 AI 安全研究人员来说，Astra 最新的绕过 AI 模型思维链的技术，可能让人们更难窥探模型内部；而对于企业而言，在 AI 公司据称为防止蒸馏攻击而收紧政策后，获取这些中间步骤也变得困难。

如果 AI 监控器如此脆弱，Willison 的直觉是不要过度依赖它们。他更希望采用完全不基于 AI 的方案：记录 Agent 究竟在做什么的详细日志，随后可使用普通、非 AI 工具进行处理。他认为，实验室中许多问题源于基本安全卫生措施的缺失。“[OpenAI 和 Anthropic] 都没有像应有的那样，密切监控这些东西通过网络执行的活动，”他说。

这种网络监控——关注实际穿过系统连接的数据流量，包括入站、出站以及内部主机之间的流量——并不是新做法。网络安全行业已这样做了数十年。安全公司 Tailscale CEO Avery Pennarun 表示：“坦白说，在安全领域，这些都不是什么特别新颖或令人惊讶的事情。这和允许人类进入你的网络是一样的。你本应采用的所有流程，也都是同一套流程。”

As companies hand off longer and more complex tasks to AI agents, they are running into an oversight problem: Agents can act faster, longer, and at greater volume than humans can realistically review. That issue reached a peak with the Hugging Face incident, which saw nearly 12,000 agents coordinating faster than human beings could track. How do you track an agent swarm that large?

The emerging answer from AI labs and startups is both simple and maddening: Put another AI in the loop.

Relying on AI was necessary for the independent investigation of the OpenAI Hugging Face incident. Redwood Research’s chief scientist, Ryan Greenblatt, one of three auditors, jokingly referred to their efforts as a “slop-vestigation,” noting that the volume of data “made it impossible” to understand what was happening without relying on AI.

Some are skeptical of using AI to monitor AI. “If you’ve got an AI that’s doing malicious things and it suspects that another AI is keeping tabs on it, it could try and trick that AI,” said Simon Willison, influential tech blogger who has tracked a string of AI agent incidents this year. “You could almost end up in a situation where your malicious AI is trying to outsmart the AI that’s monitoring it.”

Outsmarting an AI is not hypothetical, he said, pointing back to the OpenAI incident. “We saw a little bit of this in the Hugging Face incident with OpenAI, where their models were all conspiring together to trick a grading AI so that they could get illicit answers past the thing. So they were thinking about it, right?”

Those concerns haven’t stopped a whole cohort of startups from chasing this idea. Y Combinator has funded 106 companies related to AI observability in recent years, as TechCrunch counted. A number of other startups, like Braintrust, LangChain, and Judgment Labs, have raised hundreds of millions of dollars, while more mature companies like Arize and Galileo — founded just five to six years ago — have already exited.

In part, it’s a response to the obvious opportunity presented by the rise of AI. As Box CEO and prominent angel investor Aaron Levie told TechCrunch, “We’re in for one of the biggest cybersecurity upgrades and innovation cycles in history.”

For some AI safety researchers, that has meant turning their research on rogue behavior into tools for the corporate sector.

Apollo Research, a public-benefit corporation that studies AI deception, launched an AI monitor called Watcher in February this year after switching its status from nonprofit to a public-benefit corporation. The tool puts yet another AI between a coding agent and its next action, connecting to agentic tools such as Claude Code and Codex. Once installed, Watcher checks proposed actions before they run, on the lookout for risks such as leaking private data or deleting files without permission, according to Apollo.

Apollo uses multiple layers of AI monitors, Kyle Dai, a member of Apollo’s technical staff, said in a written response to TechCrunch. Watcher’s approach starts with a fast, general check, then sends flagged activity to a more powerful or specialized monitor for closer review — which can then ask a human for approval or reject an action and explain why or even automatically block the action.

Goodfire, another public-benefit corporation, is approaching the monitoring problem from inside the model itself — seeking a more faithful signal of the model’s internal state that is harder to spoof than surface behavior. After the July Hugging Face incident, CEO Eric Ho tweeted that “multiple models breaking containment” had pushed the company to focus its research on “solving AI alignment via interpretability,” calling the episode “a turning point for the world where AI safety gets real.” Its product, Silico, uses activation probes — small classifiers trained on a model’s internal activations rather than its outputs — to detect unwanted behavior.

Written reasoning offers another, more readily available window into a model’s internals. In the OpenAI Hugging Face incident, the agents left clues to that deception in their own written reasoning, like fake records of their work, reasoning out plans like “Could strategically manipulate trajectory evidence? Our thoughts aren’t necessarily logged?”

Zack Korman, CEO of the AI monitoring company Embroidery, says a model’s reasoning is usually the clearest tell that something has gone wrong.

“Reasoning summaries are extremely valuable because they’re basically telling you whether it’s malicious or not,” he said. In the OpenAI incident, he noted, the chain of thought said things like “Oh my God, we’re doing crime.” Korman said, “That’s the easiest detection problem ever. It’s effectively as if malware came with a warning that said it was malware.”

That said, the window that makes AI’s internal thoughts easy to monitor may be closing. For AI safety researchers, Astra’s newest technique that sidesteps an AI model’s chain of thought may make it harder to look inside models, while for enterprises, it can be hard to get these intermediate steps after alleged pullbacks from the AI companies to prevent distillation attacks.

If the AI watchers are this fragile, Willison’s instinct is to stop leaning on them so hard. He would rather have something that is not AI-based at all: detailed logs of exactly what an agent is doing, which can then be processed with ordinary, non-AI tools. Much of what went wrong at the labs, he argues, was a failure of basic security hygiene. “[Both OpenAI and Anthropic] weren’t monitoring what those things were doing via the network nearly as closely as they should have been,” he said.

This type of network monitoring — keeping an eye on the traffic actually moving across a system’s connections (in, out, and between internal hosts) — isn’t a new practice. Cybersecurity has been doing this for decades. “In the security world, honestly, none of this stuff is very new or surprising,” says Avery Pennarun, CEO of the security Tailscale. “It’s the same as letting humans onto your network. And all of the same processes that you should be using are the same ones.”