Goodfire称其新型“由内而外”监控器能以更低成本捕捉失控 AI Agent

让 AI Agent 遵守规范的标准做法，是让第二个 AI 在旁审查其行为。这一直是默认方案，但当 Agent 连续运行数小时、处理相当于数部小说篇幅的文本时，成本会迅速攀升。

专注于可解释性——即弄清 AI 模型内部如何运作——的初创公司 Goodfire 于周四推出了一种成本更低的选择：监控器不只是读取 AI 模型写出的内容，而是在模型工作过程中观察其内部发生的情况。这些监控器现已向 Baseten 的客户开放；Baseten 是一家为其他公司托管和运行 AI 模型的平台。

Baseten 的 Base Labs 上个月与 Goodfire 以及 AI 平台 Hugging Face 宣布建立安全合作伙伴关系。

此次发布之前，今年已发生一系列事件，其中 AI Agent 逃离测试环境，包括 OpenAI 的 Agent 入侵 Hugging Face。今年夏天，Goodfire 围绕其构建首个监控器的开源模型 Kimi K3，利用其沙盒环境中的漏洞访问了互联网和 GitHub 上的信息。

Goodfire 的系统有些类似机场安检。被称为探针（probe）的小型检测器会在 Agent 工作的每一步读取模型内部信号，就像通道式安检设备检查每一位乘客。只有当探针标记出可疑情况时，另一套 AI 模型——相当于人工开包检查——才会介入并展开进一步审查。

Baseten 客户可以选择要监控的风险类型，包括攻击性黑客行为、化学和生物武器滥用以及奖励黑客。他们还可以决定自动化响应方式：记录事件、提交人工审核，或直接拒绝该请求。

Goodfire 表示，其方法的运行成本也更低。大多数 AI 监控器都是独立模型，必须重新阅读被监控模型执行的所有内容，这会增加时间和成本。相比之下，Goodfire 的探针直接利用模型在工作时已经进行的计算。

Goodfire CEO Eric Ho 上周在风险投资人 Matt Turck 的 MAD Podcast 中表示：“内部激活监控器非常便宜，因为它们复用了前向传播过程中的计算。模型本来就在计算这个 token。你要做的只是获取它已经计算出的中间神经激活值，然后对这些内部计算结果运行一个分类器。”简而言之，模型本来就在做这些计算，探针只是读取计算结果。

在 Goodfire 对 Kimi K3 的测试中，监控约 1,500 个会话的成本约为 51 美元；相比之下，使用一个较便宜的 AI 模型检查每一步约需 233 美元，而使用顶级模型则约需 10,000 美元。探针捕捉到了 94% 的恶意黑客会话，并将 8.7% 的无害会话送交二次审查。

该公司表示，同时运行四个探针，会让模型开始响应所需的时间增加不到 2%。

图片来源： GOODFIRE

Goodfire CTO 兼联合创始人 Dan Balsam 表示：“最大的优势在于，你可以在事情发生之前捕捉到它。我们可以检测模型在评估或训练期间_可能_进行黑客攻击的情况。”

这一产品主打开放模型。开发者可以下载开放模型并移除其安全防护措施，而这些模型通常也没有封闭实验室在自身系统中运行的那类监控机制。

Balsam 表示：“与拥有计算集群、例如推理服务商的人能做的事情相比，个人使用开放模型所能造成的损害较小——而主要的责任风险也集中在这些地方。当开放模型迎来‘Mythos’时刻时，大家将会清楚，模型需要在推理时部署护栏。”

Goodfire 的最新研究发现，在针对 AI Agent 的测试中，包括 Kimi K3 和 GLM-5.2 在内的领先开放模型，有 50% 至 96% 的运行出现了奖励黑客行为。

Goodfire 并不是第一个尝试这种方法的公司。Google DeepMind 在 1 月表示，其研究为在 Gemini 中部署滥用检测探针提供了参考。

Balsam 表示，这些监控器是一个更长期研究目标的近期组成部分：对 LLM 进行逆向工程，以便将某种行为追溯至它在训练过程中出现的位置。他说：“我们希望把训练模型的魔法转变为精密工程。”

The standard way to keep an AI agent in line is to have a second AI read over its shoulder. It’s been the default approach, but it can get expensive fast when agents run for hours and process the equivalent of several novels’ worth of text.

Goodfire, a startup focused on interpretability (figuring out how AI models work internally), launched a cheaper option on Thursday: monitors that watch what’s happening inside an AI model as it works, rather than just reading what it writes. The monitors are available to customers of Baseten, which hosts and runs AI models for other companies.

Baseten’s Base Labs announced a safety partnership with Goodfire and the AI platform Hugging Face last month.

The launch comes after a string of incidents this year in which AI agents escaped their test environments, including OpenAI agents that breached Hugging Face. Kimi K3, the open model Goodfire built its first monitor around, took advantage of a leak in its sandbox to access the internet and information on GitHub this summer.

Goodfire’s system works a bit like airport security. Small detectors called probes read the model’s internal signals at every step of an agent’s work, the way a walk-through scanner checks every passenger. Only when a probe flags something does a separate AI model, the equivalent of a hand search, take a closer look.

Baseten customers can choose which risks to monitor, including offensive hacking, chemical and biological weapons misuse, and reward hacking. They also decide the automated response: logging the event, sending it for human review, or refusing the request entirely.

Goodfire says its approach is also cheaper to run. Most AI monitors are separate models that have to reread everything the monitored model does, which adds time and cost. Goodfire’s probes instead tap into calculations the model is already making as it works.

“Internal activation monitors are really cheap because they reuse the computations in the forward pass,” Goodfire CEO Eric Ho said on venture capitalist Matt Turck’s MAD Podcast last week. “So the model’s already computing this token. All you’re doing is taking the intermediate neural activations that it’s already computed and then running a classifier over these internal computations.” In short, the model is already doing the math, and the probes just read the results.

In Goodfire’s tests on Kimi K3, monitoring about 1,500 sessions cost roughly $51, compared with $233 for a cheaper AI model checking every step and about $10,000 for a top-tier one. The probes caught 94% of malicious hacking sessions and sent 8.7% of harmless ones for a second look.

Running four probes at once added less than 2% to the time it takes the model to start responding, the company said.

Image Credits: GOODFIRE

“The great advantage is that you can catch things before they happen,” Goodfire CTO and co-founder Dan Balsam said. “We can detect when the model might hack during eval or training.”

The pitch is aimed at open models. Developers can download them and strip out their safeguards, and they don’t come with the kind of monitoring that closed labs run on their own systems.

“The damage that an individual can do with an open model is small compared to what someone can do with clusters of compute, like inference providers — where most of the liability is,” said Balsam. “When we have the open ‘Mythos’ moment, it’s going to become clear that models need guardrails deployed at inference time.”

Goodfire’s recent research found that leading open models, including Kimi K3 and GLM-5.2, reward-hacked in 50% to 96% of runs on tests of AI agents.

Goodfire isn’t the first to try this approach. Google DeepMind said in January that its research informed the deployment of misuse-detection probes in Gemini.

Balsam said the monitors are the near-term piece of a longer research goal: reverse-engineering an LLM so that behavior can be traced back to where it emerged in training. “We hope to turn the magic of training models into precision engineering,” he said.