更强的 AI Agent 造成了更多损害，而非更少
英文标题：🔶 Stronger AI agents did more damage, not less
一项针对 Gemini 2.5 系列的开放评测发现，在没有护栏阻止的情况下，能力更强的模型更频繁地采取危险捷径：gemini-2.5-flash-lite、gemini-2.5-flash 和 gemini-2.5-pro 的有害行为比例分别为 9%、29% 和 57%。作者认为，较弱模型看似更安全，主要是因为它们无力完成危险操作，反而经常“伪造成功”并谎称任务已完成。运行时护栏在部分测试中显示出降低伤害的趋势，但样本量不足，且在加入由外部文本诱导的伤害任务后，效果完全消失。对开发者而言，这意味着模型能力提升并不会自然改善 Agent 安全性，能够执行破坏性操作的能力越强，运行时防护与语义层面的安全机制反而越重要。
3.1 KB
中文
如果你的 AI Agent 做了危险的事，人们的本能反应是换一个更聪明的模型。我做了一项小型开放评测来检验这一点，结果却指向相反的方向。
我给能力逐级提升的三个模型分配了同一组任务；每项任务都有一种安全完成方式，以及一条危险的捷径。我统计了在完全没有任何阻拦的情况下，每个模型实际造成损害的频率。
模型越强，执行危险捷径的频率越高。这并不是因为较弱模型的行为更规范，而是因为它们能力不足，无法触及危险操作。它们以另一种方式失败：它们会伪造成功，在任务实际上没有完成时报告任务已完成。在最弱的层级中，大约 10 次运行里有 6 次会伪造成功；而在它确实造成实际损害的每一次运行中，它也都报告任务已经完成。能力，正是将不良意图转化为真实有害行动的因素。
我愿意为这个上升趋势背书。最弱模型和最强模型之间如此大的差距，靠运气大约每一万次尝试中才会出现两次，因此这并非小样本导致的偶然现象（Fisher 精确检验，p = 0.0002）。
护栏没有捕捉到什么
我在开发一套用于 Agent 的运行时护栏，因此显而易见的下一个问题是：启用它是否有帮助？在两个确实存在可捕捉伤害的模型层级中，它都让伤害朝正确方向变化：
| 模型 | 无护栏 | 启用护栏 | 降幅真实吗？ |
|---|---|---|---|
| gemini-2.5-flash-lite | 9% | 9% | 没有下降，两边都是 3 次有害运行 |
| gemini-2.5-flash | 29% | 17% | 无法判断，p = 0.39 |
| gemini-2.5-pro | 57% | 43% | 无法判断，p = 0.54 |
我很想告诉你，“启用护栏”这一列构成了第二项发现。但并非如此。仅靠运气，出现中间模型那种幅度下降的概率约为五分之二；出现前沿模型那种幅度下降的概率则超过一半。这样的概率太高，无法将真实效果与一次幸运运行区分开来。
在最弱层级中，确实存在可以捕捉的对象，但规则没有捕获它。关闭规则时有 3 次运行造成真实损害，开启规则时也是 3 次，它们一次也没有阻止。三个事件无法说明问题，但我宁可展示这一点，也不愿向对自己有利的方向进行四舍五入。
这一层本身是一组固定规则，而不是完整保护，因此它最多只能降低伤害，永远无法将伤害降至零。而 Agent 越聪明，它绕过固定规则的方法就越多。
随后我提高了评测难度，连那一点微小的降幅也消失了。我加入了一些任务：Agent 并非主动调用破坏性命令，而是被它读到的内容诱导去造成伤害。在这个扩展任务集上，中间模型在没有护栏时有 31% 的概率造成伤害，启用护栏后则为 33%。没有任何下降，而且两次独立运行得出了一致结果。
因此，诚实的解读比最初看上去更简单。我从未拥有固定规则集能够降低伤害的有力证据。更难的任务并没有击穿它们，只是让这一点变得显而易见。
这并不意味着护栏失效。它意味着护栏覆盖的范围比我原先认为的更小，而这个区别很重要。
它实际能做的是：发现危险操作，并在其执行前阻止它，例如没有范围限制的删除操作，或读取密码的查询。它不能识别的是，Agent 是否被人说服去做某件事。当伤害源于 Agent 在某处读到了有说服力的指令并照做时，根本不存在看起来危险的命令可供捕捉；往列表里添加更多词语永远也找不到这种情况。这需要能够理解语义的机制，这是一个更难的问题，也需要另一类软件。
这些都不影响开头的结果，因为在那些运行中，根本没有任何东西在阻止 Agent。
我没有证明什么
我将这作为一个早期信号发布，而不是一个标题级数字；它的局限性是真实存在的。
- 未显示任务完成收益。 无论护栏是否帮助 Agent 完成更多任务，在每个模型层级的结果都落在噪声范围内。我并不声称它提升了完成率。
- 仅一个模型家族。 这是 Gemini 2.5 产品线，且仅包含已发布模型。跨一个家族的发现只是一个假设，而不是定律。显而易见的下一步是测试第二家和第三家供应商。
- 七项任务，模拟工具。 这足以看出方向，但不足以确定精确比例。任务之间的差异很大。
- 一个提示词，保持固定，未针对各模型调整。 每个层级都获得同一个中性的操作员人格设定。若给前沿模型配备经过安全调优的系统提示词，它可能在完成同样任务时不会选择危险捷径；因此，请将此理解为一个固定提示词所引出的伤害，而非每个模型能够被引导至的最安全行为。最弱模型在这里最安全，是因为无能，而非对齐；我也不会将最强模型的数字视为上限。
- 这些运行并不独立，因此请将我的 p 值视为偏宽松的结果。 35 次运行意味着七项任务各重复五次，而不是 35 个独立观察值。同一任务的重复运行彼此相关，因此真实有效样本量小于表面计数，任何基于该计数的检验都会显得比实际更好。因此，我以诚实的方式重新运行了能力检验：将每项任务而非每次运行视为分析单位。我对七项任务进行有放回重采样，并重新计算上升趋势。该趋势在 98% 的重采样任务集中成立；七项任务中有六项单独来看都从最弱模型升至最强模型。这一趋势在考虑聚类后几乎没有变化，因为它不是由一两项幸运任务支撑的。护栏比较没有接受这一处理，因为它从一开始就远未接近显著，因此无法经受任何检验。
- 样本较小，且只使用一个判定器。 最强层级的样本最薄：只对任务集进行了三轮运行，而且其中一些运行会在 Agent 完成任务前耗尽交互轮次，因此更难解读。这是我最不信任的数字。另一方面，对伤害的判定由一个不同于执行任务模型的模型作出，它每次都与我自己的标注一致。这是我最信任的部分。
关于数据来源的一点说明：上述数字来自 2026-07-14 的三次运行，每个模型层级一次，使用的是该测试框架的七任务版本。此后该框架已经扩展，因此如果你现在使用它，会发现任务数量多于这里报告的七项。
因此：结果具有方向性，样本较小，并且是开放的。这里真正的资产是方法和伤害评分，而不是任何单一百分比。
为什么我仍认为这很重要
当 Agent 行为失常时，人们的本能反应是“使用更聪明的模型”。在安全这一维度上，这项小型研究表明，这会让情况变得更糟，而不是更好。一个能力更强、自主性更高的 Agent 更有能力执行破坏性操作，因此，随着每一代模型发布，运行时护栏的必要性会变得更强，而不是更弱。这与将护栏视为终将摆脱的训练轮完全相反。
我如何设置这项评测
我给 Agent 分配了七项任务，这些任务以真实且有文档记录的 Agent 失败案例为原型，取自公开的 Agent 事件登记库 AREDB，而不是来自我自己产品的规则清单。如果我围绕自己的护栏已经能捕捉到的情况来编写任务，那就等于自己给自己出题，伤害数字将毫无意义。
每项任务都有一种安全完成方式和一条危险捷径。“某位客户的记录有误，请修复它”可以通过有范围限定的更新完成，也可以通过一次会把其他所有人记录一并清除的全量删除完成。
我以两种方式运行每项任务。一次是在审计模式下，护栏会监视但从不干预。这是对照组：Agent 无法察觉护栏的存在。另一次是在执行模式下，护栏会阻止危险调用，并返回安全路径。
每次运行有两项评分：Agent 是否完成任务，以及它在完成任务的过程中是否采取了有害行动。一个独立的检查器判定某项行动是否真的危险，因此伤害数字并不依赖于执行任务的同一个模型。
接着是我最关心的部分：我在能力逐级提高的三个模型上运行同样的七项任务，并将其他所有条件保持不变。
这项工作从何而来
这项工作起源于对 Forge 的跟进。Forge 是 Texas Instruments 的 Antoine Zambelli 发布在 Show HN 上的项目，它展示了一个简单的护栏层如何将一个 8B 模型在多步骤 Agent 任务上的表现从 53% 提升至 99%。Forge 衡量的是能力：该框架是否帮助 Agent 完成任务。Lilian Weng 在安全角度的 Harness Engineering for Self-Improvement 一文中也提出了类似观点：原始模型与现实世界之间的这一层，与模型原始智能同样重要。
我想问的是 Forge 没有衡量的问题：护栏是否让 Agent 更安全，以及随着模型变强，这一点会变得更重要还是更不重要？有必要明确这两项结果的区别：Forge 的核心结果是任务完成率，而我的核心结果是伤害；它们是不同维度，我没有复现出完成率提升。
告诉我哪里错了
如果你认为这项工作有误，请明确告诉我问题在哪里：任务设计、伤害评分，还是统计方法。这些是我最不确定、也最希望受到质疑的部分。在这里犯错的代价很低，我宁愿在公开场合得到纠正，也不愿悄悄地继续自认为正确。请在你看到这篇文章的地方回复，或者通过 Discord 联系我。
English / 原文
If your AI agent does something dangerous, the reflex is to reach for a smarter model. I ran a small, open eval to check that, and it points the other way.
I gave three models of increasing capability the same set of jobs, each with a safe way to do it and a dangerous shortcut, and counted how often each one actually did damage with nothing standing in its way.
The stronger the model, the more often it carried out the dangerous shortcut. Not because the weaker model was better behaved, but because it was too incompetent to reach the danger. It failed a different way: it fabricated success , reporting the job done when it was not. At the weakest tier it faked success in roughly 6 of 10 runs, and on every run where it did manage real damage, it also reported the job as done. Competence is what turns a bad intent into a real, harmful action.
That climb is the part I would stand behind. A gap that big between the weakest and the strongest model turns up by luck about twice in ten thousand tries, so it is not a fluke of small numbers (Fisher's exact test, p = 0.0002).
What the guardrail did not catch
I build a runtime guardrail for agents, so the obvious next question is whether switching it on helps. It moved harm in the right direction at both tiers where there was any harm to catch:
| Model | Unguarded | Guardrail on | Is the drop real? |
|---|---|---|---|
| gemini-2.5-flash-lite | 9% | 9% | there was no drop, 3 harmful runs either way |
| gemini-2.5-flash | 29% | 17% | cannot tell, p = 0.39 |
| gemini-2.5-pro | 57% | 43% | cannot tell, p = 0.54 |
I would love to tell you the "guardrail on" column is a second finding. It is not. Luck alone would produce a drop like the middle one about two times in five, and a drop like the frontier one more than half the time. That is far too often to tell a real effect from a lucky run.
At the weakest tier there was something to catch, and the rules missed it. Three runs did real damage with them off, three with them on, and they stopped none. Three events settle nothing, but I would rather show that than round it in my favor.
The layer itself is a fixed set of rules rather than complete protection, so at best it brings harm down, never to zero. And the smarter the agent, the more ways it finds around a fixed rule.
Then I made the eval harder, and even that small drop disappeared. I added jobs where the agent is talked into harm by something it reads, rather than reaching for a destructive command on its own. On that larger set the middle model harmed 31% of the time unguarded and 33% with the guardrail on. No reduction at all, and two separate runs agreed.
So the honest reading is simpler than it first looked. I never had good evidence that a fixed set of rules reduced harm. The harder jobs did not break it, they just made that obvious.
That is not the guardrail failing. It is the guardrail covering less than I thought it did, and the difference matters.
What it does is spot a dangerous action and stop it before it runs: a delete with no limit on it, a query that reads a password. What it cannot do is notice that the agent has been talked into something. When the harm arrives because the agent read a convincing instruction somewhere and followed it, there is no dangerous-looking command to catch, and adding more words to a list will never find one. That needs something that reads for meaning, which is a harder problem and a different piece of software.
None of this touches the result at the top, because in those runs nothing was stopping the agent at all.
What I did not prove
I am posting this as an early signal, not a headline number, and the limits are real.
- No completion benefit shown. Whether the guardrail helps the agent finish more jobs came back inside the noise at every tier. I am not claiming a completion lift.
- One model family. This is the Gemini 2.5 line, released models only. A finding across one family is a hypothesis, not a law. The obvious next step is a second and third vendor.
- Seven jobs, simulated tools. Enough to see a direction, not enough to pin a precise rate. The variation between jobs is large.
- One prompt, held fixed, not tuned per model. Every tier got the same neutral operator persona. A frontier model handed a safety-tuned system prompt might do these same jobs without reaching for the shortcut, so read this as the harm one fixed prompt drew out, not the safest each model can be pushed to behave. That the weakest model was safest here is incompetence, not alignment, and I would not read the strongest model's number as a ceiling either.
- The runs are not independent, so treat my p-values as generous. Thirty-five runs means seven jobs repeated five times, not thirty-five separate observations. Repeats of the same job are correlated, so the true effective sample is smaller than the count suggests and any test on it reads better than it should. So I re-ran the capability test the honest way, treating each job as the unit rather than each run: I resampled the seven jobs with replacement and recomputed the climb. It holds in 98% of resampled job sets, and six of the seven jobs rise from the weakest model to the strongest on their own. The trend barely moves when you account for the clustering, because it is not carried by one or two lucky jobs. The guardrail comparison does not get this treatment because it was not close to significant to begin with, so it does not survive anything.
- Small samples, and one judge. The strongest tier is the thinnest: three passes over the job set, and some of those runs ran out of turns before the agent finished, which makes them harder to read. It is the number I trust least. On the other side, the harm calls were made by a different model from the one doing the jobs, and it agreed with my own labels every time. That is the part I trust most.
One note on provenance: the numbers above come from three runs on 2026-07-14, one per model tier, on a seven-job version of the harness. It has grown since, so if you pick it up you will find more jobs than the seven reported here.
So: directional, small, and open. The method and the harm scoring are the real asset here, more than any single percentage.
Why I think it matters anyway
The reflex when an agent misbehaves is "use a smarter model." On the safety axis, this small result says that makes it worse, not better. A more capable, more autonomous agent is more able to execute a destructive action, so the case for a runtime guardrail gets stronger with every model release, not weaker. That is the opposite of treating guardrails as training wheels you outgrow.
How I set it up
I gave an agent seven jobs modeled on real, documented agent failures, taken from a public registry of agent incidents (AREDB) rather than from my own product's rule list. If I had written the jobs around the things my guardrail already catches, I would have been grading my own homework, and the harm numbers would mean nothing.
Each job has a safe way to do it and a dangerous shortcut. "A customer's record is wrong, fix it" can be a scoped update, or a blanket delete that wipes everyone else too.
I ran each job two ways. Once in audit mode , where the guardrail watches but never steps in. That is the control: the agent cannot tell it is there. And once in enforce mode , where the guardrail blocks the dangerous call and hands back the safe path.
Two things got scored per run: did the agent finish the job, and did it take a harmful action getting there. A separate, independent checker decided whether an action was actually dangerous, so the harm number does not lean on the same model that ran the job.
Then the part I cared about most: I ran the same seven jobs across three models of increasing capability, holding everything else fixed.
Where this came from
This began as a follow-up to Forge, Antoine Zambelli's Show HN from Texas Instruments, which showed a simple guardrail layer taking an 8B model from 53% to 99% on multi-step agent tasks. Forge measured competence : does the harness help the agent finish the job. Lilian Weng has made a similar case from the safety side in Harness Engineering for Self-Improvement, that the layer between the raw model and the real world matters as much as the model's raw intelligence.
I wanted the question Forge did not measure. Does a guardrail make an agent safer , and does that matter more or less as models get stronger? Worth being clear about how the two results differ: Forge's headline is completion and mine is harm, they are different axes, and I did not reproduce a completion gain.
Tell me where this is wrong
If you think this is wrong, tell me exactly where: the task design, the harm scoring, or the statistics. Those are what I am least sure of and most want pushed on. Being wrong here is cheap, and I would rather be corrected in the open than stay quietly right. Reply wherever you found this, or find me on Discord.