1Password 的 AI 补丁基准测试具有误导性

1Password 于 2026 年 8 月 6 日发布的 FLAWED 报告，向防御方呈现了一幅有关 AI 补丁能力的误导性图景。其标题称，模型仅有 26% 的时间能够生成干净的修复方案。该数字包含了刻意要求 Agent 应用错误修复方案的实验，也包含 Agent 无法编译或测试其补丁的实验。

该报告可能因劝阻防御方使用能够帮助其修复更多漏洞的技术，而降低他们的工作成效。那些按字面理解其标题的团队，可能会让本可修复的漏洞得不到处理。

我们希望我们的工作能帮助防御方修复更多漏洞。本文分享我们从咨询项目和 Patch the Planet 中获得的、关于人工与 Agent 补丁质量的真实数据。我们还将发布两项 Agent skills：post-patch-validation，帮助 Agent 测试修复方案；以及 review-walkthrough，帮助工程师审查这些修复方案。

实验如何得出一个误导性的标题

我们审查了 1Password 的代码和数据，发现其中有四项选择，使其 26% 的干净修复率无法成为日常补丁工作的可靠参考。

• 样本特意选择了难以修复的问题。 作者选择了六个漏洞，因为它们的修复方案较为复杂。这些漏洞的干净修复率介于 3% 至 60% 之间，因此平均值高度依赖于哪些漏洞被纳入列表。
• 两个提示词要求 Agent 应用错误修复方案。 这些提示词占数据的 22%。将它们与普通修复尝试合并，会使报告中的比率部分取决于研究人员多频繁地选择向 Agent 提供错误建议。
• 超过三分之一的试验禁止测试。 一种评估模式阻止 Agent 构建或运行代码，占数据的 36%。该标题将这些试验，与 Agent 能够测试其补丁并根据结果采取行动的实验合并计算。
• 模型使用了不同的推理设置。 GPT-5.5 以中等 effort 运行，Opus 4.8 则以高 effort 运行。这些是工具的默认设置。两个模型均未在其可用的最高设置下接受测试，作者也没有衡量提高 effort 对结果有何影响。

1Password 的标题还掩盖了其自身数据中的一项有用结果。我们重新分析了补丁，并记录了研究一同发布的测试结果，只保留 Agent 能运行代码、且未被要求应用错误修复方案的试验。**在这些试验中，1Password 所用模型生成的 3,067 个补丁中，有 2,634 个（86%）阻止了所提供的漏洞利用。**我们排除了被研究归类为参考过上游修复方案的运行结果。阻止该漏洞利用并不能证明修复完整，但这些结果显示，在合理工作条件下存在实用的补丁能力，而标题未能传达这一点。

指令和评分方式还引入了更多问题，其中数项也被 Davi Ottenheimer 指出：

• 停止规则与评分标准相互矛盾。 获得概念验证漏洞利用的 Agent，被指示在其补丁击败该利用后立即停止。评分器随后却评估了所提供漏洞利用未触及的脆弱路径。
• 评分会惩罚预期中的行为变化。 Agent 被要求不修改现有测试，尽管正确修复可能需要更新测试以反映行为改变。我们发现，8% 的 ActiveMQ 判定将预期中的行为变化作为回归错误进行惩罚。
• 自动评分与人工审查不一致。 对自身补丁进行评分的模型，在完整的五分类结果上仅有 65.9% 的已审查案例与人工审查者一致。对于原始漏洞是否已修复，一致率为 87.7%；对于是否引入了新漏洞，一致率为 70.5%。(表 24)
• 更换审查者会改变结果。 两个模型对同一批补丁中 36.8% 的补丁给出了不同结果。标题对它们的评估取了平均值。(表 20)
• Linux 参考修复方案本身包含漏洞。 作者发现，有 248 个生成的补丁重复了上游修复方案中的一个 off-by-one 错误。自动评分器仅在其中 24 个补丁中捕获了这一新漏洞。(第 4.4 节)
• Chromium 评分器接受不完整的修复。 即便许多补丁在回调中仍留下 use-after-free 漏洞，它仍将其标记为干净。(第 4.9 节)

这些评分错误既可能惩罚有效修复，也可能让仍有漏洞的补丁通过。结合人工挑选的样本和刻意设置的错误指令，该报告没有可信依据来支撑其标题。防御方不应将 1Password 标题中的比率当作衡量 AI 补丁能力的严肃指标。

即使在理想条件下，开发者也会把八分之一的修复做错

理解 Agent 的失败，也需要理解开发者在多大程度上会提交不完整的修复方案。我们的安全咨询工作为我们提供了详尽记录，展示开发者如何修复其自有软件中的漏洞。我们向客户提供详细的漏洞报告，随后开展一次“修复审查”，以检查其拟议补丁是否完整解决了问题。

我们的记录将每个漏洞与开发者首次提出的修复方案，以及我们对该方案是否有效的评估关联起来。它们保留了开发者在问题被视为解决前修改过的失败尝试。

我们审查了 2024 年至 2026 年间 236 次 Trail of Bits 安全评估中、针对 2,265 个漏洞提交的首次修复。开发者维护受影响的软件，获得了我们工程师撰写的详细报告，并且知道我们会审查其补丁。即使在这些有利条件下，仍有 283 个首次修复未能完整解决所报告的问题：占 12.5%，即八分之一。

展示首次修复提交结果的图表

在考虑同一次评估中多个修复方案的情况下，95% 置信区间为 10.5% 至 14.5%。有时，我们会在非正式交流中指出客户补丁里的错误，而客户会在正式修复审查之前将其纠正。这些早期失败可能从未出现在审查记录中，因此我们的数据可能低估了首次尝试失败的数量。我们还排除了现有记录无法确定修复是否有效的案例。若要直接与 Agent 进行比较，需要在相同任务和工作条件下进行。

我们的补丁在真实项目中表现如何

通过我们与 OpenAI 联合发起的 Patch the Planet 计划，Trail of Bits 已为广泛使用的开源项目共同编写了数百个补丁。Agent 在工程师指导工作并检查结果的情况下编写补丁。随后，各项目维护者决定合并、修改还是拒绝每一项提交。

维护者如何审查 Patch the Planet 补丁

我们检查了数据集中所有截至 2026 年 9 月 14 日已被维护者合并或关闭的 Patch the Planet 提交的公开审查记录：共 186 个 pull request。1Password 的基准测试使用了六个因修复复杂而被选中的漏洞。

维护者合并了我们 186 个 pull request 中的 126 个，接受率为 67.7%。在这 126 个 pull request 中，有 91 个（72.2%）被维护者接受了我们最初提出的安全修复方案。

审查结果	PR 数量	占已合并 PR 的百分比
合并总数	126	100%
接受，且未观察到安全相关修改	91	72.2%
接受，但观察到安全相关修改	33	26.2%
无法确定	2	1.6%

表 1：维护者对 126 个已合并 Patch the Planet pull request 要求的修改。安全相关修改包括对拟议修复方案的修正，以及对其安全覆盖范围的扩展。

维护者的接受并不能证明每个补丁都正确。

维护者关闭了其余 60 项提交而未予合并。大多数提交被其他工作取代，或因政策、流程、范围或维护原因而被拒绝。其中四项明确因技术理由被拒绝。

关闭原因	PR 数量	占未合并关闭 PR 的百分比
未合并即关闭总数	60	100%
被替代、重新实现或在其他位置重新合入	36	60.0%
政策、流程、范围或维护原因	14	23.3%
与另一项修复重复或趋同	3	5.0%
明确因技术理由被拒绝	4	6.7%
其他或无法确定	3	5.0%

表 2：维护者未合并即关闭 60 个 Patch the Planet pull request 的原因

其中一项被关闭的提交是我们的 freenginx 补丁。

一名维护者和一个 Agent 引入了同样的 freenginx 崩溃问题

1Password 的案例研究考察了一个 Patch the Planet 修复方案，该方案针对 freenginx 嵌入式 Perl 模块中的内存安全漏洞。一个 Agent 在 Trail of Bits 工程师的指导下编写了我们的补丁。它遗留了一条脆弱代码路径，并在请求清理期间引入了新的崩溃。该论文对我们补丁的批评是正确的。

维护者关闭了我们的 pull request，并提交了另一项独立修复。该修复覆盖了全部三条脆弱代码路径，却在清理期间引入了同样的崩溃。论文也记录了维护者的这一回归问题。

两位作者都遇到了同一个陷阱。原始漏洞允许 Perl 在 freenginx 使用一个回调之前将其销毁。

两种修复方案都让回调保持存活，以便 freenginx 稍后使用它。但如果请求先超时，freenginx 会使该请求不可用，然后释放回调。释放该回调可能会运行仍尝试使用该请求的 Perl 代码，从而导致 worker 崩溃。两位作者都忽视了其修复方案稍后可能在清理期间引发的问题。要捕获这一问题，必须超越原始漏洞，考察请求提前结束时会发生什么。

两位作者，一位人类和一个 Agent，分别独立工作，却在同一个漏洞上犯了相同错误。决定是否使用 Agent 的读者，需要知道其失败与人类开发者的失败相比如何。要确定哪一方更可靠，必须在可比条件下对两者进行测量。

我们检查了补丁合并后发生的情况

我们检查了 Patch the Planet 项目中约 33,500 次后续提交。当后续提交修改了我们的补丁曾改动过的文件时，我们会调查该提交是否修复了我们的补丁引入的问题。针对每个疑似回归，一个 Agent 会尝试通过概念验证展示其影响。其他 Agent 和我们的工程师随后会对这些发现提出质疑和验证。

审查至少发现了十个功能性 bug、四个构建、测试或发布自动化 bug，以及一个性能 bug。未发现可被利用的安全漏洞。以下两个示例说明了我们识别出的问题：

• 在 go-jose 中，PR #240 修复了缺失头部导致的崩溃，但暴露了一个已有的验证缺口：即使密钥长度与声明的算法相矛盾，加密消息仍可成功处理。PR #266 在解密前增加了明确的密钥长度检查。
• 在 Noble FROST 中，PR #250 返回缓存的第二轮结果之前，未先检查重试是否包含相同的已认证 transcript。因此，被更改或过期的重试数据可能绕过该检查。维护者通过修正该行为，在返回缓存结果前根据原始 transcript 验证重试数据。

我们正将这项调查扩展到我们编写的每一个补丁，包括有维护者贡献的补丁。这些发现将帮助我们增加检查项，以在提交未来补丁前捕获此类失败。

用于生成更好安全补丁的 Agent skills

我们将与本文同步发布两项 Agent skills：post-patch-validation，帮助 Agent 测试安全修复方案；以及 review-walkthrough，帮助工程师审查代码变更。

Post-patch-validation 是我们新编写的一项 skill，旨在帮助 Agent 在提交补丁供审查前捕获不完整修复和回归问题。上文所述的 Patch the Planet 工作并未使用该 skill。

该 skill 以一份漏洞报告，以及补丁前后的代码为起点。它引导 Agent 完成四项任务：

• 复现原始 bug。 Agent 编写一项检查，该检查必须在存在漏洞的代码上失败、在打补丁后的版本上通过。一项在两个版本上都通过的测试，无法证明修复有效。
• 测试通往同一失败结果的另一条路径。 该 skill 要求至少有一个基于 bug 根本原因的不同变体，例如不同的调用者或清理路径。
• 检查回归和新漏洞。 它对比本应保持不变的行为，并测试修改后代码周围的安全属性。计划还必须包括项目测试、sanitizer 检查或有界 fuzzing 运行。
• **将

1Password’s FLAWED report, published on August 6, 2026, gives defenders a misleading picture of AI patching. Its headline says models produced clean fixes only 26% of the time. That figure includes experiments that deliberately instructed agents to apply the wrong fix, along with experiments in which agents could not compile or test their patches.

The report risks making defenders less effective by discouraging them from using technology that could help them fix more vulnerabilities. Teams that take its headline at face value may leave repairable vulnerabilities unaddressed.

We want our work to help defenders fix more vulnerabilities. This post shares real-world data on human and agent patch quality from our consulting projects and Patch the Planet. We’re also releasing two agent skills: post-patch-validation to help agents test fixes, and review-walkthrough to help engineers review them.

How the experiment produces a misleading headline

Our review of 1Password’s code and data found four choices that make its 26% clean-fix rate a misleading guide to ordinary patching work.

• The sample was selected for difficult fixes. The authors chose six vulnerabilities because their fixes were complex. Clean-fix rates ranged from 3% to 60% across those bugs, so the average depends heavily on which vulnerabilities made the list.
• Two prompts tell agents to apply the wrong fix. Those prompts account for 22% of the data. Combining them with ordinary repair attempts makes the reported rate depend partly on how often the researchers chose to give agents bad advice.
• More than a third of the trials prohibit testing. One evaluation mode prevents agents from building or running code and accounts for 36% of the data. The headline combines those trials with experiments in which agents could test their patches and act on the results.
• The models ran at different reasoning settings. GPT-5.5 ran at medium effort and Opus 4.8 at high. These were the tools’ defaults. Neither model was tested at its highest available setting, and the authors did not measure how increasing effort affected the results.

1Password’s headline also obscures a useful result in its own data. We reanalyzed the patches and recorded test results published with the study, keeping trials where agents could run code and were not instructed to apply the wrong fix. In those trials, 2,634 of 3,067 patches generated by 1Password’s models (86%) blocked the supplied exploit. We excluded runs that the study classified as having consulted the upstream fix. Blocking that exploit does not establish a complete repair, but these results show useful patching capability under reasonable working conditions that the headline fails to convey.

The instructions and grading introduce further problems, several of which Davi Ottenheimer has also highlighted:

• The stopping rule and grading criteria disagree. Agents given a proof-of-concept exploit were instructed to stop once their patch defeated it. The grader then evaluated vulnerable paths that the supplied exploit did not exercise.
• The grading penalizes intended behavior changes. Agents were told to leave existing tests untouched, even though a correct fix can require updating tests to reflect changed behavior. We found that 8% of ActiveMQ verdicts penalized an intended behavior change as a regression.
• The automated grades disagree with human review. Models grading their own patches matched human reviewers on the full five-category outcome in 65.9% of reviewed cases. Agreement was 87.7% for whether the original bug was fixed and 70.5% for whether new bugs were introduced. (Table 24)
• Changing the reviewer changes the result. The two models assigned different outcomes to 36.8% of the same patches. The headline averages their assessments. (Table 20)
• The Linux reference fix contains a vulnerability. The authors found 248 generated patches that repeated an off-by-one error in the upstream fix. The automated grader caught that new vulnerability in only 24 of them. (Section 4.4)
• The Chromium grader accepts incomplete repairs. It marked many patches as clean even though they left a use-after-free vulnerability in a callback. (Section 4.9)

The grading errors can penalize valid fixes and let vulnerable patches pass. Combined with the handpicked sample and deliberately bad instructions, they leave the report without a credible basis for its headline. Defenders should not take 1Password’s headline rate seriously as a measure of AI patching ability.

Developers get one in eight fixes wrong under ideal conditions

Understanding agent failures also requires understanding how often developers submit incomplete fixes. Our security consulting work gives us a detailed record of how developers repair vulnerabilities in their own software. We give clients detailed vulnerability reports, then conduct a “fix review” to check whether their proposed patches fully resolve the issues.

Our records connect each vulnerability to the developer’s first proposed fix and our assessment of whether it worked. They preserve unsuccessful attempts that developers revise before an issue is considered resolved.

We reviewed the first fixes submitted for 2,265 vulnerabilities across 236 Trail of Bits security assessments from 2024 to 2026. The developers maintained the affected software, had detailed reports from our engineers, and knew we would review their patches. Even under those favorable conditions, 283 first fixes failed to fully resolve the reported issue: 12.5%, or one in eight.

Figure showing first fix submission outcomes

Accounting for multiple fixes from the same assessment, the 95% confidence interval is 10.5% to 14.5%. Sometimes we point out a mistake in a client’s patch during an informal conversation, and they correct it before the formal fix review. Those early failures may never appear in the review record, so our data can undercount failed first attempts. We also excluded cases where the available records did not establish whether the fix worked. A direct comparison with agents would require the same tasks and working conditions.

What happened to our patches in real projects

Through Patch the Planet, our joint initiative with OpenAI, Trail of Bits has co-authored hundreds of patches for widely used open-source projects. Agents wrote the patches with engineers directing the work and checking the results. Project maintainers then decided whether to merge, revise, or reject each submission.

How maintainers reviewed Patch the Planet patches

We examined the public review history of every Patch the Planet submission in our dataset that maintainers had merged or closed by September 14, 2026: 186 pull requests. 1Password’s benchmark used six vulnerabilities selected because their fixes were complex.

Maintainers merged 126 of our 186 pull requests, an acceptance rate of 67.7%. In 91 of those 126 pull requests (72.2%), maintainers accepted the security fix we originally proposed.

Review outcome	PRs	% of merged PRs
Total merged	126	100%
Accepted with no security-relevant revision observed	91	72.2%
Accepted with security-relevant revision observed	33	26.2%
Indeterminate	2	1.6%

Table 1: Changes requested by maintainers for 126 merged Patch the Planet pull requests. Security-related revisions include repairs to a proposed fix and expansions of its security coverage.

Maintainer acceptance does not establish that every patch is correct.

Maintainers closed the other 60 submissions without merging them. Most were superseded by other work or declined for policy, process, scope, or maintenance reasons. Four were explicitly rejected on technical grounds.

Reason for closure	PRs	% of closed PRs
Total closed without merge	60	100%
Superseded, reimplemented, or re-landed elsewhere	36	60.0%
Policy, process, scope, or maintenance reasons	14	23.3%
Duplicate or convergent with another fix	3	5.0%
Explicitly rejected on technical grounds	4	6.7%
Other or indeterminate	3	5.0%

Table 2: Reasons maintainers closed 60 Patch the Planet pull requests without merging

One of those closed submissions was our freenginx patch.

A maintainer and an agent introduced the same freenginx crash

1Password’s case study examines a Patch the Planet fix for a memory-safety bug in freenginx’s embedded Perl module. An agent wrote our patch under the direction of a Trail of Bits engineer. It left one vulnerable code path open and introduced a new crash during request cleanup. The paper’s criticism of our patch is correct.

The maintainer closed our pull request and committed a separate fix. That fix covered all three vulnerable code paths but introduced the same crash during cleanup. The paper documents the maintainer’s regression too.

Both authors encountered the same trap. The original bug allowed Perl to destroy a callback before freenginx used it.

Both fixes kept the callback alive so freenginx could use it later. But if the request timed out first, freenginx would make the request unusable and then release the callback. Releasing it could run Perl code that still tried to use the request, crashing the worker. Both authors missed a problem their fix could cause later, during cleanup. Catching it required looking beyond the original bug to what happened when a request ended early.

Two authors, one human and one agent, working separately, made the same mistake on the same bug. Readers deciding whether to use agents need to know how their failures compare with those of human developers. Establishing which is more reliable requires measuring both under comparable conditions.

We checked what happened after our patches were merged

We examined about 33,500 subsequent commits in Patch the Planet projects. When a later commit changed a file our patch had modified, we investigated whether it fixed a problem our patch had introduced. For each suspected regression, an agent attempted to demonstrate its impact with a proof of concept. Other agents and our engineers then challenged the findings.

The review found at least ten functional bugs; four build, test, or release automation bugs; and one performance bug. It found no exploitable security vulnerabilities. Two examples illustrate the problems we identified:

• In go-jose, PR #240 fixed a missing-header crash but exposed an existing validation gap, allowing encrypted messages to succeed even when their key length contradicted the declared algorithm. PR #266 added explicit key-length checks before decryption.
• In Noble FROST, PR #250 returned cached round-two results without first checking whether a retry contained the same authenticated transcript. Changed or stale retry data could therefore bypass that check. The maintainer corrected the behavior by validating retries against the original transcript before returning cached results.

We are extending this investigation to every patch we authored, including patches with maintainer contributions. The findings will help us add checks that catch these failures before we submit future patches.

Agent skills for better security patches

We are releasing two agent skills alongside this post: post-patch-validation to help agents test security fixes, and review-walkthrough to help engineers review code changes.

Post-patch-validation is a new skill we wrote to help agents catch incomplete fixes and regressions before submitting patches for review. It was not used in the Patch the Planet work described above.

The skill starts with a vulnerability report and the code before and after the patch. It guides the agent through four tasks:

• Reproduce the original bug. The agent writes a check that must fail on the vulnerable code and pass on the patched version. A test that passes on both revisions cannot demonstrate a fix.
• Test another path to the same failure. The skill requires at least one distinct variant based on the bug’s root cause, such as a different caller or a cleanup path.
• Check for regressions and new vulnerabilities. It compares behavior that should remain unchanged and tests security properties around the modified code. The plan must also include project tests, a sanitizer check, or a bounded fuzzing run.
• Treat broken test runs as inconclusive. A failed build or missing dependency must not be mistaken for evidence that a vulnerability was reproduced.

Failed checks give the agent specific problems to investigate and repair before submitting its patch. The skill saves the tests and results so maintainers can see what was checked.

To try post-patch-validation, install the skill and give your agent the vulnerability report and the vulnerable and patched revisions:

“Use post-patch-validation to validate the patch in HEAD against , using the vulnerability report in .”

Review-walkthrough helps engineers review the patches they are responsible for merging. It turns a branch’s complete diff into an interactive walkthrough that explains the changes in a logical reading order. Review findings appear beside the relevant code, where engineers can inspect them and draft their own comments. The walkthrough can also prepare a GitHub review for submission. Follow the quick start to generate a walkthrough for your branch.

These releases join our other public agent skills for improving security patches:

We publish these methods so other teams can use them to examine and improve their own patches.

What a useful patching benchmark should measure

A useful patching benchmark should measure whether agents help developers produce correct fixes and how much review those fixes require. The principles in our 2018 guide to evaluating fuzzing research apply here:

1. Choose a sample that matches the research question. Explain how the sample was chosen and which repair work it represents. Difficult cases can expose failure modes. General failure rates require a representative sample.
2. Measure the effects of working conditions. Give agents appropriate tools and instructions. Report model configurations and test how reasoning settings affect results. Report misleading prompts and restricted tool access separately.
3. Make correctness verifiable. Check that patches fix the vulnerability beyond the supplied exploit. Test for security, functional, and performance regressions. Validate grades against expert review and publish the tests, configurations, and results.
4. Show how results vary. Report per-vulnerability outcomes and variation across repeated attempts. Repeating trials on the same bugs cannot establish that those bugs represent everyday patching.
5. Measure what agents contribute to the repair process. Compare developers working with and without agents on comparable tasks and under comparable conditions. Measure initial patch quality and the review and revision needed to reach a correct fix.

We are optimistic about AI’s usefulness to defenders. Through Patch the Planet, we are committing engineering time to fixing vulnerabilities alongside the people who maintain the affected software. We examine failures so we can improve our methods.

We will keep putting agents to work on difficult security problems and making the tools and lessons public. We want other teams to test our conclusions and take these methods further. Our goal is to give maintainers without dedicated security teams the ability to find and fix vulnerabilities that would otherwise go unaddressed.