AI Agent 修复了 Bug，但它还改变了什么？Trail of Bits 开源自动化补丁与回归检测插件

今天，我们正式开源发布 patching-bugs 和 regression-hunt，这两款插件允许工程师将补丁开发与回归检查工作委托给 Agent。我们的 patching-bugs 插件能够复现报告的 Bug、开发修复方案并进行测试。regression-hunt 插件则用于检查变更是否破坏了此前正常工作且受支持的行为。工程师可以指导工作并审查结果；项目维护者也可以提供自己的安全假设和补丁规范，确保 Agent 在项目的既定规则内运行。

这两款插件是我们在与 OpenAI 联合发起的 Patch the Planet 计划中构建的。在该项目中，我们为众多广泛使用的开源项目共同编写了数百个安全补丁。基于我们在 Scapy 等项目上取得的重大成功，这些插件融合了我们对已合并补丁的深度分析，以及工程师数千次 Codex 会话所积累的经验。

这些插件为维护者提供了结构化的工作流、证据支撑、独立验证和最佳实践，以改善补丁质量与规范（patching hygiene）。在我们的初步评估中，相较于仅被提示“审查 PR 中的回归缺陷”的基线 Codex Agent，regression-hunt 的误报率降低了 25%。

标准化补丁流程

我们在 9 月份的文章 中曾报告过 Patch the Planet 项目的部分补丁引入了回归缺陷。自那以后，我们审查了由 Trail of Bits 工程师编写并被上游维护者合并的每一个补丁。审查确认，在合并的 130 多个 PR 中共引入了 21 处回归：其中 16 处功能性回归、1 处性能回归以及 4 处构建/测试回归。没有一处破坏了原有的安全防护机制。

（“回归”指的是某项变更引入的新缺陷，该缺陷破坏了真实用户或攻击者可触及路径上原本正确且受支持的行为。）

针对每个补丁，Agent 会搜寻潜在的回归问题，通过在变更前后运行概念验证（PoC）来逐一确认，随后将结果移交给工程师进行审查。

审查过程存在大量噪声：如今漏洞披露的速度和规模前所未有，为了让补丁开发跟上步伐，整个行业必须提高信噪比。在单次运行生成的 58 个候选对象中，确认了 21 处回归（准确率约为 36%）。但由于 Agent 会采用多种不同的途径并得出不同的结论，每次运行都会产生更多需要人工排查的候选对象。

这使我们得出了两个结论：

1. 要想减少引入的回归缺陷，我们需要一种编写补丁的标准方法，并且每次都要执行相同的检查。
2. 要想捕获仍旧引入的回归缺陷，我们需要一种标准检查流程，并对何为回归制定明确的书面规则。

将这两个步骤标准化，还能让每次运行都具有可重复性和可比性。没有任何固定的清单能捕获所有回归缺陷，因此我们从工程师的补丁会话中收集 Agent 的运行轨迹（trajectories），并将早期版本失败的任务作为下一版本的基准测试集（见图 1）。

图 1：生成与评估补丁的反馈循环。

我们如何设计这些插件

我们主要参考了两个数据源来指导插件的设计：对所有已合并至上游的补丁审查，以及工程师编写这些补丁时进行的数千次 Codex 会话。

由此总结出了塑造 patching-bugs 和 regression-hunt 插件的四大原则：

1. 以最精炼的语言提供明确导向的指导（Opinionated guidance）。 我们给予 Agent 充分的自主权，仅记录 Trail of Bits 对“何为好补丁”以及“何为真正回归”的判断标准。目标是用尽可能少的引导，指引 Agent 达成预期结果。
2. 模块化步骤。 你不必完整运行任何一个插件的端到端流程，两者都提供了多项独立技能（skills），既可串联使用，亦可单独调用。
3. 严格的输入与输出 Schema。 每个步骤都接收并返回严格定义的模式（Schema），包括断言（claim）、代码版本、必要检查、原始证据以及未决问题。固定的 Schema 减少了多次运行之间的变异性，并允许每个步骤独立验证前一步骤的工作，而不是盲目信任其摘要。
4. 对每一项断言提供实时 PoC 验证。 每一个结论都必须依赖于在真实部署环境中运行的 PoC：即一个已构建并处于运行状态的项目实例，且必须通过真实用户会使用的接口进行访问。在复现 Bug、验证修复和确认回归时，如果某个步骤无法提供实时 PoC，其结果即视为“无定论”（inconclusive）。

regression-hunt 降低误报率

regression-hunt 插件的运行流程如下（图 2）：

1. 枚举候选回归（Enumerates）： 跟踪变更代码（diff），经由其调用者一直追溯到真实的公开接口。
2. 复现（Reproduces）： 在系统的实时部署环境中逐一复现每个候选回归。
3. 重放（Replays）： 由一个拥有**全新上下文的 Agent（fresh-context agent）**在实时部署环境中重放 PoC。
4. 裁决（Judges）： 评判前面所有步骤的结果，以验证证据是否符合回归的判定要求。

图 2：regression-hunt 工作流。

除了上述四大原则外，该插件还有两个至关重要的核心维度：

1. 定义何为回归： 插件对什么是回归、更重要的是对什么不是回归有非常严格的标准。举证责任完全落在 Agent 身上（特别是在“裁决 Agent”步骤中），必须证明有充分证据表明疑似回归并非误报、不完整修复、潜在 Bug 或原先既有的 Bug 等。唯有通过所有这些关卡，插件才会报告回归。
2. 引入威胁模型（Threat Model）： 如果项目提供了威胁模型，调度器会刻意将该文档作为全局上下文的一部分传递给每一个 Agent 步骤。这是为了强制要求每个步骤/Agent 在得出结论之前，都必须对照威胁模型进行核验。

我们在审查合并 PR 时总结出的 21 处确认回归，构成了评估 regression-hunt 的标准数据集。我们使用整套数据对比了该插件与仅提示“审查 PR 中的回归缺陷”的基线 Codex Agent。

在这些测试运行中，regression-hunt 产生的误报数量比基准减少了 25%（95% 置信区间为 14% 至 38%）。然而，其严苛的证据要求也导致部分已知回归被归类为“临界状态（borderline）”：即插件识别出了可能的回归，但未能收集到足够的证据予以完全确认。因此，降低误报率的同时，在能够确定性报告的已知回归数量上也存在一定权衡。

接下来，我们将探究这些案例停留在临界状态的原因，并改进插件的证据搜集策略。我们的目标是在证据不充分时向维护者提供更明确信息的同时，确认更多真正的回归。

patching-bugs 提炼数千次 Agent 会话的最佳实践

patching-bugs 插件浓缩了我们分析 Trail of Bits 工程师数千次 Codex 会话得出的所有最佳实践（图 3）。除了更为传统的功能（如在实时部署中复现以及独立的对抗性审查）之外，该插件还着重强调：

1. 三臂证明法（The three-arm proof）： 针对三种规范运行完全相同的复现流程：原始代码必须失败，打补丁后的代码必须通过，完全回滚补丁后必须再次失败。通过移除补丁重新触发 Bug，能够提供更强有力的证据，证明确实是补丁修复了 Bug，而非由于无关的环境或测试变动所致。
2. 基于不变量的修复（Invariant-guided repair）： 补丁致力于修复底层被破坏的安全不变量（invariant），而非仅仅修复特定表象的 Bug。这有助于使补丁更具泛化能力，覆盖同类 Bug 的潜在变体。
3. 整合回归分析： 插件强制运行 regression-hunt，以确保任何补丁都经历相同的标准流水线检查，排查潜在错误。

图 3：patching-bugs 工作流。

插件的后续演进方向

patching-bugs 和 regression-hunt 都将持续迭代演进。我们的工程师目前正在 Patch the Planet 工作中使用它们，我们也在持续收集使用数据，观察哪些机制有效、哪些无效。这些数据连同更完善的评估体系，将塑造下一版本的能力：

• 评估 patching-bugs： 我们正在构建一套评测机制，用于衡量该插件生成的补丁能否在不引入回归的前提下修复报告的 Bug 及其变体。
• 从 Agent 运行轨迹（Traces）中学习： 我们正在分析来自工程师会话的第一批追踪级数据。它将揭示 Codex 是否遵循了每个步骤所需的规则和定义、跳过了哪些步骤，以及在何处证据最薄弱。
• 提升插件的可配置性： patching-bugs 已经能够读取项目自身的修补指南（下文会详细介绍）。接下来，我们希望维护者能够自主选择是否最小化补丁集；或者在更广泛的覆盖值得承受更多误报时，放宽 regression-hunt 的规则以呈现更多候选缺陷。

这些插件沉淀了我们发现的促使 Agent 正确打补丁的最佳实践，而且当你告知它们项目的运作方式时，它们的效果会进一步提升。

当你输入以下两类项目上下文知识时，插件的表现最佳：

1. THREAT_MODEL.md：告诉 Agent 什么是漏洞、什么是回归。regression-hunt 插件会将其传递给每一个步骤，因此每个 Agent 都必须根据该文件核验结论。它应当明确定义：受到破坏后会产生严重后果的资产、信任边界、访问控制、受支持的配置、关于何为漏洞与非漏洞的规范，以及有效 Bug 报告所需的证据（参考示例）。
2. PATCHING_GUIDELINES.md（或 AGENTS.md）：告诉 Agent 和 patching-bugs 你的项目期望如何编写修复代码。它应当概述如何构建和测试项目、修复规模应当控制在多大、修复代码应当置于何处、如何呈现拟议的变更，以及回归测试必须证明什么（参考示例）。

在与 GPAC 合作开展 Patch the Planet 项目时，我们的工程师编写了威胁模型和针对该项目的修补技能（patching skill），其中记录了 GPAC 维护者希望如何复现 Bug、修复需要哪些证据，以及补丁必须保持哪些原有行为。Liboqs 和 validator.js 也按照我们的指导发布了各自的威胁模型。

这些文件针对你的项目、代码规范和运行方式描述得越详尽、越具体，你获得的输出效果就会越好。

整个行业需要将重心转向补丁修复

随着 AI 推动漏洞挖掘的极速发展，我们需要同样迅速地将这些发现转化为可靠的修复方案。解决方案必须兼顾速度与质量：Agent 需要理解漏洞报告，决定需要修改的内容，并验证补丁在修复问题的同时不会引入回归缺陷。我们的 patching-bugs 和 regression-hunt 插件将我们在 Patch the Planet 中积累的经验引入到该工作流中，协助维护者从一份 Bug 报告平稳过渡到能够自信审查并发布的合格补丁。

欢迎安装 patching-bugs 和 regression-hunt，在你的代码仓库中添加 THREAT_MODEL.md 与 PATCHING_GUIDELINES.md，并在处理下一个 Bug 报告时尝试运用它们。为工具赋予你的项目上下文，以便它们能帮助你聚焦正确的修复方案并加快验证速度。欢迎向我们提交反馈意见，指出我们的默认规则在哪些方面不契合你的项目。这些真实的实际案例将帮助我们更好地打造下一个版本。

Today we’re publishing patching-bugs and regression-hunt, two plugins that let engineers delegate patch development and regression checks to agents. Our patching-bugs plugin reproduces reported bugs, develops fixes, and tests them. The regression-hunt plugin checks whether any changes break previously working and supported behavior. Engineers can direct the work and review their results; maintainers can supply their own security assumptions and patching requirements, so the agents work within the project’s rules.

We built patching-bugs and regression-hunt through Patch the Planet, our joint initiative with OpenAI, where we’ve co-authored hundreds of security patches for widely used open-source projects. Motivated by our major successes on projects like Scapy, our plugins draw on our analysis of merged patches and thousands of our engineers’ Codex sessions.

The plugins provide maintainers with a structured workflow, evidence, independent checks, and best practices to improve their patching hygiene. In our preliminary evaluations, regression-hunt produces 25% fewer false-positive findings than a baseline Codex agent prompted to “review the PR for regressions.”

Standardizing the patching process

Our September post reported regressions introduced by some of our Patch the Planet patches. Since then, we’ve reviewed every patch that a Trail of Bits engineer authored and maintainers merged upstream. The review confirmed 21 regressions from over 130 PRs merged: 16 functional, one performance, and four build/test. None broke a security protection.

(“Regression” refers to a new defect introduced by a change that breaks previously correct and supported behavior along a path a real user or attacker can reach.)

For each patch, an agent looked for potential regressions, confirmed each one with a proof-of-concept (PoC) run before and after the change, and then passed the results to an engineer for review.

The review was noisy: today’s bug disclosures arrive at an unprecedented speed and scale, and for patching to keep pace, the industry needs to increase its signal-to-noise ratio. The 21 confirmed regressions were identified among 58 candidates in a single run (~36% precision). But because agents take many different approaches and reach different results, each run generates more candidates to manually sort through.

That led us to two conclusions:

1. To introduce fewer regressions, we needed a standard way to write patches, with the same checks applied every time.
2. To catch the regressions we still introduce, we needed a standard way to check for them, with written rules for what counts as a regression.

Standardizing both steps also makes every run repeatable and comparable. No fixed checklist will catch every regression, so we collect agent trajectories from our engineers’ patching sessions, and use the tasks that an earlier version failed to benchmark for the next one (figure 1).

Figure 1: Feedback loop for generating and evaluating patches.

How we designed the plugins

We used two primary sources to guide the design of our plugins: a review of every patch we merged upstream and the thousands of Codex sessions our engineers used to write those patches.

This led to four principles that shaped our patching-bugs and regression-hunt plugins:

1. Opinionated guidance in as few words as possible. We give the agent wide latitude and record only Trail of Bits’ judgment of what constitutes a good patch and a real regression. The goal is to direct the agent towards the desired result with the least possible guidance.
2. Modular steps. You don’t have to run either plugin end-to-end, as both provide multiple skills that can be used in tandem or in isolation.
3. Strict input and output schemas. Every step takes and returns a defined schema that includes the claim, the code revision, the required check, the raw evidence, and open questions. Fixed schemas reduce variability across runs and allow each step to verify the previous step’s work rather than trusting its summary.
4. A live proof of concept for every claim. Every conclusion depends on a PoC run against a live deployment: a built, running instance of the project, reached through the interfaces a real user would use. When reproducing a bug, proving a fix, and confirming a regression, if a step can’t produce a live PoC, its result is inconclusive.

regression-hunt reduces false positive rates

The regression-hunt plugin (Figure 2):

1. Enumerates candidate regressions by tracing the changed code (the diff) through its callers to its real interfaces.
2. Reproduces each candidate regression on a live deployment of the system.
3. Replays the PoC on the live deployment on a fresh-context agent.
4. Judges the results of all the previous steps to validate that the proof meets the requirements of a regression.

Figure 2: The regression-hunt workflow.

In addition to the four principles above, there are two core aspects that are critical to the plugin:

1. Defining a regression: The plugin has strong opinions about what a regression is and, more importantly, what a regression is NOT. The burden of proof is placed on the agent (specifically during the “adjudication agent” step) to prove that there is sufficient evidence to show that the suspected regression is not a false positive, an incomplete fix, a latent bug, a pre-existing bug, and so on. The plugin only reports a regression if all of these gates are passed.
2. Incorporating a threat model: If a threat model is available, the orchestrator will intentionally pass this document as part of the global context for each individual agentic step. This is an attempt to require each step/agent to refer back to the threat model before reaching its conclusion.

Our review of merged PRs yielded a set of 21 confirmed regressions that we could use to evaluate regression-hunt. We used the whole set to compare the plugin with a baseline Codex agent prompted to “review the PR for regressions.”

Across those runs, regression-hunt produced 25% fewer false-positive findings than the baseline, with a 95% confidence interval of 14% to 38%. Its strict evidence requirements also left some known regressions classified as borderline: the plugin identified a possible regression but did not gather enough evidence to confirm it. Fewer false positives therefore came with a tradeoff in how many known regressions it could confidently report.

Next, we’ll investigate why these cases remained borderline and improve the plugin’s strategies for gathering evidence. Our goal is to confirm more real regressions while giving maintainers clearer information when the evidence remains inconclusive.

patching-bugs streamlines best practices from thousands of agent sessions

The patching-bugs plugin is a distillation of all the best practices we’ve identified by analyzing thousands of Codex sessions by Trail of Bits engineers (Figure 3). In addition to the more traditional capabilities (such as reproduction on live deployments and independent adversarial review), the plugin emphasizes:

1. The three-arm proof : The proof runs the same reproduction against three specifications: the original code must fail, the patched code must pass, and a complete revert must fail again. Bringing the bug back by removing the patch provides stronger evidence that the patch—rather than an unrelated environment or test change—actually fixed the bug.
2. Invariant-guided repair: The patch attempts to fix the underlying, violated security invariant rather than fix the specific bug. This helps generalize the patch for possible variants of the same bug class.
3. Incorporates regression analysis: The plugin mandates that regression-hunt is run to ensure that any patch goes through the same standard pipeline to check for mistakes.

Figure 3: The patching-bugs workflow.

What’s next for the plugins

Both patching-bugs and regression-hunt will continue to evolve. Our engineers are using them in their Patch the Planet work now, and we’re collecting data on how they’re being used, what works, and what doesn’t. That data, along with better evaluations, will shape the next version’s capabilities:

• Evaluate patching-bugs. We’re building an evaluation that measures whether the plugin’s patches fix the reported bug and its variants without introducing regressions.
• Learn from agent traces. We’re analyzing the first batch of trace-level data from our engineers’ sessions. It will show whether Codex follows the rules and definitions required at each step, which steps it skips, and where its evidence is weakest.
• Make the plugins more configurable. patching-bugs already reads a project’s own patching guidelines (more on that in a bit). Next, we want maintainers to be able to choose whether to minimize patch sets—or, when broader coverage is worth more false positives, to loosen regression-hunt’s rules to surface more candidates.

These plugins encode best practices we’ve found for an agent to patch a bug properly, and they improve when you tell them how your project works.

The plugins work best when you input two kinds of project knowledge:

1. A THREAT_MODEL.md , which tells agents what counts as a vulnerability or a regression. The regression-hunt plugin passes it to every step, so each agent has to check its conclusions against the file. It should define the following: assets whose compromise matters, trust boundaries, access controls, supported configurations, specifications on what is or isn’t a vulnerability, and evidence required for a valid bug report (example).
2. A PATCHING_GUIDELINES.md (or AGENTS.md), which tells agents and patching-bugs how your project wants fixes written. It should outline how to build and test the project, how big a fix should be, where a fix should go, how to present proposed changes, and what a regression test must show (example).

For our Patch the Planet work with GPAC, our engineers wrote a threat model and a project-specific patching skill that captures how GPAC’s maintainers want bugs reproduced, what evidence a fix needs, and what a patch must preserve. Liboqs and validator.js have also published threat models in accordance with our guidance.

The more comprehensive and specific these files are to your project, code conventions, and operating style, the better the output you’ll receive.

The industry needs to focus on patching

As AI accelerates bug discovery, we need to turn those findings into reliable fixes just as quickly. Solutions need to balance speed and quality: an agent needs to understand a report, decide what needs to change, and check that a patch fixes the issue without introducing regressions. Our patching-bugs and regression-hunt plugins bring what we’ve learned from Patch the Planet into that workflow, helping maintainers move from a bug report to a patch they can confidently review and ship.

Install patching-bugs and regression-hunt, add a THREAT_MODEL.md and a PATCHING_GUIDELINES.md to your repository, and put them to work on your next bug report. Give the tools your project’s context so they can help you focus on the right fix and validate it faster. Then share your feedback, including comments about where our defaults don’t fit your project. Those real-world cases will help us build the next version.