Show HN：基准测试显示，除非明确告诉 AI 哪里出错，否则它找不到 bug

现有大多数软件工程基准测试都会让编程 Agent 处理具体且定义明确的任务，通常会提供一个代码库，以及一份需要解决的用户报告问题。然而，随着用户将越来越广泛的目标委托给编程 Agent，一个自然的下一步是：Agent 不仅要判断如何完成有用的工作，还要判断哪些有用的工作需要完成。

一个被托付管理代码仓库的 Agent，应该能够在问题被报告之前自行判断哪里出了问题、哪些问题重要，以及如何解决它们。我们推出了面向这一开放式场景的基准测试 SWE-sweep。

给定一个包含许多并发 bug、且没有任何关于其性质或位置的信息的代码库，Agent 必须自主发现并修复尽可能多的 bug。

SWE-sweep 基于开源仓库构建。对于每个仓库，我们收集 issue—pull request 配对，然后识别出一个单独的 commit：在该 commit 中，最多数量的 bug 同时存在。

每项修复都会通过对应 pull request 中的隐藏测试进行评估，同时也会运行现有测试套件以检查是否引入回归。

成功要求 Agent 能够在较长的工作周期中探索并理解大型代码库，在不引入回归的前提下修复 bug，并管理彼此并非独立的修复之间的相互影响。

如何构建任务？

我们收集真实的 issue—pull request 配对，识别出许多此类 bug 共存的 commit，并保留那些能够在该仓库状态下复现其修复和测试的 bug。除与其他基准共享的多项质量过滤标准外，我们还进行了广泛过滤，以便只评估能够仅通过阅读仓库内容发现的 bug。

Agent 会获得什么？

Agent 会获得处于固定基础 commit 的一个仓库，以及一条宽泛的指令：尽可能发现并修复更多 bug。它不会获得 issue 描述、文件名、行号范围或任何其他针对特定 bug 的提示。

你可以在这里查看完整提示词。

如何评估 SWE-sweep？

我们会针对一组先前识别出的参考 bug 集合（见构建部分）为每项任务评分，统计 Agent 成功修复了多少 bug。

对于每项任务，我们都会使用两组测试运行 Agent 提交后的代码库。首先，我们恢复仓库原始测试套件并运行它，以验证既有行为没有被破坏。其次，对于每个 bug，我们运行一组隐藏测试；其中至少有一项测试会在未修改的代码库上失败，并在 bug 修复后通过（fail-to-pass）。只有当一个 bug 的全部隐藏测试通过，且原始测试套件仍然通过时，该 bug 才被视为已解决。

基准分数是所有仓库中已解决 bug 占全部 bug 的比例。Agent 做出的其他任何改动又如何处理？

基准中包含哪些 bug？你们如何保证任务可行？

我们会过滤 bug（由测试补丁和修复补丁表示），以确保任务可行。标准如下：

1. 测试补丁和修复补丁都能够独立应用到基础 commit。
2. 将修复补丁应用到基础 commit 后，所有目标测试均会通过；但至少有一项目标测试会在基础 commit 上失败（F2P 测试）。还可能存在一些在应用修复补丁前后均能通过的测试（P2P 测试）。
3. F2P 测试能够揭示基础 commit 中一个单一的、可发现的 bug。
4. 目标测试不会过度具体化；针对该可发现 bug 的任何合理修复都应能通过目标测试。
5. 目标测试不与基础 commit 的测试相矛盾。
6. 不同 bug 实例的目标测试之间不相互矛盾。

论文附录 A.2 对可行性进行了详细讨论。

什么样的 bug 是可发现的？

预期行为必须能够从仓库本身推断出来，例如通过文档、类型、现有测试、调用方、不变量、标准，或者明显不应出现的失败情况，如崩溃或数据丢失。后一类的使用极为保守；基准中的 bug 除了 2 个以外，全部都有描述预期行为的具体仓库契约。你可以在这里查看一些我们所说的_仓库契约_示例。我们投入了大量时间验证该基准的这一方面，更多细节可见论文附录。

Agent 做出的其他任何改动又如何处理？

任何基于历史记录构建的基准都必然会低估一个仓库中存在的 bug 数量。通过将 PandoraBench 限制为由上游修复确认的缺陷，我们确保基准中的每个 bug 都有强有力的证据支持：仓库维护者认为观察到的行为是错误的。因此，我们只会根据那些由上游修复确认、由可执行回归测试支持，并通过其他质量过滤标准的 bug 来评估 Agent 的改动。不过，如果 Agent 导致原始测试套件出现回归（Agent 已被明确告知应避免这一点），其得分将为 0%。这意味着，尽管 Agent 的其他改动不会由该 bug 集合计分，它们仍很可能是非破坏性的，并与仓库既有行为兼容。更多讨论请参阅论文中的 A.3 和 A.4。

更多推理时计算量有帮助吗？

重复尝试能够找回额外的 bug，但收益会递减。单次运行中后续的工作也可能撤销先前的修复，因此仅仅延长执行轨迹并不能保证改进。参见论文图 7。

Astra、Fable、5.5 等模型呢？

我们正在评估更多模型！当前模型选择是在我们提交 ICLR 论文时最终确定的。我们也在研究更高的推理模式，但这可能会使单次运行成本超过 1 万美元。我们还希望在排行榜上加入更多开放权重模型。

为什么选择 mini-swe-agent？其他脚手架或多 Agent 方案能否取得更高性能？

我们的论文包含一项使用 Claude Code 和 Codex 的消融实验。两者似乎都没有显著优于 mini-swe-agent（恰恰相反，mini-swe-agent 甚至明显优于 Codex）。这与许多其他基准的结果一致：mini-swe-agent 一直极具竞争力。不过，我们绝对希望推动更多关于 Agent 脚手架作用的研究，并将很快开放提交。

如何提交到排行榜？

公开提交即将推出。

Most existing software engineering benchmarks evaluate coding agents on concrete, well-specified tasks, commonly by providing a codebase together with a user-reported issue to resolve. However, as users delegate increasingly broad outcomes to coding agents, the natural next step is for agents to determine not only how to perform useful work, but also what useful work needs to be done.

An agent entrusted with a repository should be able to decide what is broken, which problems matter, and how to solve them before they are reported. We introduce SWE-sweep, a benchmark for this open-ended setting.

Given a codebase containing many concurrent bugs and no information about their nature or location, an agent must autonomously discover and fix as many bugs as possible.

SWE-sweep is constructed from open-source repositories. For each repository, we collect issue-pull request pairs, then identify a single commit where the maximum number of bugs are present at the same time.

Each repair is evaluated against hidden tests from the corresponding pull requests, along with the existing test suite to check for regressions.

Success requires agents to explore and understand a large codebase over long horizon work, repair bugs without introducing regressions, and manage interactions among fixes that are not independent.

How are tasks constructed?

We collect real issue–pull request pairs, identify a commit where many of those bugs coexist, and retain bugs whose fixes and tests can be reproduced at that repository state. Besides many quality filters shared with other benchmarks, we apply extensive filtering to evaluate only bugs that can be discovered from reading the repository alone.

What does an agent receive?

A repository at a fixed base commit and a broad instruction to find and fix as many bugs as possible. It receives no issue descriptions, filenames, line ranges, or other bug-specific hints.

You can find the full prompt here.

How is SWE-sweep evaluated?

We score every task against a reference set of previously identified bugs (see Construction, counting how many the agent successfully repairs.

For every task, we run the agent's submitted codebase against two sets of tests. First, we restore the repository's original test suite and run it to verify no existing behavior was broken. Second, for each bug, we run a set of hidden tests; at least one of these tests fails on the unmodified codebase, and passes once the bug is fixed (fail-to-pass). A bug is considered resolved if all its hidden tests pass and the original suite still passes.

The benchmark score is the fraction of all bugs across all repositories that have been resolved. What about any other changes that the agent makes?

What bugs are in the benchmark? How do you guarantee the task is feasible?

We filter bugs (represented by a test patch and a fix patch) to ensure the task is feasible. The criteria are:

1. The test patch and fix patch independently apply to the base commit.
2. All target tests pass after the fix patch has been applied to the base commit, but at least one target test fails on the base commit (F2P tests). There might be additional tests that pass before and after the fix patch has been applied (P2P tests).
3. The F2P tests reveal a single discoverable bug in the base commit.
4. The target tests are not overly specific; any reasonable fix to the discoverable bug will pass the target tests.
5. The target tests do not contradict the base commit tests.
6. Target tests of different bug instances do not contradict each other.

Appendix A.2 in the paper discusses feasibility in detail.

What makes a bug discoverable?

The expected behavior must be inferable from the repository itself, for example through documentation, types, existing tests, callers, invariants, standards, or an unambiguously undesirable failure such as a crash or data loss. The latter category is used extremely conservatively and all but 2 bugs in the benchmark have concrete repository contracts that describe the expected behavior. You can find some examples about what we mean with repository contracts here. We have spent a lot of time validating this aspect of the benchmark and you can find more details in the appendix of our paper.

What about any other changes that the agent makes?

Any history-derived benchmark necessarily under-counts the bugs present in a repository. By restricting PandoraBench to defects confirmed by an upstream fix, we ensure that every bug in the benchmark is backed by strong evidence that the observed behavior was considered erroneous by the repository maintainers. We therefore only score the agent's changes on the bugs that are confirmed by an upstream fix and supported by executable regression tests, as well as the other quality filters. However, if an agent causes a regression in the original test suite (the agent is explicitly told to avoid this), it will be scores as 0%. This means that the agent's changes that are not scored by the set of bugs are still likely to be non-destructive and compatible with the repository's existing behavior. See A.3 and A.4 in the paper for more discussion.

Does more inference-time compute help?

Repeated attempts recover additional bugs, but the gains diminish. Later work within one run can also undo earlier repairs, so simply extending a trajectory does not guarantee improvement. See Fig. 7 in the paper.

What about Astra, Fable, 5.5, ...?

We're working on evaluating more models! The current selection was finalized for our ICLR submission. We're also looking into even higher reasoning modes, but this might push over $10k for a single run. We also want to have more open weights models on the leaderboard.

Why mini-swe-agent? Could other scaffolds/multiagents achieve higher performance?

Our paper has an ablation with Claude Code and Codex. Neither seems to significantly outperform mini-swe-agent (to the contrary, mini-swe-agent is even quite a bit better than Codex). This follows many other benchmarks, where mini-swe-agent has been extremely competitive. However, we absolutely hope to kick off more research into the role of agent scaffolds and will open for submissions soon.

How do I submit to the leaderboard?

Public submissions are coming soon.