🔶 ReviewBench：面向 AI 代码审查的开放基准测试

Agentic 代码审查正成为软件开发流程中的关键环节。它可以帮助你检查 Pull Request、发现问题，并在代码发布前判断哪些内容值得关注。

但现有 AI 审查器的质量可能很难衡量；在判断某个审查器是否有帮助之前，你需要了解它的优势。一些审查器能发现更多问题，一些产生的噪声更少；还有一些更擅长捕获关键问题，另一些则也能发现较小的改进空间。你的工作流可能需要代码审查承担不同任务。

因此，了解审查器之间实际表现如何至关重要：不同系统能发现什么、会遗漏什么，以及它们做出了哪些权衡。优秀的代码审查基准应反映真实 Pull Request 的多样性，覆盖广泛的审查发现项，并支持按严重性、类别及精确率—召回率偏好进行有意义的细分。对于构建代码审查 Agent 的团队而言，该基准还应提供一种离线信号，能够可靠地反映某项改动是否可能改善生产环境中的体验。现有基准通常需要在标签质量、覆盖面以及真实代码审查代表性之间权衡，这使得行业仍缺少一种严谨、可复现且能将这些要素结合起来的评估方法。

为填补这一空白，我们构建了 ReviewBench——一项新的代码审查离线基准测试——现已可供使用。它遵循 GitHub 上超过 1 亿个真实 Pull Request 所呈现的编程语言、仓库规模和 Pull Request 大小分布。它采用多源黄金集和一致的评估准则，并已由高级工程师独立验证。同样重要的是，在 ReviewBench 的帮助下，我们对 Copilot code review（CCR）的离线评估在预测生产实验趋势方面变得更加有效，使我们更有信心确认：测得的改进代表了对用户有意义的提升。

本文将介绍 ReviewBench 的构建方式、它如何建立可靠的事实标准与评分机制，以及如何接入你自己的代码审查系统并提交结果。

本文使用的术语定义

• 基准测试（Benchmark）：使用相同评分方法，在一组共同的 Pull Request 上测试代码审查器的标准化评估。
• 发现项（Finding）：在代码审查期间发现的某个具体问题。
• 黄金集（Golden set）：针对每个 Pull Request、经过验证的已知发现项集合，用作评估审查器发现或遗漏内容的参考。
• 精确率（Precision）：审查器发现的问题中，有效问题所占的比例。精确率越高，通常意味着噪声越少。
• 召回率（Recall）：所有已知有效问题中，审查器发现的问题所占比例。召回率越高，意味着覆盖范围越广。
• F1 分数（F1 score）：对精确率和召回率给予同等权重的单一平衡分数。
• Fβ 分数（Fβ score）：F1 的一种变体，可根据审查偏好为精确率或召回率赋予更高权重。

1

我们构建了什么

面向 AI 代码审查 Agent 的真实、全面基准测试

103.9M

GitHub Pull Request

按编程语言、仓库规模和改动形态分析分布。

具有代表性的基准语料库

来自 19 种语言的 219 个公开 Pull Request，分布与整个 GitHub 保持一致，同时保留具有实质性审查价值的案例。

多源黄金集

• 人类审查者
• 前沿 LLM
• 静态分析

结构化发现项

每个发现项均按严重性和类别进行标注，以支持用户定制的细分视图。

类别

• 正确性
• 安全性
• 可靠性
• 可维护性
• 测试
• ……

评估指标

四项指标同时衡量已知问题和新发现的问题。

• Grounded 精确率
• Grounded 召回率
• Augmented 精确率
• Augmented 召回率

客观评估

客观衡量改进，并在不同 Agent 之间进行比较。帮助用户选择最符合其需求的审查器。

2

我们如何确保其可信

从评估准则到专家验证及生产环境检查的可审计链路

已发布的评估准则

所有发现项均遵循一项明确标准。

人工标注的开发集

高级工程师建立事实标准。

经校准的评审器

与人工判断保持一致。

统一标注

所有来源采用相同标准。

已发布的一致性结果

专家审计基准质量。

96.6% 一致率

高级工程师在发布前独立标注黄金集中的真阳性结果。

可预测生产表现的离线信号

通过线上实验检验基准变化。

• 改进通常会在线上体现
• 回归通常也会在线上体现

JavaScript 绘制连接这些卡片的箭头。ReviewBench 概览：1 我们构建了什么——面向 AI 代码审查 Agent 的真实、全面基准测试；1.039 亿个 GitHub Pull Request：按语言、仓库规模和改动形态分析分布；具有代表性的基准语料库：来自 19 种语言的 219 个公开 Pull Request，分布与整个 GitHub 保持一致，同时保留具有实质性审查价值的案例；多源黄金集：人类审查者、前沿 LLM、静态分析；结构化发现项：每个发现项均按严重性和类别标注，以支持用户定制的细分视图。严重性：严重、中等、低。类别：正确性、安全性、可靠性、可维护性、测试、……；评估指标：四项指标同时衡量已知问题和新发现的问题，包括 Grounded 精确率、Grounded 召回率、Augmented 精确率、Augmented 召回率；客观评估：客观衡量改进并在 Agent 间比较，帮助用户选择最符合其需求的审查器。2 我们如何确保其可信——从评估准则到专家验证及生产环境检查的可审计链路；已发布的评估准则：所有发现项均遵循一项明确标准；人工标注的开发集：高级工程师建立事实标准；经校准的评审器：与人工判断保持一致；统一标注：所有来源采用相同标准；已发布的一致性结果：专家审计基准质量；端到端可审计；96.6% 一致率：高级工程师在发布前独立标注黄金集真阳性结果；可预测生产表现的离线信号：通过线上实验检验基准变化，改进通常会在线上体现，回归通常也会在线上体现。连接关系：GitHub Pull Request 连接至具有代表性的基准语料库；具有代表性的基准语料库连接至多源黄金集；多源黄金集连接至结构化发现项；结构化发现项连接至评估指标；评估指标连接至客观评估；已发布的评估准则连接至人工标注的开发集；人工标注的开发集连接至经校准的评审器；经校准的评审器连接至统一标注；统一标注连接至已发布的一致性结果。

ReviewBench 如何运作

我们的基准测试围绕五项原则构建：

1. 具有代表性的 Pull Request，而非演示集

我们分析了 1.039 亿个 GitHub Pull Request，以刻画真实世界中代码审查工作负载的分布。ReviewBench 包含来自 187 个采用开放源代码许可证的公开仓库的 219 个 Pull Request，覆盖 19 种语言，其语言和仓库规模分布与 GitHub 整体高度一致。完整基准数据集现已公开提供。

我们对该分布进行了一项有意调整：虽然语言和仓库规模直接映射 GitHub 的分布，但 Pull Request 的规模会向更适合审查的中等规模和长尾部分倾斜。这减少了微小单文件改动的过度代表性，同时保留了更多审查质量最重要的、具有实质内容的多文件 Pull Request。

语料库快速概览：

2. 广泛发现事实标准，并进行独立裁决

无论是人类还是模型，单个审查者都无法识别 Pull Request 中所有值得发现的问题。为了为事实标准发现项构建更广泛、更可靠的黄金集，我们采用三阶段流程：

• **从多样化来源收集候选发现项。**我们收集真实人类审查者的发现项、从作者后续提交中推断的问题、确定性分析工具的结果，以及跨多个模型家族的多个前沿 LLM 的发现项。
• **在语义层面对重叠发现项去重。**我们合并识别出同一底层问题的发现项，在扩大覆盖范围的同时，避免不同产出者之间的共识人为抬高黄金集，也避免黄金集依赖于某个来源的盲区。
• **依据共享评估准则验证发现项。**发现项的来源并不决定它是否正确：只有当发现项真实、相关且并非微不足道时，才会被计为真阳性。我们使用 Claude Sonnet 5 作为 LLM 评审器，并对所有提交内容应用一致的评估准则。为了确保透明性和可复现性，我们同时公开评估准则以及用于执行该准则的评审器。

3. 同时衡量已知问题和新发现问题的指标

大多数基准测试会针对固定黄金集报告精确率和召回率。ReviewBench 报告两类共六项指标：

• Grounded 精确率、召回率和 F1 分数仅使用现有黄金集标签。它们提供严格、可公平比较的结果：在我们已经知道的问题中，Agent 发现了多少；以及它发现的项目中，有多少与已知问题匹配？
• Augmented 精确率、召回率和 F1 分数还会评估未与黄金集中任何内容匹配的发现项。评审器会独立判定这些不匹配的发现项属于真阳性还是假阳性，使审查器能够因发现黄金集中没有任何产出者提出的有效问题而获得认可。

随着审查 Agent 能力不断增强，这一区别变得愈发重要。当系统发现基准创建者未曾预见的问题时，固定黄金集不可避免地会变得不完整。Augmented 指标让 ReviewBench 能够识别这种能力，而不是自动对其施加惩罚。由于 Augmented 召回率会根据每个 Agent 发现的问题扩大分母，我们使用 Grounded 召回率作为跨系统比较的核心指标，并将 Augmented 指标作为针对单个系统的补充诊断手段。

4. 面向不同审查偏好的可配置评估

不存在一种普遍最优的审查体验。一些开发者可能只希望关注关键问题，另一些人也重视严重性较低、不会破坏功能的发现项。一些人偏好更广泛的覆盖，而另一些人优先考虑精确率和最小化噪声。还有人可能存在专门需求，例如侧重安全或隐私的审查。

ReviewBench 可按严重性和类别对结果进行切分，而精确率和召回率则反映不同的运行偏好。用户还可以调整 Fβ 分数中的 β 值，以便为更广覆盖赋予召回率更高权重，或为降低噪声赋予精确率更高权重。随着这些偏好变化，排行榜也会相应重新排序，帮助用户识别最符合自身审查优先级的系统。

5. 经过内部审计且可复现地评估

在发布前，我们邀请未参与基准数据集构建的高级工程师，从头开始独立重新标注每一项事实标准发现项。他们对真阳性或假阳性的判断与 ReviewBench 的一致率为 96.6%。我们会对每次评估所用的基准数据集、评审器和匹配器进行版本管理，因此可以在同一基准配置下比较结果，并在基准发生变化时重新验证结果。我们还公开验证方法、一致性测量结果及已知的有效性威胁，使读者能够了解基准质量如何被评估，以及不确定性仍存在哪些地方。

探索 ReviewBench

ReviewBench 的研究预览版现已通过 ReviewBench 网站 提供。你可以探索完整基准、比较代码审查 Agent，并引入自己的 Agent 进行评估和迭代。

借助 ReviewBench，你可以：

• **探索完整基准数据集。**完整的 ReviewBench 数据集现已公开提供，包括 Pull Request、发现项、标签、严重性和类别标注。这使你可以准确检查系统评估的对象，并复现基准测试结果。
• **在排行榜上比较系统。**使用完整基准数据评估的代码审查 Agent 结果会发布在统一排行榜中，并提供整体表现、严重性、类别及不同精确率—召回率偏好的视图。
• **引入你自己的 Agent 并持续优化。**完整基准数据集、评估方法、LLM 评审提示词、评审模型配置和自助运行器均已公开，因此你可以评估自己的代码审查 Agent，检查其优势和不足，并在相同基准配置下持续迭代。

我们如何使用 ReviewBench

我们已使用 ReviewBench 对 Copilot code review（CCR） 的连续迭代版本进行评估，这为我们提供了一种一致的方法来衡量进展、发现回归并确定有前景的改动优先级。随着时间推移，这帮助我们改进了产品。ReviewBench 最有价值的收益之一，是它能够为产品变更在生产环境中的潜在表现提供早期离线信号。在进入 A/B 测试前使用 ReviewBench 评估的各项实验中，离线变化始终与之后在生产环境中观察到的结果指向同一方向。

最近的一项 lite-tier 实验为这一更广泛的规律提供了具体示例。我们引入了一种多模型集成审查机制：它将多个独立模型运行结果合并为一次审查，而非依赖单次运行。ReviewBench 预测，该机制将带来更高的精确率、召回率和评论量，同时降低每次审查的成本。

为了比较离线结果与生产环境结果，

Agentic code review is becoming an essential piece of how development happens. It helps you inspect pull requests, catch issues, and decide what deserves attention before code ships.

But the quality of existing AI reviewers can be hard to measure, and you need to know the strengths of a reviewer before you know if it will help you. Some reviewers surface more issues, some produce less noise, and some are stronger at catching critical problems while others surface smaller improvements, too. You may need code review to do different things within your workflow.

That makes it important to understand how reviewers actually compare: what different systems catch, what they miss, and the tradeoffs they make. A good code review benchmark should reflect the diversity of real pull requests, capture a broad set of review findings, and support meaningful breakdowns by severity, category, and precision-recall preferences. For teams building code review agents, the benchmark should also provide an offline signal that reliably tracks whether changes are likely to improve the experience in production. Existing benchmarks often make tradeoffs between label quality, coverage, and how well they represent real-world code review, leaving a gap for a rigorous and reproducible evaluation methodology that brings these pieces together.

We built ReviewBench, a new code review offline benchmark , to address that gap, and it is available for you to use today. It follows the language, repo size, and size distribution of pull requests, modeled after over 100 million real pull requests on GitHub. It uses a multi-source golden set and a consistent evaluation rubric and has been independently validated by senior engineers. Just as important, with the help of ReviewBench, our offline evaluation of Copilot code review (CCR) has become more effective at anticipating the direction of production experiments, giving us greater confidence that measured improvements reflect meaningful gains for users.

In this post, we’ll walk through how ReviewBench is constructed, how it establishes reliable ground truth and scoring, and how to onboard your own code review system and submit results.

Definitions of terms used in this blog post

• Benchmark : A standardized evaluation that tests code reviewers on a common set of pull requests using the same scoring methodology.
• Finding : A specific issue surfaced during code review.
• Golden set : A validated collection of known findings for each pull request, used as a reference for evaluating what a reviewer catches or misses.
• Precision : Of the issues a reviewer surfaces, the proportion that are valid. Higher precision generally means less noise.
• Recall : Of the known valid issues, the proportion the reviewer finds. Higher recall means broader coverage.
• F1 score : A single score that balances precision and recall equally.
• Fβ score : A variation of F1 that lets you put more weight on either precision or recall, depending on your review preference.

1

What we built

A realistic, comprehensive benchmark for AI code review agents

103.9M

GitHub pull requests

Analyze distributions by language, repository size, and change shape.

Representative benchmark corpus

219 public pull requests across 19 languages, aligned to GitHub-wide distributions while preserving substantive review cases.

Multi-source golden set

• Human reviewers
• Frontier LLMs
• Static analysis

Structured findings

Every finding is labeled for severity and category, enabling user-tailored slices.

Category

• Correctness
• Security
• Reliability
• Maintainability
• Testing
• ......

Evaluation metrics

Four metrics measure both known and newly discovered issues.

• Grounded precision
• Grounded recall
• Augmented precision
• Augmented recall

Objective evaluation

Measure improvement and compare across agents objectively. Help users choose the reviewer that fits their needs the best.

2

How we keep it trustworthy

An auditable chain from rubric to expert validation and production checks

Published rubric

One explicit standard for all findings.

Human-labeled dev set

Senior engineers establish ground truth.

Calibrated grader

Aligned with human judgment.

Uniform labeling

Same standard across all sources.

Published agreement

Expert audit of benchmark quality.

96.6% agreement

Senior engineers independently labeled golden true-positives before release.

Offline signals that anticipate production

Benchmark movement is checked against online experiments.

• Improvements tend to show up online
• Regressions tend to show up online too

JavaScript draws the arrows connecting these cards. ReviewBench at a glance 1 What we built A realistic, comprehensive benchmark for AI code review agents 103.9M GitHub pull requests Analyze distributions by language, repository size, and change shape. Representative benchmark corpus 219 public pull requests across 19 languages, aligned to GitHub-wide distributions while preserving substantive review cases. Multi-source golden set Human reviewers; Frontier LLMs; Static analysis Structured findings Every finding is labeled for severity and category, enabling user-tailored slices. Severity: Critical Medium Low Category: Correctness Security Reliability Maintainability Testing ...... Evaluation metrics Four metrics measure both known and newly discovered issues. Grounded precision; Grounded recall; Augmented precision; Augmented recall Objective evaluation Measure improvement and compare across agents objectively. Help users choose the reviewer that fits their needs the best. 2 How we keep it trustworthy An auditable chain from rubric to expert validation and production checks Published rubric One explicit standard for all findings. Human-labeled dev set Senior engineers establish ground truth. Calibrated grader Aligned with human judgment. Uniform labeling Same standard across all sources. Published agreement Expert audit of benchmark quality. Auditable end to end 96.6% agreement Senior engineers independently labeled golden true-positives before release. Offline signals that anticipate production Benchmark movement is checked against online experiments. Improvements tend to show up online; Regressions tend to show up online too Connection: GitHub pull requests to Representative benchmark corpus. Connection: Representative benchmark corpus to Multi-source golden set. Connection: Multi-source golden set to Structured findings. Connection: Structured findings to Evaluation metrics. Connection: Evaluation metrics to Objective evaluation. Connection: Published rubric to Human-labeled dev set. Connection: Human-labeled dev set to Calibrated grader. Connection: Calibrated grader to Uniform labeling. Connection: Uniform labeling to Published agreement.

How ReviewBench works

Our benchmark is built around five principles:

1. Representative pull requests, not a demo set

We analyzed 103.9 million GitHub pull requests to characterize the real-world distribution of code review workloads. ReviewBench contains 219 pull requests from 187 public open source licensed repositories spanning 19 languages, with its language and repository-size distributions closely matching GitHub overall. The complete benchmark dataset is publicly available.

We make one deliberate adjustment to this distribution: while language and repository size mirror GitHub directly, pull request size is weighted toward the reviewable middle and tail. This reduces the overrepresentation of tiny, single-file changes while preserving more substantive, multi-file pull requests where review quality matters most.

Quick corpus snapshot:

2. Broad ground truth discovery, independently judged

No single reviewer, whether human or model, can identify everything worth finding in a pull request. To build a broader and more reliable golden set for ground truth findings, we follow a three-stage process:

• Gather candidate findings from diverse sources. We collect findings from real human reviewers, issues inferred from author follow-up commits, deterministic analysis tools, and multiple frontier LLMs across model families.
• Semantically deduplicate overlapping findings. We merge findings that identify the same underlying issue, broadening coverage without allowing agreement across producers to artificially inflate the golden set or making it dependent on any one source’s blind spots.
• Validate findings under a shared rubric. The source of a finding does not determine whether it is correct: a finding counts as a true positive only if it is true, relevant, and non-trivial. We use Claude Sonnet 5 as the LLM grader, applying a consistent evaluation rubric across all submissions. For transparency and reproducibility, we publish both the evaluation rubric and the judge used to apply it.

3. Metrics that measure both known and newly discovered issues

Most benchmarks report precision and recall against a fixed golden set. ReviewBench reports six metrics in two families:

• Grounded precision, recall, and F1 score use only the existing gold-set labels. They provide the strict, apples-to-apples comparison: of the issues we already know about, how many did the agent find, and what share of its findings matched a known issue?
• Augmented precision, recall and F1 score also evaluate findings that do not match anything in the golden set. The judge independently determines whether those unmatched findings are true or false positives, allowing a reviewer to receive credit for valid issues that no producer in the golden set surfaced

That distinction becomes more important as review agents become more capable. A fixed golden set inevitably becomes incomplete as systems discover issues its creators did not anticipate. Augmented metrics let ReviewBench recognize that behavior rather than automatically penalizing it. Because augmented recall expands the denominator based on what each agent discovers, we use grounded recall as the headline cross-system comparison and augmented metrics as an additional per-system diagnostic.

4. Configurable evaluation for different review preferences

There is no single universally optimal review experience. Some developers may want to focus only on critical issues, while others also value lower-severity, non-breaking findings. Some prefer broader coverage, while others prioritize precision and minimal noise. Others may have specialized needs, such as security- or privacy-focused review.

ReviewBench lets results be sliced by severity and category, while precision and recall capture different operating preferences. Users can also adjust β in the Fβ score to place more weight on recall for broader coverage or precision for lower noise. As these preferences change, the leaderboard is re-ranked accordingly, helping users identify the systems that best match their review priorities.

5. Internally audited and reproducibly evaluated

Before release, we asked senior engineers who had not participated in building the benchmark dataset to independently re-label every ground-truth finding from scratch. Their true/false-positive judgments agreed with ReviewBench 96.6% of the time. We version the benchmark dataset, judge, and matcher used in every evaluation, so results can be compared under the same benchmark configuration and revalidated when the benchmark changes. We also publish the validation methodology, agreement measurements, and known threats to validity, so readers can see how benchmark quality is assessed and where uncertainty remains.

Explore ReviewBench

ReviewBench’s research preview version is now available through the ReviewBench website, where you can explore the full benchmark, compare code review agents, and bring your own agent to evaluate and iterate.

With ReviewBench, you can:

• Explore the full benchmark dataset. The complete ReviewBench dataset is publicly available, including the pull requests, findings, labels, severity and category annotations. This allows you to inspect exactly what systems are evaluated on and reproduce benchmark results.
• Compare systems on the leaderboard. Results from evaluated code review agents using the full benchmark data are published on a common leaderboard, with views across overall performance, severity, category, and different precision–recall preferences.
• Bring your own agent and hill-climb. The full benchmark dataset, evaluation methodology, LLM judge prompt, judge model configuration, and self-serve runner are publicly available, so you can evaluate your own code review agent, inspect its strengths and gaps, and iterate against the same benchmark configuration.

How we’ve used ReviewBench

We have used ReviewBench to evaluate Copilot code review (CCR) across successive iterations, giving us a consistent way to measure progress, catch regressions, and prioritize promising changes. Over time, this has helped us improve the product. One of the most valuable benefits of ReviewBench is that it provides an early offline signal of how a change to the product is likely to perform in production. Across experiments evaluated with ReviewBench before A/B testing, offline changes have consistently pointed in the same direction as what we see later in production.

A recent lite-tier experiment provides a concrete example of this broader pattern. We introduced a multi-model ensemble review that combines several independent model runs into a single review rather than relying on a single run. ReviewBench predicted higher precision, recall, and comment volume, along with lower cost per review.

To compare offline and production results, we use corresponding online signals. Addressed rate, our online counterpart to precision, is the percentage of CCR comments that an LLM determines prompted a developer to make a corresponding code change, based on the diff, thread, reactions, resolution state, and post-review code. For recall, we measure how much additional human review is still needed.

The online A/B test moved in the same direction as ReviewBench predicted: addressed rate (precision) rose 8.0%, recall rose 13.6%, and comment volume rose 61%, while cost per review fell 8.0%, all relative to the production control.

Comment volume alone, however, does not capture comment quality. More critical findings mean something very different from low-severity nits. ReviewBench’s severity-level evaluation captured this too: it predicted a 227% increase in critical comments, compared with 262% online, along with the same broader shift toward more moderate comments and fewer nits.

This gives us a fast and repeatable signal before running production experiments. Online experiments remain the ultimate measure of user impact, but ReviewBench gives us greater confidence in which changes are worth taking there.

How to submit your own run

1. Sign in with GitHub on the ReviewBench website.
2. Register your agent. Provide a container image, your configuration, and your own model key. We provide the judge.
3. Try it on the test set. Run against a 25-PR test set with per-PR detail and repeat as you tune your configuration.
4. Do a final run. When you’re ready, run the full set of 219 pull requests (three rounds), scored by the same judge as every other entry.
5. Publish to the leaderboard. Your scores remain private until a maintainer reviews and approves the submission. Scores are published to the leaderboard only if they outperform the agent’s current leaderboard score, or if this is the agent’s first leaderboard entry.

We invite you to explore ReviewBench, evaluate your own system, challenge our assumptions, and help us improve the benchmark. We’re excited to collaborate with researchers and practitioners to make code review evaluation more open, reliable, and useful—and ultimately help move AI code review forward.

Acknowledgments

ReviewBench was a team effort across GitHub and Microsoft. We’re grateful to the researchers and engineers who built it: those who designed the methodology, curated the pull requests, built the golden set and the evaluation pipeline, and made the benchmark something anyone can run.

Written by

Michelle develops and evaluates agentic AI systems for code review, focusing on repository-level context retrieval, review and fix quality, and rigorous benchmarking of AI code reviewers.

Alejandro's work focuses on evaluating coding agents, improving how they reason over code changes and repository context, and optimizing model quality, coverage, latency, and token cost.