🔶 DoGBench：首个面向用户的文档生成基准测试，没有模型得分超过 50%
英文标题：🔶 DoGBench: The first user-facing docs generation benchmark. No model scores >50%
DoGBench 是一个面向用户文档生成的评测基准，用于检验 AI Agent 能否判断文档是否需要更新，并生成能通过专家审核的修改。该基准包含来自真实开源项目的 292 项任务，其中 205 项需要修改文档、87 项应保持文档不变；在 117 项保留测试集上，最高综合得分仅为 47.3/100。对 1,267 份提交的审计显示，45.5% 存在导致用户无法完成任务的信息缺口，36.6% 存在技术错误。研究表明，AI 生成的文档即使语言流畅，仍需要开发者和文档维护者依据真实用户工作流进行严格审查。
145.4 KB
129.2 KB
中文
AI Agent 可以写出语言精致的文档，但它们往往遗漏读者完成任务所需的信息。DoGBench 评估 Agent 能否识别文档何时需要更新，以及能否生成符合专家评审标准的修改。
该基准包含 292 项任务，来源于真实的开源项目，包括 Helm、PostHog、Mautic 和 Doc Detective。其中，205 项任务需要修改文档，87 项任务要求文档保持不变。任务涵盖创建新文档、更新现有指引，以及删除过时或冗余内容。**我们的主要评估采用随机分层抽取的 117 项保留测试集：其中 82 项需要更新，35 项无需修改。**为保证可复现性，我们发布了来自 7 个非云端 Agent 的 735 条完整 Agent 轨迹。
Agent 难以判断文档何时需要更新
文档工作始于一项判断：这项变更是否会影响用户？用户是否需要新的指引？
Agent 会在两个方向上犯错。有些 Agent 会为无需新增用户指引的内部重构或性能优化编写文档。另一些 Agent 则会因为找不到关于某项功能的现有页面，而忽略必要的更新——即使缺失的覆盖内容恰恰是它们需要解决的问题。
在论文报告的 7 个模型与运行框架组合中，117 项保留测试集上的最高综合得分为 47.3/100，由搭配 OpenCode 的 Qwen3.8 Max 取得。排行榜还包括 3 个在同一测试集上评估的云端 Agent 系统。
文笔流畅，仍可能让读者无法完成任务
一个补丁可以表述清晰、事实正确，却遗漏前置条件、决策点或关键步骤。它也可能被放在读者不太可能找到的页面上，或与其他地方的指引相矛盾。
在对 1,267 份提交进行的一项独立且更广泛的审计中：
- 45.5% 存在任务完成缺口，例如缺少前置条件、操作步骤、验证步骤或恢复路径。
- 36.6% 包含技术性错误。
- 32.5% 遗漏了核心概念或参考信息的一部分。
这些类别彼此存在重叠。6.1% 的提交中出现了捏造内容，例如虚构的类、标志或端点。对于前沿 Agent 而言，幻觉也可能以更隐蔽的形式出现：扩大真实功能的适用范围、错误陈述默认值，或将仅在特定条件下成立的行为描述为普遍适用。我们的审计将这些失真归类为技术性错误。
轨迹分析还发现，Agent 经常错过决定性证据，或者在找到第一个看似可以编辑的页面后就停止调查。
专家评审仍然必不可少
当一个补丁在评分量表下不存在任何严重失败时，它被视为 P0-clean。
- **论文中 7 个组合的最高 P0-clean 交付率为 39.0%。**搭配 Codex 的 GPT-5.6 Sol 在 82 项需要更新的保留测试任务中，有 32 项交付了 P0-clean 补丁。
文档维护者拥有代码变更本身未必能揭示的上下文：读者是谁、他们想完成什么，以及变更将如何影响其工作流。这些上下文有助于确定是否需要更新文档，以及文档必须涵盖哪些内容。
应根据该工作流审查 Agent 的草稿。检查读者能否找到相关指引、完成操作步骤并验证结果。确认每项技术声明所依赖的条件，并检查相关页面是否存在矛盾或过时说明。
我们如何评估 Agent
每个 Agent 都会获得一个处于变更前状态的代码仓库，以及一个触发信号，例如代码 Pull Request 或报告的文档缺口。已合并的文档内容不会提供给 Agent。
我们在同一任务集上评估了 7 个 Agent 组合。针对任务的评分量表评估准确性、完整性、读者指引、内容放置位置、写作风格和仓库约定。在 205 项需要文档更新的任务中，当前评分量表共包含 3,273 项标准，其中包括 798 项 P0 标准。
DoGBench 将更新决策与补丁质量分开报告。评分规则同时衡量两者：
- 已交付补丁质量：对 82 项需要更新的保留测试任务的质量取平均值。遗漏补丁或空补丁计为零分。
- 正确弃权：衡量 Agent 在 35 项无需更新的保留测试任务中，正确保持文档不变的频率。
- 综合得分采用已交付补丁质量和弃权召回率的调和平均数：
2DN / (D + N)。某一项表现优异，无法完全弥补另一项表现薄弱。 - **严重失败会将补丁得分上限限制为 60/100。**每项评分标准权重相同，但未满足任一 P0 标准都会触发该上限。次要标准上的优势无法抵消严重缺陷。
English / 原文
AI agents can write polished documentation, but they often miss what readers need to complete a task. DoGBench evaluates whether agents can recognize when documentation needs updating and produce changes that meet expert review standards.
The benchmark contains 292 tasks drawn from real open-source projects, including Helm, PostHog, Mautic, and Doc Detective. Of these, 205 require a documentation change; 87 require leaving the documentation unchanged. Tasks include creating new documentation, updating existing guidance, and deleting stale or redundant content. Our primary evaluation uses a random, stratified 117-item held-out split: 82 requiring updates and 35 requiring no change. For reproducibility, we are publishing 735 complete agent trajectories from seven non-cloud agents.
Agents struggle to decide when documentation needs updating
Documentation work begins with a judgment: does this change affect users, and do they need new guidance?
Agents make mistakes in both directions. Some document internal refactors or performance optimizations that require no new user guidance. Others overlook necessary updates because they find no existing page about the feature, even when that missing coverage is the problem they need to solve.
Among the seven model-and-harness lanes reported in the paper, the highest combined score on the 117-item held-out split was 47.3 out of 100 , achieved by Qwen3.8 Max with OpenCode. The leaderboard also includes three cloud-agent systems evaluated on that same split.
Polished writing can still leave readers unable to finish a task
A patch can sound clear and contain correct facts while omitting a prerequisite, decision point, or essential step. It can also appear on a page readers are unlikely to find or contradict guidance elsewhere.
In a separate, broader audit of 1,267 submissions:
- 45.5% had a task-completion gap , such as a missing prerequisite, procedural step, verification step, or recovery path.
- 36.6% contained technical inaccuracies.
- 32.5% omitted part of the central concept or reference information.
These categories overlap. Fabricated content, such as invented classes, flags, or endpoints, appeared in 6.1% of submissions. With frontier agents, hallucinations can also take subtler forms: distorting real functionality by extending its scope, misstating defaults, or presenting behavior that holds only under specific conditions as universally true. Our audit classified these distortions as technical inaccuracies.
The trajectory analysis also found that agents frequently missed decisive evidence or stopped investigating after finding the first plausible page to edit.
Expert review remains necessary
A patch is P0-clean when it has no critical failures under the rubric.
- Highest P0-clean delivery rate among the paper's seven lanes: 39.0%. GPT-5.6 Sol with Codex delivered a P0-clean patch for 32 of the 82 held-out tasks requiring an update.
Documentation owners bring context that a code change alone may not reveal: who the reader is, what they are trying to accomplish, and how the change affects their workflow. That context helps determine whether an update is needed and what it must cover.
Review an agent’s draft against that workflow. Check that the reader can find the guidance, complete the steps, and verify the result. Confirm the conditions behind each technical claim and check related pages for contradictions or stale instructions.
How we evaluated agents
Each agent received a repository as it existed before the change and a triggering signal, such as a code pull request or a reported documentation gap. The merged documentation was withheld.
We evaluated seven agent lanes on the same task set. Task-specific rubrics assessed accuracy, completeness, reader guidance, placement, style, and repository conventions. Across the 205 documentation-needed items, the current rubrics contain 3,273 criteria, including 798 P0 criteria.
DoGBench reports update decisions and patch quality separately. The scoring rules capture both:
- Delivered patch quality averages quality across all 82 held-out tasks requiring an update. Missed or empty patches receive zero.
- Correct abstention measures how often the agent leaves documentation unchanged across the 35 held-out tasks requiring no update.
- The combined score uses the harmonic mean of delivered patch quality and abstention recall: 2DN / (D + N). Strong performance on one cannot fully compensate for weak performance on the other.
- A critical failure caps a patch’s score at 60 out of 100. Each rubric criterion has equal weight, but failing any P0 criterion triggers the cap. Strengths on secondary criteria cannot average away a critical defect.