验证闭环能为编码智能体带来什么：初次观察

这是持续系列文章的开篇。我们从一个项目上的一对模型开始，分析将在更多模型和更多数据集上继续进行。我们有意提前分享，并会在过程中持续更新。

引言

编码智能体可以快速生成大量代码。但它自己无法知道这些代码是否真的能用。模型写出了看似合理的内容，运行继续进行，任何错误都会随之传递下去。在实际的多步骤项目中，这种情况会叠加：早期一个未经验证的错误可能摧毁所有构建在其上的东西。人类团队使用的修复方法古老而乏味：在有人继续构建之前，会对照正在运行的应用检查工作成果。

这篇文章只问一个问题：增加这样的检查步骤，能在多大程度上改变编码智能体所能完成的任务？我们最初的假设是，对于一大类失败模式，封闭的验证闭环（运行应用、演练变更、发现失败、驱动修复、然后证明修复有效）比原始模型能力更重要。代码在被验证之前不算完成。

为了测试这一点，我们需要一个具体的验证层，我们使用了自研的工具。 IronBee 是面向 AI 编码智能体的验证和修复层：当智能体编写代码时，IronBee 会在浏览器中打开应用，演练变更，并查找问题。发现问题后，它会分析失败原因，找到根因，驱动修复，然后重新验证，因此修复只有在被证明有效后才算数。我们构建了 IronBee，所以请将此视为一份方法公开、利益相关的实验。在本文中，IronBee 是测量仪器，而非被观察对象：我们关心的结论是关于验证闭环的普遍意义，且实验设置已公开，任何人都可以复现或质疑。

我们花了一周时间构建了一个严谨的测试，现在分享初步结果，后续还会有更多内容。我们选取了真实的 Web 开发工作，并运行了两个模型，一个是低成本的开源模型，另一个是商业前沿模型。做好实验设置比听起来要困难，下一节会解释原因。我们视此为一个开端，而非定论。本系列文章的重点是观察当我们加入更多模型和更多工作类型时，局面会发生怎样的变化。

方法论

选择合适的数据集。 我们需要看起来像真实工作，而非玩具问题。我们选了 Web-Bench，字节跳动发布的一个公开基准。它是公开的，任何人都可以检查我们的设置：代码在 GitHub 上，数据集在 Hugging Face 上，设计思路在 Web-Bench 论文中有描述。Web-Bench 包含 50 个项目。每个项目是一个单页 Web 应用，由 20 个必须按顺序完成的任务组装而成，因此每个任务都建立在前一个任务之上。这种顺序结构正是我们没有选择独立一次性任务基准的原因：当工作建立在早期工作之上时，验证闭环的价值最为明显，因为一个未经验证的错误会复合影响，而不仅限于局部。每个任务都有一个隐藏的测试来检查结果，且智能体永远看不到它。如果某个任务失败，运行就停在那里。这保证了基准的诚实性：智能体无法跳过失败步骤或隐藏一个损坏的环节。

在首轮工作中，我们使用其中一个项目——调查问卷应用，它构建一个包含问题、必填字段和预览功能的表单。它涵盖了我们关心的难度范围，从一个简单的表单到必须在设计页面与预览页面之间往返传递才能存活的逻辑。这也非常适合我们想要测量的东西。验证闭环在代码编写过程中发挥价值，恰恰针对这类失败：单独看起来正确的代码，只有在运行应用进行演练时才会出问题。因此，我们在这里看到的提升反映了验证在真实开发中实际带来的增益，而非源于简单或刻意构造的任务。增加更多项目是优先事项之一。

选择模型。 我们从两个模型开始。DeepSeek (deepseek-v4-pro) 是一个能力强但价格低的模型，因此代表了我们想要提升的低成本编码器。Opus (claude-opus-4-8) 是一个前沿模型，它设定了质量标杆。将它们并排对比，让我们可以问一个清晰的问题：验证闭环能否将低成本模型提升到前沿模型的水平？我们构建了测试框架，可以沿着同一条路径运行多个模型，这一对模型是我们开始的起点。关于设置有一点说明：在首次运行中，DeepSeek 仅支持文本，因此它看不到屏幕。这没关系，因为检查程序通过无障碍树（a11y tree）、DOM 和控制台以结构化文本形式读取页面，而非将其作为图像。模型基于结构化反馈得出结论，而不是通过查看像素。

验证器能及不能看到的内容。 有两条规则保持了对比的公平性，值得明确说明，因为它们可能是持怀疑态度的读者首先会问的。第一，隐藏的 Web-Bench 测试对所有人保持隐藏：编码模型永远看不到它们，验证层也看不到。测试仅存在于 Web-Bench 的评估器中，在智能体返回文件后运行；IronBee 的检查依据来自任务描述和运行中的应用、无障碍树、DOM 和控制台，而非基准的测试文件。第二，闭环并非零成本，而且我们纳入了统计：验证层会发起自己的模型调用，而这些调用都包含在本文中 DeepSeek + IronBee 方案的所有成本数据中。还有一点值得说明：验证层运行在与编码器相同的模型提供商上，而非更强模型。其分析和修复步骤使用 DeepSeek 更小、更便宜的模型 deepseek-v4-flash，而代码本身由 deepseek-v4-pro 编写。因此，能力提升并非来源于偷偷通过验证器引入更强的模型。

设置重试次数。 智能体接到任务，编写代码，由隐藏测试检查它。我们给每个任务少量的重试预算，而重试级别的影响比表面看起来要大。有些任务几乎不可能一次就通过，这其实是刻意设计的：隐藏测试检查一个精确的机制，而任务描述对该机制的表述很松散，甚至具有误导性。

任务 9 要求用一个复选框来设置星星数量，但测试需要一个能容纳 5 或 10 这样数字的控件，而复选框做不到。任务 15 要求将问题设为必答，但测试只接受浏览器内置的必填标记，而不是仅表面看起来正确的手写检查。一个逐字遵循描述的模型会选错机制而失败，并且在看到错误之前它无从知晓。由于它不总是做相同选择，同一个任务可能一次运行通过，另一次却失败，而第二次尝试，获得了之前的错误信息，通常能修复问题。这正是关键所在：这个基准测试衡量的是模型能否读懂失败并修复，而不仅仅是闭眼猜测意图。因此，一次尝试主要衡量的是盲目猜测。三次尝试后，大多数智能体最终都能达到，因此很难看到额外增益。两次尝试能让真正的差异显现出来，所以我们就以两次尝试来运行。

闭环仅仅是多了几次尝试吗？ 有一个质疑值得事先回应。验证闭环在每个任务上花费了额外的推理开销，那么提升是否仅仅是变相地增大了计算预算？一定程度上确实如此。闭环做了更多工作。但基准提供的重试是盲目的：模型看到失败信号后再次猜测。闭环的迭代由来自运行应用的证据驱动，这是另一种性质的尝试，而不仅仅是又一次。一定成本下，有证据引导的迭代是否能击败同等预算的盲目重试，正是这个框架所要求的消融实验，并计划在后续文章中探讨：在花同样的钱的前提下，给 DeepSeek 单独分配更多重试预算，与给 DeepSeek 加上闭环进行对比。在那项实验完成之前，请带着这个开放性问题阅读下面的结果。

诚实地衡量成本。 成本问题原来很微妙。内置的成本数对于第三方模型来说并不靠谱，因为它将所有模型都按前沿模型的价格计价。差距很大：用前沿模型的价格给 DeepSeek 计价，会高估其成本约二十倍。因此，对于低成本模型，我们不信任内置数字。我们根据模型提供商自己公布的 token 价格计算了成本，本文中的每个成本数字都是这样得出的。

表 1. Opus 和 DeepSeek 模型价格目录。

相信数字。 相同的模型在两次运行之间可能完成工作量差异很大。单次运行能说明的问题很少。因此，我们每种设置运行五次并读取平均值。我们还固定了 IronBee 的一个发布版本，因此每次运行都使用相同的工具，并且我们保持并行运行完全独立，互不干扰。

按难度评分。 Web-Bench 将每个任务标记为简单、中等或具有挑战性，且任务按顺序进行。因此，我们把一次运行能达到的进度转换成一个从 0 到 100 的单一分数，并按难度加权。基础加上全部 20 个调查任务总计恰好为 100。这个权重是我们自己辅助阅读的方式；Web-Bench 原生的度量标准是简单的通过率，不按难度加权。

表 2. 难度加权分数。基础加上 20 个调查任务总计恰好为 100。

结果

在展示数字之前，先说明我们如何解读它们。本文的问题——闭环验证能在多大程度上改变智能体所能完成的工作——可以拆分为两个比较。首先，我们将模型与其自身比较，分别在有和没有闭环的情况下。然后，我们进行跨模型比较：将处于闭环内的低成本模型与没有闭环的前沿模型在分数和价格两方面进行比较。我们按此顺序讨论。

分析一：各测试组最远能走多远

这里比较 DeepSeek 独自运行时，与 DeepSeek 加上 IronBee 时的表现。

表 3. DeepSeek 单独运行。分数满分 100，按难度加权。

表 4. 相同模型，由 IronBee 检查并修复工作。

DeepSeek 单独运行时能完成简单任务，然后在第一个中等任务上停滞，五次运行中有四次如此；只有一次运行到达了任务 12。平均完成 20 个任务中的 6.8 个，平均得分 20.4，中位数 11。有了闭环，同一个模型越过了中等难度区域，并深入高难度区域：平均完成 17 个任务，得分 80.6，中位数 86，并且有一轮运行完成了整个项目。闭环中最差的运行得分（62）也超过了没有闭环时最好的运行得分（48）。实际上，使用相同的模型和提示词，IronBee 提供了相对于模型独自运行时 4 倍的问题解决智能水平。

难度标签让这一点很容易想象。基线在简单和中等任务之间的边界撞墙停了下来。闭环将同一个模型带过了那堵墙，进入高难度任务区域。关键点在于什么没有改变。我们没有换用更大的模型，也没有编写更好的提示词。模型是相同的。改变的是，智能体现在对自己的工作有了第二次、基于证据的审视机会。检查程序在真实浏览器中演练变更，报告哪里出了问题，智能体在继续之前进行修复。

这种机制值得详细阐述，因为它解释了为何提升如此巨大。在这个项目中，绝大多数难以发现的漏洞都隐藏在从设计页面到预览页面的往返过程中：每个页面的代码单独看可能都正确，而它们之间的交互却是断裂的。这正是那种首次编写容易遗漏、而对运行应用进行检查则往往能捕捉到的失败类型。基线遇到的墙并非编码能力不足。而是缺乏关于代码是否有效的任何真实反馈。

图 1. 各测试组在 20 个顺序任务上的完成进度。单元格按任务难度着色；实心单元格表示该测试组平均能完成的任务。DeepSeek 单独运行时停在简单到中等的边界处，而 DeepSeek + IronBee 和 Opus 都推进到了高难度区域。

分析二：相同结果，不同价格

现在，我们比较 Opus 单独运行与 DeepSeek + IronBee 的表现。首先展示 Opus 自己的运行及其成本；然后将两个测试组并排对比。

表 5. Opus 的历次运行及成本，以及闭环组的相应成本。得分范围：21-100。成本基于表 1 公布的 Opus 价格，为单次运行成本。

表 6. 两个测试组在得分上落在同一区间，但每次运行成本约为七分之一。DeepSeek + IronBee 的成本包含了验证闭环自身的模型调用。DeepSeek + IronBee 各次运行成本：2.27, 2.75, 1.77, 2.27, 2.81。

这两者的得分落在同一区间。Opus 基线得分 82.8。DeepSeek + IronBee 得分 80.6，来自上面的表 4。前一节中 IronBee 带来的约 4 倍提升，正是将低成本模型拉升至与前沿模型持平的原因。而且它成本低廉：Opus 基线每次运行成本接近 15 美元，DeepSeek + IronBee 约 2.4 美元，约便宜 7 倍。因此，是验证，而非一个更大的模型，以一小部分价格达到了前沿水平的智能。

对此补充两点诚实的说明。第一，中位数差异大于平均值差异：Opus 在正常工作完成情况良好时，倾向于完成整个项目，而闭环组则更多仅差一点完成。第二，请注意 Opus 的 k=1 运行：这个前沿模型也有一次运行在任务 7 就崩溃了。开环失败并非弱模型的专属问题；只是它的成本更高。

What a Verification Loop Adds to a Coding Agent: A First Look

This is the opening post in an ongoing series. We start with one model pair on one project, and the analysis will continue across more models and more datasets. We are sharing early on purpose, and we will keep sharing as we go.

Press enter or click to view image in full size

Introduction

A coding agent can produce a lot of code quickly. What it cannot do, on its own, is know whether that code works. The model writes something plausible, the run moves on, and any mistake travels with it. On real multi-step projects this compounds: one unverified error early on can sink everything built on top of it. The fix human teams use is old and boring. Someone checks the work, against the running application, before anyone builds on it.

This post asks one question: how much does adding that check change what a coding agent can finish? Our hypothesis going in was that a closed verification loop (run the app, exercise the change, find the failure, drive the fix, then prove the repair) matters more than raw model strength for a large class of failures. Code is not done until it is verified.

To test this we needed a concrete verification layer, and we used our own. IronBee is a verification and fix layer for AI coding agents: when an agent writes code, IronBee opens the app in a browser, exercises the change, and looks for problems. When it finds one, it analyzes the failure, finds the root cause, drives the fix, and then re-verifies, so a fix counts only once it is proven. We built IronBee, so read this as a disclosed-interest experiment with an open method. In this post IronBee is the instrument, not the subject: the conclusions we care about are about verification loops in general, and the setup is public so anyone can rerun or challenge it.

We spent a week building a careful test, and we are sharing the opening results now, with more to come. We took real web development work and ran two models, one low-cost and open, one a commercial frontier model. Setting this up well was harder than it sounds, and the next section explains why. We treat this as a beginning, not a verdict. The point of the series is to see how the picture changes as we add more models and more kinds of work.

Methodology

Choosing the right dataset. We wanted work that looks real, not toy problems. We picked Web-Bench, an open benchmark from ByteDance. It is public, so anyone can check our setup: the code is on GitHub, the dataset is on Hugging Face, and the design is described in the Web-Bench paper. Web-Bench has 50 projects. Each project is a single web app assembled over 20 tasks that must be done in order, so each task builds on the one before it. That sequential structure is exactly why we chose it over a benchmark of independent, one-off tasks: a verification loop earns its keep when work builds on earlier work, so an unverified error compounds instead of staying local. Every task has a hidden test that checks the result, and the agent never sees it. If a task fails, the run stops there. This keeps the benchmark honest: an agent cannot skip ahead or hide a broken step.

For this first work we use one of these projects, the survey app, which builds a form with questions, required fields, and a preview. It spans the difficulty range we cared about, from a simple form to logic that has to survive the round trip between the design page and the preview. It is also a good match for what we set out to measure. A verification loop pays off while the code is being written, on exactly this kind of failure: code that reads as correct on its own but only breaks when the running app is exercised. So the lift we see here reflects what verification actually adds during real development, not an artifact of an easy or contrived task. Adding more projects is one of the first things on the list.

Choosing the models. We started with two. DeepSeek (deepseek-v4-pro) is a strong model with a low price, so it stands in for the low-cost coder we want to lift. Opus (claude-opus-4–8) is a frontier model, and it sets the quality bar. Putting them side by side lets us ask a clear question: can a verification loop bring a low-cost model up to the level of a frontier model? We built the harness so we can run several models through the same path, and this pair is where we begin. One note on the setup. In this first run DeepSeek is text-only, so it cannot see the screen. That is fine here, because the checks read the page as structured text, through the accessibility tree, the DOM, and the console, not as an image. The model reaches its result on structured feedback, not by looking at pixels.

What the verifier can and cannot see. Two rules keep the comparison fair, and they are worth stating plainly because they are the first things a skeptical reader will ask. First, the hidden Web-Bench tests stay hidden from everyone: the coding model never sees them, and neither does the verification layer. The tests live only in Web-Bench’s evaluator, which runs after the agent returns its files; IronBee derives its checks from the task description and the running application, the accessibility tree, the DOM, the console, not from the benchmark’s test files. Second, the loop is not free, and we count it: the verification layer makes model calls of its own, and those calls are included in every cost figure for the DeepSeek + IronBee arm in this post. And one more thing worth stating: the verification layer runs on the same provider as the coder, not a stronger one. Its analysis and fix steps use DeepSeek’s smaller, cheaper model, deepseek-v4-flash, while the code itself is written by deepseek-v4-pro. So the lift is not a more capable model slipped in through the verifier.

Setting the retry level. The agent gets a task, writes code, and a hidden test checks it. We give each task a small retry budget, and the retry level matters more than it looks. Some tasks are almost impossible to pass on the first try, by design: the hidden test checks a precise mechanism that the description states loosely, or even in a misleading way.

Task-9 asks for a checkbox to set the number of stars, but the test needs a control that can hold a number like 5 or 10, which a checkbox cannot. Task-15 asks to make questions required, where the test only accepts the browser’s built-in required marking, not hand-written checks that merely look right. A model that follows the wording literally picks the wrong mechanism and fails, and it cannot know until it sees the error. Because it does not always make the same choice, the same task can pass on one run and fail on the next, and the second try, now armed with the error, often recovers it. That is the point: the benchmark asks whether a model can read a failure and repair it, not just guess the intent blind. So one try mostly measures blind guessing. With three tries most agents get there in the end, so extra help is hard to see. With two tries real differences show up, so we run with two.

Is the loop just more attempts? One objection deserves an answer up front. A verification loop spends extra inference per task, so is the lift just a bigger compute budget in disguise? Partly it has to be. The loop does more work. But the retries the benchmark grants are blind: the model sees a failure signal and guesses again. The loop’s iterations are guided by evidence from the running application, which is a different kind of attempt, not just another one. Whether guided iteration beats an equal budget of blind retries at matched cost is exactly the ablation this framing demands, and it is planned for a future post: DeepSeek alone with a larger retry budget, against DeepSeek with the loop, dollar for dollar. Until that runs, read the results below with this open question in mind.

Measuring cost honestly. The cost turned out to be subtle. The built-in cost number is not reliable for a third-party model, because it prices every model as if it were the frontier model. The gap is large: applying frontier prices to DeepSeek overstated its cost by roughly twenty times. So for the low-cost model we did not trust the built-in number. We computed the cost from the provider’s own published token prices instead, and every cost figure in this post is built that way.

Press enter or click to view image in full size

Table 1. Price Catalog for Opus and DeepSeek models.

Trusting the numbers. The same model can finish very different amounts of work from one run to the next. A single run tells you little. So we run each setup five times and read the averages. We also pinned one published version of IronBee, so every run used the same tool, and we kept parallel runs fully separate so they could not interfere with each other.

Scoring by difficulty. Web-Bench labels every task easy, moderate, or challenging, and the tasks are sequential. So we turn how far a run gets into a single score from 0 to 100, weighted by difficulty. The base plus all 20 survey tasks add up to exactly 100. This weighting is our own reading aid; the native Web-Bench metric is a simple pass rate that does not weigh by difficulty.

Press enter or click to view image in full size

Table 2. The difficulty-weighted score. The base and the 20 survey tasks sum to exactly 100.

Results

Before the numbers, here is how we read them. The question of the post, how much does closing the loop change what an agent can finish, splits into two comparisons. First we compare a model against itself, with and without the loop. Then we compare across models: a low-cost model inside the loop against a frontier model without it, on both score and price. We take them in that order.

Analysis 1: How far each arm gets

Here we compare DeepSeek on its own against DeepSeek with IronBee.

Press enter or click to view image in full size

Table 3. DeepSeek on its own. Score is out of 100, weighted by difficulty.

Press enter or click to view image in full size

Table 4. The same model, with IronBee checking and fixing the work.

DeepSeek on its own clears the easy tasks and then stalls at the first moderate one in four runs out of five; one run reaches task 12. That is an average of 6.8 tasks finished out of 20, an average score of 20.4, and a median of 11. With the loop, the same model pushes through the moderate band and well into the challenging band: 17 tasks finished on average, a score of 80.6, a median of 86, and one run that completed the entire project. The worst run with the loop (62) beats the best run without it (48). In effect, IronBee provided 4x the problem solving intelligence compared to the model on its own, utilizing the same model and prompts.

The difficulty labels make this easy to picture. The baseline hits a wall at the line between easy and moderate tasks, and it stops. The loop carries the same model past that wall and into the hard tasks. The key point is what did not change. We did not switch to a bigger model or write a better prompt. The model is the same. What changed is that the agent now gets a second, evidence-based look at its own work. The checker exercises the change in a real browser, reports what is wrong, and the agent fixes it before it moves on.

The mechanism is worth dwelling on, because it explains why the lift is this large. In this project most of the hard bugs hide in the round-trip from the design page to the preview page: the code for each page can look correct in isolation while the hand-off between them is broken. That is exactly the kind of failure a first pass tends to miss and a check against the running application tends to catch. The wall in the baseline is not a lack of coding ability. It is the absence of any ground truth about whether the code works.

Press enter or click to view image in full size

Figure 1. How far each arm gets across the 20 sequential tasks. Cells are shaded by task difficulty; solid cells are the tasks an arm completes on average. DeepSeek on its own stops at the easy-to-moderate wall, while DeepSeek with IronBee and Opus both push into the challenging tier.

Analysis 2: Same result, different price

Now we compare Opus on its own against DeepSeek with IronBee. First, Opus’s own runs and their cost; then the two arms side by side.

Press enter or click to view image in full size

Table 5. Opus’s runs and their cost, with the loop arm’s cost alongside. Score range: 21-100. Cost is per run, at the published Opus prices from Table 1.

Press enter or click to view image in full size

Table 6. The two arms land in about the same place on score, at roughly one-seventh of the cost per run. The DeepSeek + IronBee cost includes the verification loop’s own model calls. Per-run DeepSeek + IronBee costs: 2.27, 2.75, 1.77, 2.27, 2.81.

These two land in about the same place. Opus baseline scores 82.8. DeepSeek with IronBee scores 80.6, from Table 4 above. The roughly 4x that IronBee produced in the previous section is what brought a low-cost model level with a frontier one. And it did so cheaply: Opus baseline costs near 15 dollars per run, DeepSeek with IronBee near 2.4 dollars, about 7x cheaper. So verification, not a bigger model, reached frontier-level intelligence for a fraction of the price.

Two honest footnotes to that. First, the medians differ more than the averages: Opus, when it works, tends to finish the whole project, while the loop arm more often lands just short. Second, look at Opus’s k=1: the frontier model also had one run collapse at task 7. Open-loop failure is not a weak-model problem; it just costs weak models more often. That is exactly why the next arm we want to run is Opus with the loop, to see whether verification adds reliability at the top end too.

This is the practical payoff we wanted to test first. When quality matters, teams often reach for the most capable model by default. This first data point suggests another path: for some workloads, a low-cost model with a verification loop lands in the same place for far less. Whether that generalizes beyond one project and one loop implementation is what the rest of the series is for.

Closing

So what does this first look suggest? Closing the loop did not just polish the output. It changed how far a low-cost model could go, from a wall at the first moderate task to matching a frontier model’s average on this project, and it did so at a fraction of the frontier model’s cost. Those are two clear early signals, and they point at a simple idea: a lot of what separates a weak run from a strong one is not raw model power. It is whether anything checks the work, against the running application, before the agent moves on.

We also want to be honest about the limits, because this is a starting point. It is one project, one pair of models, and one implementation of a verification loop, ours. The same model can swing a lot from run to run, so we read these as early signals, not final numbers. The gains are not automatic either: verification helps most when the model can understand and act on what the check reports, so the size of the lift can vary from one model to the next. And the ablation that would separate guided iteration from a plain bigger retry budget has not run yet.

That last point is exactly why the series continues from here. The most interesting question is not whether verification helped one low-cost model on one project. It is how much it helps across the board, and where it stops helping. So next we are adding more projects, so the result does not rest on a single app. We are running the matched-cost retry ablation described in the methodology. We are adding an Opus-with-loop arm, to see whether verification raises reliability for frontier models too. We are moving to harder problems, where a first attempt is more likely to be wrong and a verification loop has more to catch. And we are adding more models, including multimodal ones, the text-only setup here is a lower bar, since the model reached this result without ever seeing the screen. We will keep the method and the numbers open as we go.

If you work on AI coding quality, we would love to hear how you measure it, and what you would want to see us test next. This is only the first post, and more is coming soon.

References

1. GitHub Web-Bench repository: github.com/bytedance/web-bench
2. Hugging Face: huggingface.co/datasets/bytedance-research/Web-Bench
3. Web-Bench paper: arxiv.org/abs/2505.07473
4. Claude Pricing: https://platform.claude.com/docs/en/about-claude/pricing
5. Deepseek Pricing: https://api-docs.deepseek.com/quick_start/pricing