🔶 Computer Anthology：面向 AI Agent 的持续演进基准测试家族

介绍 Computer Anthology

Agent 基准测试正以两种方式失效。显而易见的一种是饱和：在定义了过去两年的那些测试套件上，前沿模型与 Agent 组合的成绩都集中在接近上限、彼此只差几分的范围内，而仍能拉开它们差距的许多因素，其实是那些会把正确解判错、或把错误解判对的任务所带来的噪声。处于这种状态的排行榜，已经无法告诉任何人应该选用哪一个模型。

更隐蔽的失效是，基准测试通常被作为一次性项目构建。数月时间投入到任务编写、审查和校准中，随后发布一个静态成果；前沿能力追上来后，这个成果便被耗尽，而用于构建它的大部分机制也无法延续到下一个基准。构建一套更难的单次任务集，所能争取的时间已经比过去少得多，因为最近几代模型会在数月内消化掉绝大部分新增难度。

Computer Anthology 针对的是第二种失效，我们认为这会让第一种失效变得可控。它是一个基准测试家族，每个基准衡量一种计算机技能——终端操作、GUI 计算机使用、逆向工程、程序合成、仓库级工程——而不是将所有能力压缩为一个 Agent 分数。我们为产出这些基准而构建的东西，更接近一个数据引擎而非一个数据集：它由 Agent、专有模型、人在回路中的审核者以及基础设施构成，这些资源会在不同基准之间复用，并随着模型进步持续改进。每个基准都会将这个生态系统组装为自己的流水线，因为不同技能需要不同的环境、验证器和校准方式；因此，每一项新基准都从比前一项更强的机制起步。当一套任务饱和时，意味着应当组装它的继任者，而不是从头再做一遍。

每一项基准都会保留未公开任务：这些任务从不用于训练，并且在仍具有测量价值时绝不发布，因此同一任务集可以跨模型世代重新评分而不受污染。一些条目可能会完整发布，保留任务也可以在基准饱和或新版本取代旧版本后开放。只要条件允许，我们发布的内容都会采用开放格式，即 Harbor 或 cua。

终端任务 v1.0

第一项基准包含 100 个自包含、由验证器评分的任务，Agent 需要从终端完成它们；它重建了 Terminal-Bench 的模式：在 Terminal-Bench 2.0 上，前沿模型与 Agent 组合如今聚集在约 85% 附近，而中位任务大约每十次尝试可通过七次。v1.0 由四项属性定义，每项都在方法论中有可操作的定义：

• 困难。 每个任务在编写过程中都针对一个前沿模型进行了校准，只有当该模型在校准试验中的通过率不超过 60% 时才会被接受。我们测得的最佳配置首轮尝试通过率为 61.8%，其中 17 个任务在十次尝试中最多只会被解出一次。
• 公平。 只要把任务写坏，构建困难任务会变得轻而易举。任务集中的每个任务都带有经验证据，证明它可以仅基于自身提供的材料被解决；每条轨迹都由一名依据缺陷分类法工作的评审者阅读，任何被标记的问题都会由人工裁决。
• 确定性验证。 每个任务都附带一个沙箱化验证器，输出二元奖励。没有 LLM 评审者为任何尝试评分，结果在不同运行之间可复现。
• 人工审查。 在进入任务集之前，Vetto 研究人员审阅了全部 100 个任务的指令、环境、验证器和参考解答。

每个任务都是一名合格从业者有可能实际接到的工作。该任务集覆盖 17 个类别、11 种语言和运行时，重点偏向数据科学、ML、优化和模型训练工作；它评估成果是否能在指定约束内运行，而非代码质量或风格。

v1.0 已经显得过时，这也是所有基准测试的共同命运：它是在 Claude Opus 4.6 和 GPT-5.4 的时代完成校准的，而仅仅几代模型就将其汇总通过率从约 15% 提升至 61.8%。我们认为这是一个好信号。一个根据两个具名模型的失败案例构建的任务集，原本可能会变成一堆特殊怪癖；但实际上，每一代新模型都在其上稳步提升，这表明这些任务确实追踪到了实验室正在刻意改进的能力。我们使用了一些技术来尽量排除这类偏差。这个标尺正在发挥作用，同时它也在开始倒计时。

这些任务在仍然具备测量价值期间保持不公开，因此我们发布其结构、协议和结果。我们也可以应请求运行模型，包括实验室在其自有脚手架下的私有模型（任务绝不会离开我们的基础设施）。

结果

以下每个数字都来自同一套协议：每个配置、每个任务进行 5 次独立试验，即每个配置 500 次试验；试验在 Harbor 上运行，并由每项任务自己的验证器评分。每次试验都会获得一个新构建、相互隔离的容器，并配备任务声明的 CPU 和内存。Agent 可使用一小时的实际时钟时间；这一上限是从更短的预算逐步提高，直到时间限制不再影响大多数结果后确定的。我们将模型与 Agent 作为一个组合进行评估和报告。达到时间上限或因基础设施故障而终止的试验，会被计为失败而非剔除，因此每个分母始终为 500。区间为试验层面的 bootstrap 95% 置信区间；相同模型在不同脚手架下的比较，则按每个任务的差异进行配对。

排行榜

28 种配置的通过率范围为 2.8% 至 61.8%，平均为 30.5%。难度并非呈双峰分布，而是较为分散：中位任务在其收到的 140 次尝试中通过率为 29.6%，17 个任务不高于 10%，另有 19 个任务不低于 50%。其中有三种显著模式。

投入量无法解释能力。 MuseSpark 1.1 在排行榜上每次试验产生的输出 token 最多，达到 174k，但通过率为 39.0%；GPT-5.6 Sol 在 Codex CLI 下仅用 12.9k token 就达到 58.0%，而 DeepSeek V4 Pro 花费 72k token 后通过率仅为 10.4%。Terminus-2 各行内部的步数也印证了这一点：在这里，不同模型的步骤粒度相同。Claude Fable 5 每次试验的中位步数为 15 步，得分为 50.8%；Gemini 3.5 Flash 为 41 步、22.8%；Devstral 2 为 38 步、2.8%。失败试验也不会系统性地比成功试验更长：在大多数配置中，两者的中位数只相差几步；差距最大的是 MuseSpark 1.1 + Terminus-2，成功时为 19 步，失败时为 36 步。

使用按钮可在输出 token、每次试验成本和中位步数之间切换 x 轴。

Harness 可能价值两位数百分点。 从 Terminus-2 切换到原生脚手架后，GPT-5.4 的 pass@1 增加 26.0 个百分点，GPT-5.4 Mini 增加 17.8 个百分点，GPT-5.5 增加 13.0 个百分点，Claude Opus 4.8 增加 8.2 个百分点，GPT-5.6 Sol 增加 6.4 个百分点；在按每任务差异进行的配对 bootstrap 下，它们都可以区分开来。不过，原生脚手架并不总是获胜：排行榜最强的配置是运行于 Terminus-2 的 Claude Opus 5，比同一模型运行于 Claude Code 时高出 7.8 个百分点；它也是我们测得的首个在自身脚手架之外显著表现更好的模型。Claude Fable 5 和两个 Gemini 的差异均处于噪声范围内。

仅几代模型就将汇总通过率从约 15% 推升至 60% 以上。 将 Agent 固定为 Terminus-2，仅替换模型时，Claude 系列依次为 14.8%、29.8%、50.8%、61.8%（Opus 4.6、Opus 4.8、Fable 5、Opus 5），GPT 系列则为 12.0%、38.6%、51.6%（GPT-5.4、GPT-5.5、GPT-5.6 Sol）。这正是该基准存在的目的：用一个固定标尺衡量每代模型增加了多少可验证的计算机控制能力。

按类别划分的结果

使用筛选器添加或移除配置。

对于任务数不超过五个的类别，我们不报告按类别划分的结果，因为这些单元格不具统计意义。这些试验仍会计入排行榜的所有整体数字。

按排行榜平均成绩看，调试是最难的类别，通过率为 19%；优化、软件工程和数据查询最容易，通过率为 34% 至 36%。不过，不同模型对这个排序并不一致：任意两个配置类别画像之间的中位秩相关系数为 0.20，因此模型认为何种任务困难，大多是模型特定的。脚手架也是如此。Claude Opus 5 从 Claude Code 切换至 Terminus-2 后，在调试上的表现翻倍（从 26% 升至 51%），而在数据处理上从 35% 降至 20%；GPT-5.6 Sol 在相同切换下的软件工程通过率从 77% 崩落至 27%，但模型训练从 44% 升至 58%，数据科学从 40% 升至 51%。

解读任何单个单元格时需要注意一点：每次试验均采用全有或全无的评分，因此单元格表示通过该任务完整验证器的试验比例，而许多失败试验已经通过了大部分底层检查。在那 17 个十次尝试中最多解出一次的任务上，失败试验仍能通过验证器各项独立检查中的中位数 70%。我们在内部追踪部分奖励，但对外报告二元结果。

最难任务的剖析

有七个任务在其 140 次尝试中最多只被解出 7 次。我们阅读了它们所有的轨迹，失败模式惊人地一致。这些任务都取决于一个关于环境含义的判断，而非知识面的广度或冷门程度：缓存中保存的是原始频谱还是归一化频谱；携带相同载荷的两次投递是一个事件还是两个事件；两次已完成的重试中哪个才是规范版本；持久化事实应存放在检查点之内还是旁边。击败这些试验的通常不是无法推导问题，而是一个早早采纳、从未验证、并一路带到自信确认的假设。

部分得分显示这些试验离正确答案有多近。在状态机生成器任务中，三分之一的全部试验只差一项检查即可完全通过；而身份拼接任务的平均得分是其 30 项检查中的 25 项。少数成功者的差别不在于进行了更多搜索，而在于多进行了一轮怀疑：它们都为其他人默认接受的事情构建了测试。同一模型、相同 harness、相同提示词下的两次运行之所以不同，仅仅是因为其中一次回头检查了种子数据，并推翻了自己原先的去重规则。在我们阅读的所有轨迹中，没有发现任何试图操纵验证器、修改测试或硬编码预期值的尝试。

我们将这些视为困难而非损坏的任务，并在任务集的这一端坚持这种区分，因为在我们能够证明并非如此之前，没有任何解法的任务是对我们自身的质疑。对任务集的最终审查专门寻找未能通过这项检验的任务，阅读了处于最低通过率范围内每个候选任务的每一次试验。部分任务确实存在缺陷：它们发布时缺少自身测试所需的 fixture，或强制执行了说明中从未陈述的约定；我们替换了这些任务，而没有将其包装为难度。排行榜上如今没有未解任务：每项任务都至少被一种配置通过过，当前最难的任务在 140 次尝试中通过了 2 次。

一个任务，从头到尾

以下是一个完整任务：go-netscan-pattern-budget，它是一项优化任务，在 25 次展示试验中有 15 次被解决（在全部 140 次试验中有 48 次）。它不在基准测试中。发布一个保留任务就意味着消耗掉它，因此该任务已从评分任务集中退役，并由同一流水线产出的任务替换；下方浏览器中的另外两个示例任务也是如此。

Agent 会逐字收到以下指令，以及一个位于小型 Go 仓库中的 shell：

netscan-pattern 生成器会生成正确但过长的模式：在生产配置中，移动序列超出了固件的 64 字节模式槽位，netscan-pattern 会因预算错误而退出。它需要输出尽可能短的序列，同时该序列必须覆盖 rows × cols 地址空间中的每一个单元格，并在完整重复若干次后将探针返回至 (0, 0)，以便在舰队使用的每一种 (rows, cols) 配置下都能放入该槽位。netscan-pattern 二进制程序读取 -rows 和 -cols，并打印该模式。make test 会运行一个可见的小型测试套件，作为典型小规模场景的冒烟检查；生产签核要求生成器在完整配置集上均正确，而不只是覆盖可见测试套件中的那些配置。默认构建命令是 make。

探针会扫描一个在两个轴上都环绕的 rows × cols 地址空间，而随附的生成器采用显而易见的方式逐列移动，生成一个长度与行数相同的序列。在这个表象之下，Agent 必须推导出一个真实结论：覆盖一个环面并返回原点的最短模式长度为 max(2, gcd(rows, cols))，因为每次重复都会让探针沿固定向量位移，而该向量生成的子群的每个陪集都需要一次移动。得到这个下界只是容易的一半。大多数配置可通过单调阶梯 D^a R^b 达到该界限，但评分集合中有 11 种配置根本不存在这种拆分：每个与行数互素的拆分都与列数共享一个因子。这些配置需要一个包含回退步的词序列，且必须选取使其前缀落在不同陪集中的序列。工作区提供了一个可用于测试候选解的暴力覆盖检查器，以及两个看似合理、用于计算 min 和 lcm 的辅助模块，但两者计算的量都不正确。未修改的错误生成器可以通过可见的 make test。

无需提交任何内容。验证器会从 Agent 的源代码重新构建二进制程序，运行 Agent 自己的可见测试套件，然后在 25 个保留配置上调用该二进制程序；对于每个配置，它会检查生成的模式是否覆盖整个空间、是否返回原点、是否能放入 64 字节槽位，以及其长度是否恰为 max(2, gcd)。覆盖性通过代数方法而不是模拟进行检查，因此可在数秒内对包含 2.13 亿个单元格的网格评分。配置集的设计旨在杜绝记忆化：它包含每个受阻网格的轴交换版本和标量倍数版本，因此针对每个 gcd 值硬编码一个获胜词序列的生成器会在转置配置上失败；还有一个受阻配置族从未出现在可见材料中。

五种配置各尝试了 5 次：运行于 Claude Code 的 Claude Opus 5 和 Claude Fable 5，以及运行于 Codex CL 的 GPT-5.6 Sol 和 GPT-5.5

Introducing Computer Anthology

Agent benchmarks are failing in two ways. The visible one is saturation: on the suites that defined the last two years, frontier pairs cluster within a few points of the ceiling and of each other, and much of what still separates them is noise from tasks that fail correct solutions or pass broken ones. A leaderboard in that state has stopped telling anyone which model to reach for.

The quieter failure is that benchmarks get built as one-shot projects. Months go into authoring, review, and calibration, a static artifact ships, the frontier catches up, and the artifact is spent with little of the machinery surviving to build the next one. Building a single harder set buys less time than it used to, since recent generations have consumed most of the added difficulty within months.

Computer Anthology attacks the second failure, on the theory that it makes the first one manageable. It is a family of benchmarks, each measuring one computer skill (terminal work, GUI computer use, reverse engineering, program synthesis, repository-scale engineering) rather than collapsing everything into a single agent score. What we are building to produce them is closer to a data engine than a dataset: an ecosystem of agents, proprietary models, humans in the loop, and infrastructure that is reused across benchmarks and keeps improving as models improve. Each benchmark assembles that ecosystem into its own pipeline, since different skills need different environments, verifiers, and calibration, so every entry starts from stronger machinery than the last and a saturated set means assembling its successor rather than starting the work over.

Every entry keeps tasks held out, never trained on and never released while they are still measuring something, so a set can be re-scored across model generations without contamination. Some entries may be published in full, and held-out tasks can open up later, as a benchmark saturates or a new version supersedes it. Whatever we release ships in an open format when possible, either Harbor or cua.

Terminal Tasks v1.0

The first entry is 100 self-contained, verifier-graded tasks an agent completes from a terminal, rebuilding the Terminal-Bench formula: on Terminal-Bench 2.0 frontier pairs now cluster near 85% and the median task is passed in roughly seven attempts out of ten. Four properties define v1.0, each with an operational definition in Methodology:

• Hard. Every task was calibrated against a frontier model while it was being written, and accepted only when that model passed at most 60% of its calibration trials. The best configuration we have measured passes 61.8% of first attempts, and 17 tasks are solved on at most one attempt in ten.
• Fair. A hard task is trivially easy to write by making it broken. Every task in the set carries empirical evidence that it is solvable from its own materials, and every trajectory is read by a judge working from a defect taxonomy, with a human ruling on anything it flags.
• Deterministically verified. Each task ships a sandboxed verifier that writes a binary reward. No LLM judge grades any attempt, and results reproduce run over run.
• Human-reviewed. Vetto researchers read the instruction, environment, verifier, and reference solution of all 100 tasks before they entered the set.

Each task is a job a competent practitioner could plausibly be handed. The set spans 17 categories and 11 languages and runtimes, weighted toward data science, ML, optimization, and model-training work, and it grades whether the artifact works within the stated constraints, not code quality or style.

v1.0 is already dated, in the way every benchmark is: it was calibrated in the era of Claude Opus 4.6 and GPT-5.4, and a couple of model generations took it from roughly 15% pooled to 61.8%. We read that as a good sign. A set built from the failures of two named models could have turned out to be a pile of quirks, and instead each new generation has climbed on it steadily, which suggests the tasks track something the labs are deliberately working on. We use a few techniques to try to keep those biases out. The yardstick is working, and it is also counting down.

The tasks stay held out while they remain useful for measurement, so we publish the structure, the protocol, and the results. We also run models on request, including a lab's private model under its own scaffold (tasks never leave our infrastructure).

Results

Every number below comes from one protocol: 5 independent trials per task per configuration, 500 trials per configuration, run on Harbor and graded by each task's own verifier. Each trial gets a freshly built, isolated container with the task's declared CPU and memory. Agents get one hour of wall clock, raised from shorter budgets until the limit stopped shaping most outcomes. We evaluate and report model and agent as a pair. Trials that hit the wall or die to infrastructure count as failures rather than being dropped, which holds every denominator at 500. Intervals are trial-level bootstrap 95% CIs, and same-model scaffold comparisons are paired on per-task differences.

The leaderboard

The 28 configurations span 2.8% to 61.8% and average 30.5%. Difficulty is spread rather than bimodal: the median task is passed on 29.6% of the 140 attempts it has received, 17 tasks sit at or below 10%, and another 19 at or above 50%. Three patterns stand out.

Spend does not explain capability. MuseSpark 1.1 produces the most output tokens per trial on the board, 174k, and passes 39.0%; GPT-5.6 Sol under Codex CLI passes 58.0% on 12.9k, and DeepSeek V4 Pro spends 72k to pass 10.4%. Step counts inside the Terminus-2 rows, where step granularity is identical across models, agree: Claude Fable 5 scores 50.8% on a median of 15 steps per trial, Gemini 3.5 Flash 22.8% on 41, Devstral 2 2.8% on 38. Failing trials are not systematically longer than passing ones either, with medians differing by a few steps in most configurations and the widest gap under MuseSpark 1.1 + Terminus-2 (19 steps when passing, 36 when failing).

Use the buttons to switch the x-axis between output tokens, cost per trial, and median steps.

The harness can be worth double-digit points. Moving from Terminus-2 to its native scaffold, GPT-5.4 gains 26.0 points pass@1, GPT-5.4 Mini 17.8, GPT-5.5 13.0, Claude Opus 4.8 8.2, and GPT-5.6 Sol 6.4, every one of them separated under a paired bootstrap on per-task differences. The native scaffold does not always win, though: the board's strongest configuration is Claude Opus 5 under Terminus-2, 7.8 points above the same model under Claude Code, and it is the first model we have measured that does significantly better outside its own scaffold. The Claude Fable 5 and the two Gemini deltas are within noise.

A couple of model generations moved the pooled rate from about 15% to above 60%. Holding the agent fixed at Terminus-2 and swapping only the model, the Claude line runs 14.8%, 29.8%, 50.8%, 61.8% (Opus 4.6, Opus 4.8, Fable 5, Opus 5) and the GPT line 12.0%, 38.6%, 51.6% (GPT-5.4, GPT-5.5, GPT-5.6 Sol). This is the measurement the benchmark exists for: a fixed yardstick for how much verifiable computer-control capability each generation adds.

Results by category

Use the filter to add or remove configurations.

We do not report per-category results for categories holding five or fewer tasks, since those cells are not statistically meaningful. Those trials still count toward every board-wide number.

Debugging is the hardest category on board average at 19%, and Optimization, Software Engineering, and Data Querying the easiest, at 34% to 36%. Models do not agree on that ordering, though: the median rank correlation between any two configurations' category profiles is 0.20, so what a model finds hard is mostly specific to the model. The same goes for the harness. Claude Opus 5 doubles on Debugging when moved from Claude Code to Terminus-2 (26% to 51%) and drops from 35% to 20% on Data Processing; GPT-5.6 Sol collapses from 77% to 27% on Software Engineering under the same move, while gaining on Model Training (44% to 58%) and Data Science (40% to 51%).

One caveat on reading any single cell: scoring is all-or-nothing per trial, so a cell is the share of trials that passed the task's full verifier, and many failing trials clear most of the underlying checks. On the 17 tasks solved on at most one attempt in ten, a failing trial still clears a median of 70% of the verifier's individual checks. We track partial rewards internally and report the binary outcome.

Anatomy of the hardest tasks

Seven tasks were solved on no more than 7 of their 140 attempts. We read the trajectories on all of them, and the failures are strikingly uniform. Each of those tasks turns on a single judgment about what the environment means rather than on breadth or obscurity: whether a cache holds raw or normalized spectra, whether two deliveries carrying identical payloads are one event or two, which of two completed retries is the canonical one, whether durable facts belong inside the checkpoint or beside it. What sinks these trials is rarely an inability to work the problem out; it is an assumption adopted early, never checked, and carried all the way to a confident sign-off.

Partial credit shows how close those trials run to correct. On the state-machine generator a third of all trials finish one check short of a full pass, and the identity-stitching task averages 25 of its 30 checks. What separates the few winners is not more searching but one extra round of doubt: each of them built a test for the thing everyone else assumed. Two runs of the same model, under the same harness and the same prompt, diverged only because one went back to the seed data and reversed its own deduplication rule. Across every trajectory we read we found no attempt to game a verifier, edit a test, or hardcode an expected value.

We read these as hard rather than broken, and that is a distinction we hold ourselves to at this end of the set, because a task nothing solves is a claim about us until we can show otherwise. The final review of the set went looking for tasks that fail that test, reading every trial of every candidate at the bottom of the range. Some turned out to be genuinely defective, shipping without a fixture their own tests require or enforcing a convention their instructions never state, and we replaced them rather than present them as difficulty. Nothing on the board is unsolved today: every task has been passed by at least one configuration, and the hardest one standing is passed on 2 of its 140 attempts.

One task, end to end

Here is one task in full: go-netscan-pattern-budget, an Optimization task that 15 of 25 showcase trials solved (48 of its 140 trials in all). It is not in the benchmark. Publishing a held-out task burns it, so this one was retired from the scored set and replaced from the same pipeline, as were the two other sample tasks in the explorer below.

The agent receives this instruction, verbatim, and a shell in a small Go repository:

The netscan-pattern generator emits patterns that are correct but too long: on the production configs the move sequence overruns the firmware's 64-byte pattern slot, and netscan-pattern exits with a budget error. It needs to emit the shortest sequence that still covers every cell of the rows × cols address space and returns the probe to (0, 0) after a whole number of repetitions, so it fits the slot on every (rows, cols) the fleet uses. The netscan-pattern binary reads -rows and -cols and prints the pattern. make test runs a small visible suite as a smoke check on the typical small cases; production sign-off needs the generator correct across the full configuration set, not only the ones the visible suite covers. The default build is make.

The probe sweeps a rows × cols address space that wraps on both axes, and the shipped generator walks it the obvious way, one column at a time, emitting a sequence as long as the row count. Behind that framing is a real result the agent has to derive: the shortest pattern that covers a torus and returns to the origin has length max(2, gcd(rows, cols)), because each repetition displaces the probe by a fixed vector and one move is needed per coset of the subgroup that vector generates. Getting the bound is the easy half. Most configurations reach it with a monotone staircase, D^a R^b, but the graded set contains eleven where no such split exists at all: every split coprime to the row count shares a factor with the column count. Those require a word with back-steps in it, chosen so its prefixes land in distinct cosets. The workspace ships a brute-force coverage checker the agent can test candidates against, and two plausible-looking helper modules computing min and lcm that are both the wrong quantity. The visible make test passes on the unmodified broken generator.

There is nothing to submit. The verifier rebuilds the binary from the agent's source, runs the agent's own visible suite, and then invokes the binary on 25 held-out configurations, checking for each one that the emitted pattern covers the space, returns to the origin, fits the 64-byte slot, and has length exactly max(2, gcd). Coverage is checked algebraically rather than by simulation, which is how grids of 213 million cells are graded in seconds. The configuration set is built to close off memorization: it includes axis swaps and scalar multiples of each obstructed grid, so a generator that hardcodes one winning word per gcd value fails on the transposes, and one obstruction family never appears in the visible material at all.

Five configurations attempted it 5 times each: Claude Opus 5 and Claude Fable 5 under Claude Code, GPT-5.6 Sol and GPT-5.5 under Codex CLI, and Gemini 3.5 Flash under Terminus-2.

Hover any dot to read the agent's step at that moment.

Claude Opus 5 passed 4 of 5, GPT-5.6 Sol and GPT-5.5 4 of 5 each, Claude Fable 5 3 of 5, and Gemini 3.5 Flash 0 of 5. Every trial built cleanly and left the visible suite green; nobody broke the repository they were handed. Three of the ten failures are not reasoning failures either: they are Claude Code trials the provider cut off mid-run, twelve to fourteen minutes in, with the workspace still untouched and the reasoning demonstrably on track. We count them as failures rather than dropping them, which is why this task's Claude Code numbers understate both models.

The rest of the board fails in one shape. Across all 140 trials, 21 of them, spanning twelve different model-and-harness configurations, emitted the byte-identical wrong answer DRRRRRRRRRRRRRRRR. That is not a coincidence: it is the true optimum of the wrong search space. Seventeen moves is the shortest staircase for that grid, and every agent that assumed the alphabet had two letters instead of four converged on it. Another 20 trials never got past a gentler version of the same trap, a grid where a staircase does reach the bound but only if it starts with seven down-moves instead of the obvious one. The single missing insight is small and sharp, and the winners all state it explicitly before writing any code: the length bound still holds, so allow back-steps. Their answers look like DRRRDRRRDRRRDRUR. One Opus 5 trial went further and enumerated, from the one obstructed grid visible in its workspace, both of the obstruction families the hidden verifier is built from, including the one it had never seen.

What makes the task worth showing is that the wrong answer is a good answer. DRRRRRRRRRRRRRRRR genuinely covers all 3.8 million cells, genuinely returns to the origin, and sits 47 bytes inside the firmware budget it was asked to fit. Judged against the business outcome the instruction opens with, it is a fix; it fails only the minimality clause in the next sentence. All seven failures that ran to completion signed off with confidence, and one went further: having convinced itself that 17 was the true minimum, it went back and relaxed a minimality assertion it had earlier added to its own test suite so that its suite would agree. Its final report lists the length as verified. The self-verification loop worked perfectly and was pointed at the wrong hypothesis, which is the argument for grading against a verifier the agent never sees.

Explore the task yourself

Everything above is summarized from the raw artifacts; the explorer beside the trajectory chart holds the artifacts themselves, for this task and two more retired samples spanning Data Querying and File Operations. It shows each task as the agent saw it, the full verifier, and all 25 trajectories step by step: stated reasoning, every command, and what came back.

The verifier sources are published here only because these three tasks are retired; for every live task they stay hidden. The one remaining redaction is each task's build-time seeding script, which the agent's container never contains.

We keep this level of instrumentation for all 100 tasks. Every leaderboard number above is backed by trajectories like these.

What is in the set

Each task gives the agent a working directory, an instruction.md, and a definition of done enforced by a hidden verifier, with every requirement the verifier checks stated in the instruction. The agent works in a sandboxed Linux environment under fixed resource limits (1 CPU and 2 GB RAM by default, plus storage and wall-clock caps) and is graded on the artifact it produces. No task depends on the public internet: even the web tasks run against local sites and localhost APIs, so results do not change when external services do. Containers may have network access for the agent's own tooling, but no verifier checks anything outside the sandbox.

The 100 tasks span 17 categories, allocated in three tiers rather than in proportion to where tasks are easiest to author:

Tier	Categories	Tasks	Rationale
Moat	Data Science, Machine Learning, Optimization, Model Training	44	Research, data, and training work that other agent benchmarks largely lack.
Core	Software Engineering, Data Querying, Data Processing, Security, System Administration	32	Essential computer-control work; SWE de-emphasized because code-specific suites already cover it.
Base	Debugging, Scientific Computing, File Operations, Video Processing, Web Browsing, Personal Assistant, Games, Mathematics	24	Quintessential or niche surfaces, kept at a representative floor.

The set runs against real engines, including PostgreSQL, SQLite, MongoDB, nginx, PAM, git, ffmpeg, and ext4/debugfs, across 11 languages and runtimes. The intent is to measure computer control, not only Python fluency.

Across categories the tasks cluster into a handful of overlapping capability themes, and most exercise several, so the counts sum to more than 100. The largest theme, diagnosing and repairing a subtle defect in working-looking code, covers more than a third of the set.

21 tasks impose production-style constraints (bounded memory, out-of-core processing, latency SLAs, deterministic or idempotent output), and on those a correct but resource-blind solution fails. One live Optimization task hands the agent a working threat-intel scanner that walks a 50,000-entry blocklist once per line of a multi-gigabyte gateway log. The headline idea (a multi-pattern string index) is standard; the difficulty is the contract around it, from hostname-label boundaries to byte-identical output across input modes, verified against a hash under a peak-RSS cap.

How the tasks were built

Hard and fair, operationally

Hard means anchored to a low frontier pass rate, and the anchoring rule was explicit and single: every task was calibrated during authoring against one of two then-current frontier pairs, 62 tasks against GPT-5.4 + Codex and 38 against Claude Opus 4.6 + Terminus-2, and accepted only when its calibration target passed at most 60% of calibration trials. A set this hard for the calibration frontier leaves headroom for the next generation, so the leaderboard keeps discriminating instead of pinning at 100%. Selecting tasks because named models fail them also biases the set toward those models' weaknesses, which we measure in Selection on failure.

Fair is the harder property, and we would rather be transparent about why than claim we have solved it. A fair task clears three bars at once. It must not reject correct work, so every requirement the verifier checks is stated in the instruction and no check pins an incidental choice of the reference solution. It must not accept incorrect work, so there is no path to the reward through hardcoding, guessing, or reading the answer out of the environment. And it must be worth measuring, grounded in work that matters in real use rather than a puzzle designed backward from a verifier. These bars pull against each other: every check added to block a false positive is a new chance to create a false negative, and the realism that makes a task worth measuring multiplies the places ambiguity can hide. There is no oracle for any of the three, so we use instruments that cover one another's blind spots: human expert review in several independent stages, LLM judges armed with a human-built taxonomy of recurring defects, and empirical evidence from two directions, easing (does information derivable from the task's own materials move the pass rate) and semantic perturbation (does the difficulty survive rewording).

The authoring pipeline

The current process is the third iteration. Experts-only authoring cost roughly 20 hours per task and still produced ambiguous specifications and verifier false positives at a high rate; adding agents after the fact caught real defects but made for a slow loop where errors compounded across rounds. What worked was putting the agents inside the loop, packaged into an SDK the authors use directly for debugging environments, generating boilerplate, and running calibration models against drafts. It beat both expert-only authoring with model-based checking and synthetic generation with human review, and it lets experts spend their hours on what agents handle poorly: choosing problems from real production incidents, deciding what makes a task genuinely hard, catching the ambiguity a verifier would miss.

The authoring pipeline Authoring domain experts, ~5 h active time Agent SDK environment debugging · boilerplate calibration runs against drafts INDEPENDENT REVIEW — IN PARALLEL Reviewer 1 no SDK access · frontier trajectories in hand Reviewer 2 no SDK access · frontier trajectories in hand feedback — up to 3 rounds Curation Vetto triage — fairness, difficulty & quality audit Hinting (easing) optional hints · measured pass-rate lift Judge scoring quality, difficulty & metadata from task, oracle, trajectories, pass rates & hint lift Human review final ready / not-ready call LLM judge re-scores after every fix

Each step is gated by a different group, and no task reaches the benchmark without clearing all of them:

1. Authoring. Domain experts create each task inside the SDK, running the calibration models against drafts as they iterate. The 60% difficulty bar is measured here, against live calibration runs. Around 5 hours of active human time per task, a quarter of the expert-only cost.
2. Independent review, two reviewers in parallel. Both work independently of each other and of the author, without SDK access but with frontier-model trajectories in hand. A task typically survives up to three rounds of findings before both sign off. Around 2 hours per task.
3. Curation. Vetto-side triage: we audit fairness, difficulty, and quality, with AI assistance on the analysis, and select which tasks enter the candidate pool.
4. Hinting. Every candidate gets optional hints and an easing run measuring how much they lift the pass rate, the solvability evidence described below. The benchmark ships hints-off.
5. Judge scoring. The LLM judge reads everything produced so far (instruction, environment, reference solution, verifier, calibration trajectories, pass rates, hint lift) and writes structured metadata: a task-quality score, a difficulty estimate, capability tags.
6. Human review. A Vetto researcher makes the final ready-or-not call on the task's full history plus the judge's metadata, and after any fix the judge re-scores, which is the loop at the bottom of the diagram. Every task in the held-out set passed this stage manually. The errors researchers catch here feed back into the defect taxonomy: we add categories and few-shot examples and rework the judge, so each failure mode found by hand becomes one the judge screens for afterward.

Two instruments for fairness at scale

Human review does not reach every trajectory of every candidate, so two instruments carry the load where it does not.

The first is judges with a defect taxonomy. Unaided LLM judges are unreliable reviewers: asked open-endedly whether a task is fair, they miss the defects that matter and flag things that are not defects. Structure is what made them useful. Across audit sweeps of hundreds of submissions we catalogued how tasks actually fail: verifiers that grep the source for expected strings instead of testing behavior, absolute timing gates that tie the reward to host hardware, byte-exact output pins on formatting the instruction never specifies, probabilistic concurrency assertions that fail a correct solution some fraction of the time, requirements enforced by the verifier but absent from the instruction. Each entry carries few-shot examples, including near-misses that resemble the defect but are legitimate. Judges prompted with this taxonomy screen every candidate and its calibration trajectories, and everything they flag routes to a human for the ruling. Nothing is accepted or rejected by a judge alone, and the shipped benchmark uses no LLM judge anywhere in scoring.

The second is easing, which exists because a task no model passes is ambiguous evidence: a 0% pass rate cannot distinguish a genuinely hard task from an underspecified, mis-verified, or impossible one, and a benchmark full of unfalsifiable zeros measures nothing. So we author optional hints for every candidate and measure how much they move the pass rate, on the logic that if information legitimately derivable from the task's own materials lifts the rate, a path from prompt to solution exists. A task sitting at 0% is admitted only after easing has demonstrated a legitimate pass. This is evidence, not proof, since a hint-assisted pass does not show the unhinted prompt is free of every ambiguity, but it is a stronger check than the field's default of author intuition and spot checks. The benchmark ships hints-off; the hints exist for calibration, and when tasks like these are used for RL they also give a less capable model a denser reward signal than the unhinted prompt, which would return little but zeros.

Selection on failure: what it costs us

The difficulty bar is adversarial by construction. A task enters the set because a named model failed it, so filtering on the failures of two models over-samples whatever those two happen to be bad at, and difficulty here is not an intrinsic property of the tasks: it was manufactured against GPT-5.4 + Codex and Claude Opus 4.6 + Terminus-2. Two distortions follow. The set may over-represent those two models' failure modes, so other families get scored on terrain chosen to be hard for someone else, in either direction. And a task can clear the 60% bar not because the underlying work is hard but because it trips an idiosyncrasy of the target: a formatting habit, a tool-use blind spot, a family-specific prior. Such a task measures a quirk, not capability.

To size this, we split the 100 tasks by calibration target and compared today's pass rates on each subset.

The GPT-calibrated tasks are simply harder for everyone: all but one of the 28 configurations score higher on the Opus subset, a board-average gap of 14.6 points (25.1% vs 39.7%). That is a composition difference rather than a bias signal, and it is the baseline the rest is read against.

Against that baseline only one pair carries a self-targeting mark, and it fades. GPT-5.4 + Codex, the pair its 63 tasks were calibrated against, gaps 28.9 points where the board averages 14.6, while Claude Opus 4.6 + Terminus-2 gaps 13.4, slightly below the board. Under Codex that mark then shrinks from 28.9 points (GPT-5.4) to 22.8 (GPT-5.5) to 14.3 (GPT-5.6 Sol), and the lighter bars make the endpoint explicit: the current generation of each calibration family sits at the board average, so neither is measurably disadvantaged on the set built from its ancestor's failures.

Semantic perturbation: does the difficulty survive rewording

Quirk exploitation calls for a more direct attack: if a task's difficulty lives in its phrasing rather than its work, rewording the instruction should move the pass rate. So we ran a paraphrase fuzzer against the frozen set. Gemini 3.5 Flash rewrites an instruction as aggressively as it can (sentence structure, voice, vocabulary, paragraph organization) while a validator forces every file path, identifier, number, and enumerated requirement to survive verbatim; the environment and verifier stay untouched. We drew 20 tasks from each calibration subset and re-ran each reworded task under the board protocol with its own calibration pair, GPT-5.4 + Codex on the GPT subset and Claude Opus 4.6 + Terminus-2 on the Opus subset.

20 tasks per subset, before and after paraphrase, 5 trials per task per condition with the calibration pair.

Neither subset moved in a direction the experiment can distinguish from noise: 36.0% to 40.0% on the GPT subset, 54.0% to 48.0% on the Opus subset. The two moved opposite ways, by 4 points up and 6 points down, which is the signature of sampling variation rather than of an effect.

Our reading is that the difficulty is in the work, not in the wording. Two bounds are worth stating. A null on the average does not rule out individual tasks that are phrasing-sensitive; it says there is no systematic effect, not that no task has one. And each task saw a single paraphrase from a single rewriter, so what the experiment establishes is robustness to this kind of rewording rather than to every possible restatement.

What is next

More entries are coming over the next weeks and months, each held to the same standards of deterministic verification, calibrated difficulty, uncertainty-aware reporting, and held-out tasks. We have also started on Terminal Tasks v2, calibrated against the current frontier the same way v1.0 was calibrated against the frontier of its day.

We will keep running v1.0 against new models as they ship and publish each result. Today the best configuration completes just over three attempts in five, and the hardest tasks on the set are still solved on fewer than one attempt in twenty.

Questions and evaluation requests: