Agent 坚称任务已完成，数据库却说了“不”：微软与 Hugging Face 推出 ThinkingBox 评估基准

微软 ThinkingBox 根据 AI Agent 留下的实际记录而非生成的语句来进行评分，并进一步检验它们是否能够连续成功运行二十次。该工具现已通过 Hugging Face 提供。

图 1：ThinkingBox 在隔离的 MCP 工具会话中运行 Agent，然后对其留下的终端后端状态及副作用进行评分。摘自我们的 ThinkingBox 论文。

这是微软与 Hugging Face 联合撰写的博客，特别感谢 Tommy Guy（Enderis AI 创始人，前微软成员）、来自 Hugging Face 的 Sergio Paniego，以及前实习生 Zhuochun Li（匹兹堡大学）、Ali Keramati（加利福尼亚大学尔湾分校）和 Youngmin Ko（西北大学）的共同撰写与审阅贡献。

一位客户发来咨询。她花费 745 美元购买的厨房用具因物流“异常”被滞留在纳什维尔配送中心，已经超出预估送达时间 15 天。

AI Agent 做得很细致。它总共调用了九次工具：调取订单、核查物流轨迹、查看客户档案、两次检索退款政策、确认此前不存在工单并新建了一个、记录了时间线，并且对政策的理解完全正确——该客户的账户层级确实不符合延误赔偿的资格。

接着，它将工单标记为**已解决（resolved）**并关闭，然后回复客户：“鉴于您的问题已解决，请问还有什么我可以为您效劳的吗？”

然而有两处严重错误：首先，物流异常状态仍然处于开启状态，因此规定的最终状态应为**挂起（on hold）**并等待解决；其次，客户提出的核心疑问从未得到实质性解答。

如果用 AI 评测器来检查工具调用，会发现这九次调用均格式良好；如果检查 Agent 是否写入了数据库，它也确实写入了。真正提出异议的，是数据库本身。

这种差距正是 ThinkingBox 所衡量的核心。在涵盖 507 个有状态的真实业务工作流中，每项任务都会针对各类 LLM 模型独立运行 20 次，根据终端后端状态与副作用对 Agent 进行综合评级。本文将介绍我们的发现、一致性可靠性的成本，以及如何通过 OpenEnv 自行运行该基准。

你可以亲自运行这个用例： 上述示例改编自基准任务 sandbox_external_retail_group1.py:test_case_ST003_006，未通过的可执行检查仅涉及一个字段：工单状态本应为 hold，实际却被设为了 solved。完整执行轨迹见我们论文的附录 D.4 案例 3。

目录

1. 工具调用不等于最终结果
2. 单次成功不代表系统可靠
3. 你能信赖 Agent 背后的模型吗？
4. 一致性（Consistency）的成本有多大
5. 失败特征分类
6. 运作原理
7. 动手亲自运行
8. 未来的探索方向

想在阅读结果前亲自尝试？可直接跳至“动手亲自运行”一节。

1. 工具调用不等于最终结果

最终回复和合法的工具调用仅仅是代理指标（proxies）。Agent 可以在语言表述上听起来滴水不漏，却留下了错误的数值、修改了错误的记录，或者产生了多余的副作用。只有它留下的数据库记录才能真正定论。

这之间的差距极其惊人。在涵盖 12 款 LLM 模型的 121,680 次有效通用集合消融实验中，共有 79,853 次尝试未能通过可执行检查。在这些失败尝试中，有 67.24% 实际上干净利落地正常退出，成功调用了变更状态的工具，且没有报告任何终端工具错误。然而，可执行检查依然发现：77.61% 存在错误的字段值，43.30% 产生了非预期的额外副作用，25.36% 遗漏了必须的状态变更（这些状态检查结果存在重叠）。

轨迹只是一种陈述，数据库状态才是铁证，重复运行才是建立信任的试金石。

2. 单次成功不代表系统可靠

一个退款 Agent 如果成功处理了一次退款，但在接下来的四次中全部搞砸，那它绝不能算作一个合格的退款 Agent。因此，每项任务都会从一个完全一致的干净后端独立运行 20 次，我们据此汇报三个不同指标：

表 1：我们汇报的三个核心数据及各自解答的问题

指标	衡量维度	解答的核心问题
pass@1	所有单次尝试中的成功比例	它通常表现如何？
pass@20	20 次尝试中至少解决一次的任务比例	它到底能不能做出来？（广度）
观测到的 20/20	在记录的全部 20 次尝试中均完全通过的任务数	它能否始终保持正确？

在本博文中，我们使用观测到的 20/20（Observed 20/20）来代表 507 项任务中实际拿下 20 次全胜的任务绝对数量。不使用估算器，不做平滑处理。

首先从大家熟悉的视角看起。下表按领域拆分了单次尝试评分估算值 pass@1。这是多数排行榜公布的数字，单独看起来就像一份普通的模型能力排行榜。

表 2：ThinkingBox-Bench 按领域划分的 pass@1（%）。每个模型在每项任务上均进行 20 次重复测试。粗体表示分组第一名；下划线表示第二名。单次尝试评分估算的标准误差详见我们 ThinkingBox 论文中的表 4。

模型	零售 (98)	汽车保险 (100)	旅游出行 (104)	数字银行 (104)	咨询咨询 (101)	任务加权总评 (507)
专有商业模型
Claude Opus 5.5	80.97	68.40	54.28	71.25	61.58	67.16
Claude Opus 5	80.71	65.80	49.95	70.62	66.19	66.50
GPT-5.4	76.33	62.65	68.12	65.34	54.60	65.36
GPT-5.6 Sol	67.65	65.30	60.34	59.09	57.52	61.91
Claude Sonnet 4.6	72.35	54.40	58.94	56.39	54.31	59.19
GPT-6 Astra	71.73	46.55	55.87	60.87	56.83	58.31
GPT-5.2	70.20	22.40	53.70	51.15	34.06	46.28
Claude Opus 4.6	68.62	8.30	21.11	35.67	27.82	32.09
o3-pro	37.70	2.95	17.31	24.28	14.60	19.31
Grok-4.3	43.93	2.60	15.14	1.78	9.55	14.38
开源权重模型
Kimi-K3	82.24	50.80	61.83	41.35	51.63	57.37
Qwen3.8-27B	64.03	47.85	53.41	47.88	45.69	51.70
DeepSeek-V4-Pro	68.21	29.65	43.13	44.86	31.04	43.26
Kimi-K2.6	53.72	24.50	39.52	33.65	37.33	37.66
GLM-5.1	58.67	25.70	35.43	13.27	34.06	33.19
Qwen3.6-27B	43.11	29.00	46.39	27.84	18.37	32.94
Qwen3.5-9B	19.90	0.70	4.71	1.15	2.33	5.65
Mistral-Large-3	11.28	1.30	8.99	1.15	0.74	4.66

Claude Opus 5.5 以 67.16% 的总得分位列榜首，比 Claude Opus 5 高出约 0.66 个百分点。Kimi-K3 是表现最强的开源权重模型，与 GPT-6 Astra 的差距在 1 个百分点以内。业务领域的影响同样显著：Claude Opus 4.6 在零售领域拿下了 68.62% 的高分，但在汽车保险领域却骤降至 8.30%。

单次良好的运行只能表明该模型有能力完成这项任务，却无法证明它再次执行时依然能做好。因此，我们需要将每项任务重复运行 20 次，看这一得分究竟能保留多少。

图 2：在 20 次重复测试中，各模型的单次尝试得分还能保住多少。

仅有三款模型能保留下大部分的 pass@1 得分：GPT-6 Astra 保留了其单次尝试成功率的 78%，Claude Opus 5.5 和 Claude Opus 5 各自保留了 71%。而在另一端，GLM-5.1、Kimi-K2.6 和 DeepSeek-V4-Pro 各自仅保留了大约 8%。

模型单次能完成的上限与每次都能稳妥完成的下限之间的巨大鸿沟，才是整个评测中最关键的故事。

3. 你能信赖 Agent 背后的模型吗？

图 3：广度与一致性出现明显脱节。展示了 18 款模型中的 12 款；为便于辨识，省略了 pass@1 低于 33% 的 6 款模型。

Kimi-K3 在我们测试的所有模型中拥有最广泛的任务覆盖度。 它在整个基准中至少成功解决过一次的比例高达 93.89%（507 项任务中的 476 项）。仅有 31 项任务彻底难倒了它，是全场彻底失败任务数最少的模型。在零售工作流中，它以 82.24% 的 pass@1 拔得头筹，领先所有专有商业模型。

然而，Kimi-K3 也是一致性最差的模型之一。 507 项任务中，仅有 68 项（13.41%）在全部 20 次尝试中均获得成功。

Claude Opus 5 则截然相反。它至少成功解决一次的任务较少（79.09%；有 106 项任务彻底失败），但却在每单次尝试中稳稳完成了全基准 47.53% 的任务。

换用更新的模型并不能自动解决这个问题。 Claude Opus 5.5 在单次尝试平均得分上高于 Claude Opus 5（67.16% vs 66.50%），且至少解决过一次的任务也更多。但它在全部 20 次尝试中均成功通过的任务数却与 Opus 5 完全一致：都是 241 项。大盘准确率提升半个百分点，并没有换来丝毫更高的可靠度。

• Kimi-K3 至少解决过一次的任务比 Opus 5 多出 75 项。
• Opus 5 保持完全一致可靠的任务比 Kimi-K3 多出 173 项。

如果你的业务场景涉及对真实数据库记录的操作，那么 pass@20 绝对是一个具有误导性、看错了方向的指标。

4. 一致性（Consistency）的成本有多大

传统的能力对比往往止步于跑分成绩。但对于任何实际部署者而言，最关心的现实问题是：完成一个成功的有效工作单元究竟需要花费多少钱？我们将其衡量为“单次成功任务尝试的成本”（cost per successful task attempt）。我们之所以强调“任务尝试”，是因为基准中的每项任务都会被反复运行，成本是按次产生的，因此 pass@1 是与之匹配的质量分母。

我们统计了各模型在完整的 507 × 20 测试中所记录的 Token 消耗量，并按照 OpenRouter 上的无折扣公开标价计费（剔除了促销折扣，排除了标明量化的端点）。每个模型的输入、输出和缓存单价均取自同单一提供商端点。

随后，我们将单轮运行的总成本除以成功的尝试次数：

单次成功任务尝试的成本 = 507 次尝试（每项任务一次）的预估成本 ÷ (507 × pass@1)

这属于对比性质的效率指数，并非最终账单，也非实际生产环境单一请求的服务售价。同时，它衡量的是单次成功的成本，而非系统一致性。接下来我们将为一致性定价。

计算示例： GPT-5.4 运行 507 次尝试（每项任务一次）需花费 43.49 美元，其 pass@1 为 65.36%，因此单次成功任务尝试的成本为 $43.49 ÷ (507 × 0.6536) = 0.131 美元。

帕累托成本前沿（Pareto cost frontier）

如果没有任何其他模型能做到在成本不高于它的同时准确率至少与其相当，则该模型就处于前沿面上。共有三款模型符合条件；其余所有模型在至少一个维度上均被超越。

图 4：单次成功任务尝试的成本与 pass@1 关系图。带圆环的点代表处于帕累托成本前沿线上的模型。

该成本前沿呈现三个台阶： GPT-5.6 Sol 拥有最低的单次成功成本（0.127 美元）；GPT-5.4 仅需每次多花 0.004 美元，就能将 pass@1 提升 3.45 个百分点；Claude Opus 5.5 则在每次成功花费 0.276 美元的前提下，又增加了 1.80 个百分点。这三款模型之所以一直留在成本前沿曲线上，是因为没有更便宜的模型能匹敌它们的 pass@1 得分。

Claude Opus 5 则是最典型的劣势案例：单次成功成本为 0.475 美元，pass@1 为 66.50%，相比单次成功成本 0.276 美元、pass@1 达 67.16% 的 Claude Opus 5.5 而言，既昂贵又不够准确。

评估可靠一致性的成本

单次成功成本奖励的是那些既便宜又经常能答对的模型。但它并不能奖励那些“每一次都能做对”的模型。因此，我们进一步计算了“高可靠任务成本”（cost per dependable task）：即完整 20 轮运行的总成本，除以该模型在全部 20 次尝试中均获得成功的任务数量。

高可靠任务成本 = 20 轮（共 507 次尝试）的总预估成本 ÷ 达到 20/20 全胜的任务数

计算示例： GPT-6 Astra 整个测试阶段的成本为 20 × $86.03 = 1,720.60 美元，并在每项尝试中均通过了 231 项任务，因此其高可靠任务成本为 $1,720.60 ÷ 231 = 7.45 美元。

表 3：在至少有一项任务达到 20/20 全胜的模型中，高可靠任务成本最低的九款模型（由低到高排序）。数字为预估美元成本，非实际云账单。

模型	20/20 全胜任务数	20 轮预估总成本	高可靠任务成本
GPT-5.4	128 (25.25%)	$869.80	$6.80
GPT-6 Astra	231 (45.56%)	$1,720.60	$7.45
Claude Opus 5.5	241 (47.53%)	$1,880.77	$7.80
GPT-5.6 Sol	82 (16.17%)	$800.00	$9.76
Claude Opus 5	241 (47.53%)	$3,206.00	$13.30
Claude Sonnet 4.6	102 (20.12%)	$1,587.60	$15.56
GPT-5.2	44 (8.68%)	$878.00	$19.95
Kimi-K3	68 (13.41%)	$1,406.40	$20.68
Qwen3.8-27B	38 (7.50%)	$925.80	$24.36

再来看一致性可靠性的排名。 GPT-5.4 凭借 6.80 美元位列单价最便宜，但仅有 128 项任务能达到完全可靠的门槛。GPT-6 Astra 在 7.45 美元的单价下达到了 231 项，而 Claude Opus 5.5 则在 7.80 美元单价下拿到了并列最高的 241 项。

这三款模型互不占优：每增加一个高可靠任务，边际成本都会上升。Claude Opus 5 虽然同样达成了 241 项任务全胜，但成本高达 13.30 美元，因而被 Opus 5.5 全面碾压。GPT-5.6 Sol 虽然在单次成功成本上最低（0.127 美元），但其高可靠任务成本却上升到了 9.76 美元。这表明：获取一次正确答案最便宜的方法，绝非获得一个可靠答案的最经济途径。

5. 失败特征分类

我们为每一次失败的执行轨迹分配了一个确定性的诊断特征标签，得出的核心结论极具实操参考价值：大约五分之四的失败是由工具处理不当引起的，而非推理逻辑问题。根据我们论文表 5中的消融研究：

失败特征	占失败总数的比例
工具使用错误 (Tool usage)	79.9%
状态更新错误 (Wrong state updates)	10.3%
用户问题解决不完整 (Incomplete user resolutions)	7.0%
未产生状态变更行为 (No state-changing action)	2.9%

这些是各模型占比的未加权平均值与可观测标签，并非唯一原因

Microsoft ThinkingBox grades AI agents on the records they leave behind, not the sentences they generate, and then asks whether they can do it twenty times in a row. It is now available through Hugging Face.

Figure 1: ThinkingBox runs an agent against isolated MCP tool sessions, then grades the terminal backend state and side effects it leaves behind. From ourThinkingBox paper.

This is a joint blog by Microsoft and Hugging Face, special thanks to Tommy Guy (founder at Enderis AI, previously Microsoft), Sergio Paniego from Hugging Face and our former interns Zhuochun Li (University of Pittsburgh), Ali Keramati (UC Irvine), Youngmin Ko (Northwestern) for co-authoring/reviewing efforts.

A customer writes in. Her $745 kitchen appliance has been stuck in a courier "exception" at a Nashville distribution center, fifteen days past its estimated delivery date.

The AI agent does careful work. Nine tool calls: it pulls the order, checks tracking, looks up her customer profile, searches the refund policy twice, confirms no ticket exists, opens one, documents the timeline, and reads the policy correctly; her account segment genuinely does not qualify for late-delivery compensation.

Then it closes the ticket as resolved and replies “ Since your query is resolved, is there anything I may assist you with? ”

Two things are wrong. The carrier exception is still open, so the required end state was on hold , pending resolution. And the customer never got a real answer to what she actually asked.

An AI grader checking tool calls would see nine well-formed ones. The grader checking whether the agent wrote to the database would see that too. The database is what disagrees.

That gap is what ThinkingBox measures. Across 507 stateful business workflows, each run 20 times against various LLM models, it grades agents on terminal backend state and side effects. This post covers what we found, what consistency costs, and how to run the benchmark yourself through OpenEnv.

You can run this one yourself: the example above is adapted from a benchmark task sandbox_external_retail_group1.py:test_case_ST003_006, and the executable check that fails is a single field: the ticket's status is solved where the required end state is hold. The full trace is in Appendix D.4, Case 3 of our paper.

Contents

1. A tool call is not an outcome
2. One success is not reliability
3. Can you depend on the model behind your agent?
4. What consistency costs
5. Failure signatures
6. How it works
7. Run it yourself
8. Where this goes next

Want to try it before reading the results? Skip to sectionRun it yourself.

A tool call is not an outcome

Final responses and valid tool calls are only proxies. An agent can sound correct while leaving the wrong value, changing the wrong record, or creating an extra side effect. Only the records it leaves behind settle the question.

The gap is substantial. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error. Executable checks nevertheless found wrong field values in 77.61% of them, unintended extra effects in 43.30%, and missing required effects in 25.36%. Those state-check findings overlap.

A trajectory is a claim. Database state is the evidence. Repetition is the trust test.

One success is not reliability

An agent that processes a refund correctly once and mishandles it the next four times is not a working refund agent. So every task runs 20 independent times , each from an identical clean backend, and we report three different things:

Table 1: The three numbers we report, and the question each one answers.

Metric	What it measures	What it answers
pass@1	Share of all attempts that succeeded	How does it usually do?
pass@20	Share of tasks solved at least once in 20 tries	Can it ever do this? Breadth.
Observed 20/20	Tasks that actually passed all 20 recorded attempts	Can it always be correct?

We use observed 20/20 in this blog post as the literal count of how many of the 507 tasks passed 20 out of 20. No estimator, no smoothing.

Starting with the familiar view. The table below reports pass@1, the single-attempt score estimate, broken out by domain. This is the number most leaderboards publish, and on its own it reads like an ordinary capability ranking.

Table 2: ThinkingBox-Bench pass@1 (%) by domain. Each model is evaluated on every task for 20 repeated trials. Bold marks the group leader; underline marks the runner-up. The standard errors for the single attempt score estimates are provided inTable 4 in our ThinkingBox paper.

Model	Retail (98)	Auto insurance (100)	Travel (104)	Neobank (104)	Consulting (101)	Overall, task-weighted (507)
Proprietary models
Claude Opus 5.5	80.97	68.40	54.28	71.25	61.58	67.16
Claude Opus 5	80.71	65.80	49.95	70.62	66.19	66.50
GPT-5.4	76.33	62.65	68.12	65.34	54.60	65.36
GPT-5.6 Sol	67.65	65.30	60.34	59.09	57.52	61.91
Claude Sonnet 4.6	72.35	54.40	58.94	56.39	54.31	59.19
GPT-6 Astra	71.73	46.55	55.87	60.87	56.83	58.31
GPT-5.2	70.20	22.40	53.70	51.15	34.06	46.28
Claude Opus 4.6	68.62	8.30	21.11	35.67	27.82	32.09
o3-pro	37.70	2.95	17.31	24.28	14.60	19.31
Grok-4.3	43.93	2.60	15.14	1.78	9.55	14.38
Open-weight models
Kimi-K3	82.24	50.80	61.83	41.35	51.63	57.37
Qwen3.8-27B	64.03	47.85	53.41	47.88	45.69	51.70
DeepSeek-V4-Pro	68.21	29.65	43.13	44.86	31.04	43.26
Kimi-K2.6	53.72	24.50	39.52	33.65	37.33	37.66
GLM-5.1	58.67	25.70	35.43	13.27	34.06	33.19
Qwen3.6-27B	43.11	29.00	46.39	27.84	18.37	32.94
Qwen3.5-9B	19.90	0.70	4.71	1.15	2.33	5.65
Mistral-Large-3	11.28	1.30	8.99	1.15	0.74	4.66

Claude Opus 5.5 leads overall at 67.16%, two-thirds of a point above Claude Opus 5. Kimi-K3 is the strongest open-weights model , within a point of GPT-6-Astra. Domain matters just as much : Claude Opus 4.6 scores 68.62% on retail but 8.30% on auto insurance.

One good run tells you a model can do the work. It does not tell you whether it will do it again. So run every task 20 times and ask how much of that score survives.

Figure 2: How much of each model's single-attempt score survives 20 repeats.

Only three hold on to most of their pass@1 scores: GPT-6 Astra retains 78% of its single-attempt rate, and Claude Opus 5.5 and Claude Opus 5 each retain 71%. At the other end, GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8%.

The gap between what a model can do once and what it does every time is the whole story.

Can you depend on the model behind your agent?

Figure 3: Breadth and consistency pull apart. Twelve of the eighteen models are shown; six below 33% pass@1 are omitted for legibility.

Kimi-K3 has the broadest coverage of any model we tested. It solves 93.89% of the benchmark at least once: 476 of 507 tasks. Only 31 tasks defeat it entirely, the lowest count in the field. On retail workflows it leads outright at 82.24% pass@1, ahead of every proprietary model.

Kimi-K3 is also among the least consistent. Just 68 of 507 tasks, 13.41%, succeed in all 20 attempts.

Claude Opus 5 inverts this. It solves fewer tasks at least once (79.09%; 106 defeat it entirely) but completes 47.53% of the benchmark on every single attempt.

A newer model does not fix this. Claude Opus 5.5 scores higher than Claude Opus 5 on every-attempt average, 67.16% against 66.50%, and solves more tasks at least once. It passes exactly the same number of tasks on all 20 attempts: 241. Half a point of headline accuracy bought no additional dependability at all.

• Kimi-K3 solves 75 more tasks at least once than Opus 5.
• Opus 5 solves 173 more tasks consistently than Kimi-K3.

If you are choosing a model for work that touches real records, pass@20 is the wrong column to look at.

What consistency costs

Capability comparisons usually stop at the score. For anyone deploying, the relevant question is what a successful unit of work costs. We measure that as cost per successful task attempt. We say task attempt because every benchmark task is run repeatedly and cost is incurred per attempt, so pass@1 is the matching quality denominator.

We took each model's recorded token usage from its full 507 × 20 campaign and priced it at undiscounted list rates available on OpenRouter+, reversing promotional discounts and excluding endpoints that declare quantization. Input, output and cache rates all come from one provider endpoint per model.

Then we divided one run's cost by the number of attempts that succeeded :

Cost per successful task attempt = estimated cost for 507 attempts, one per task ÷ (507 × pass@1)

This is a comparative efficiency index, not an invoice, and not the price of serving one production request. It also prices single successes, not consistency. We price consistency next.

Example: GPT-5.4 costs $43.49 for 507 attempts (one attempt per task) and has 65.36% pass@1, so $43.49 ÷ (507 × 0.6536) = $0.131 per successful task attempt.

Pareto cost frontier

A model is on the frontier if no other model is both no more expensive and at least as accurate. Three models qualify; every other model is dominated on at least one axis.

Figure 4: Cost per successful task attempt against pass@1. Ringed dots are pareto cost frontier models.

The frontier has three steps. GPT-5.6 Sol has the lowest cost per success at $0.127; GPT-5.4 raises pass@1 by 3.45 percentage points for $0.004 more per success; Claude Opus 5.5 adds another 1.80 points at $0.276 per success. Each of the three remains on the cost frontier line because no cheaper model matches its pass@1.

Claude Opus 5 is the clearest case: at $0.475 per successful attempt and 66.50% pass@1, it is both more expensive and less accurate than Claude Opus 5.5 at $0.276 and 67.16%.

Now price consistency

Cost per success rewards a model that is cheap and often right. It does not reward a model that is right every time. So we also compute cost per dependable task: the cost of the full 20-run campaign divided by the number of tasks the model passed on all 20 attempts.

Cost per dependable task = estimated cost of 20 runs of 507 attempts ÷ tasks passing 20/20

Example: GPT-6-Astra costs 20 × $86.03 = $1,720.60 for the campaign and passes 231 tasks on every attempt, so $1,720.60 ÷ 231 = $7.45 per dependable task.

Table 3: The nine lowest costs per dependable task among models with at least one observed 20/20 task, sorted low to high. Estimated $, not actual cloud bills.

Model	Tasks passing 20/20	Est. cost, 20 runs	Cost per dependable task
GPT-5.4	128 (25.25%)	$869.80	$6.80
GPT-6 Astra	231 (45.56%)	$1,720.60	$7.45
Claude Opus 5.5	241 (47.53%)	$1,880.77	$7.80
GPT-5.6 Sol	82 (16.17%)	$800.00	$9.76
Claude Opus 5	241 (47.53%)	$3,206.00	$13.30
Claude Sonnet 4.6	102 (20.12%)	$1,587.60	$15.56
GPT-5.2	44 (8.68%)	$878.00	$19.95
Kimi-K3	68 (13.41%)	$1,406.40	$20.68
Qwen3.8-27B	38 (7.50%)	$925.80	$24.36

Now rank by consistency. GPT-5.4 is the cheapest at $6.80, though only 128 tasks meet the bar. GPT-6 Astra reaches 231 at $7.45, and Claude Opus 5.5 the joint-highest 241 at $7.80.

None of the three dominates the others: each additional dependable task costs more. Claude Opus 5 also passes 241, but at $13.30, so Opus 5.5 dominates it outright. GPT-5.6 Sol, the cheapest per single success at $0.127, costs $9.76 per dependable task. The cheapest way to get a right answer is not the cheapest way to get a dependable one.

Failure signatures

We assign each failed trace one deterministic diagnostic signature, and the headline is actionable: roughly four in five failures are tool handling, not reasoning. Across an ablation study in Table 5 of our paper:

Failure signature	Share of failures
Tool usage	79.9%
Wrong state updates	10.3%
Incomplete user resolutions	7.0%
No state-changing action	2.9%

These are unweighted averages of per-model shares and observable labels, not unique causal explanations.

The practical pattern is simple: agents usually get far enough to attempt the workflow, then fail to recover from tool errors, failed preconditions, or empty lookups. That is a retry and error-recovery problem before it is a model problem.

Difficulty also changes by domain: across the models listed in Table 2 above, retail averages 59.52% pass@1 while auto insurance averages 33.83%.

What to do about it. Treat the 20/20 rate as a design input, not a verdict. The same signal the benchmark grades on is available in production: check the terminal state before you commit, not the model's summary of it.

Classify tool and system errors so retries target the recoverable ones. Cut the tool surface to what the workflow needs. And require human approval on the changes you cannot cheaply reverse. We have not measured the lift from any of these on this benchmark, which is exactly the kind of thing the environment now makes testable.

How it works

ThinkingBox is the agent sandbox, while ThinkingBox-Bench is a dataset benchmark to evaluate agents. The diagram at the top of this post shows the loop; here is what each part does.

Figure 5: The sandbox loop from panel A in Figure 1 above: isolated tool session, terminal database state, side effects, executable judges.

Each task defines a starting backend state, a user goal, the available MCP tools, the domain policy, and executable checks over the terminal state. A simulated user holds private context (a booking reference, a preference, or a date of birth) and releases it only when asked.

Every attempt gets an isolated MCP session with freshly initialized state. Two attempts of the same task never share a database row or cached tool state, which is what makes 20-trial comparison meaningful.

At the end, a side-effect extractor derives what actually changed, and deterministic judges compare it against the required end state, accepting any trajectory that produces the right outcome while rejecting wrong, missing or extra effects. For requirements with no clean database value ("did the agent disclose this is not guaranteed?"), a narrow binary rubric question handles the semantics. 477 of the 507 tasks are graded on state alone; 30 add response rubrics.

The trust boundary: the model sees tasks, dialogue and tool schemas. Golden state, assertions, grading internals and credentials stay on the evaluator side.

Run it yourself

ThinkingBox is now on Hugging Face, both the harness and the dataset. ThinkingBox-Bench now sits behind the OpenEnv interface, and each finished episode returns a binary pass/fail reward. The released adapter is designed for evaluation; separate, non-benchmark scenarios can use the same interface in training workflows.

Before you start

Tested on Linux and WSL, with Python 3.11+, uv and Docker. You also need a thinkingbox-data checkout at the pinned release and model endpoints for the agent, simulated user and judge. One endpoint can serve all three roles, which is the simplest way to start. The OpenEnv image starts only the OpenEnv API ; everything else you run yourself.

Install

git clone https://github.com/huggingface/OpenEnv
cd OpenEnv
uv sync --project envs/thinkingbox_env --frozen

git clone https://github.com/microsoft/thinkingbox-data
git -C thinkingbox-data checkout thinkingbox-bench-v1.0

uv tool install "thinkingbox @ git+https://github.com/microsoft/thinkingbox"

Start Typesense

In a second terminal , start Typesense 30.1 and wait for its health check:

mkdir -p .typesense-data
docker run --rm -d --name thinkingbox-typesense \
  -p 8108:8108 \
  -v "$PWD/.typesense-data:/data" \
  typesense/typesense:30.1 \
  --data-dir /data --api-key=Fake --enable-cors
until curl -fsS http://127.0.0.1:8108/health; do sleep 1; done

Start the MCP servers

In a third terminal , start the Session Proxy and MCP servers.

cd OpenEnv
tb mcp-start --host 127.0.0.1 --port 7111 \
  --servers "$PWD/thinkingbox-data/servers/servers.yaml"
curl -fsS http://127.0.0.1:7111/health

Start the OpenEnv server

Back in the first terminal, start the OpenEnv server against a ThinkingBox YAML config naming your three models (config guide):

OPENENV_TB_CONFIG="$PWD/thinkingbox.yaml" \
uv run --project envs/thinkingbox_env --frozen server

Check readiness

Gate on readiness before running anything. It returns 503 until its observable data, configuration and Session Proxy checks pass. It cannot observe Typesense or live-probe every model endpoint, so confirm those separately:

curl -sS http://127.0.0.1:8000/ready

Score an episode

Now score a real episode. example_usage.py only resets and lists tools; for agent actions, effects and assertions use the packaged evaluator:

echo "- sandbox_external_retail_group1.py:test_case_ST002_001" > one_task.yaml
uv run --project envs/thinkingbox_env thinkingbox-eval \
  one_task.yaml \
  --config "$PWD/thinkingbox.yaml" \
  --output results.jsonl \
  --errors-output errors.jsonl \
  --repeat 1 --message-timeout 1800

The OpenEnv adapter writes operational failures to an errors sidecar so they can be rerun rather than silently mixed with model outcomes. A canonical result must resolve or explicitly account for those attempts; we counted system errors as unsuccessful trials.

Runs are gated on a pinned framework commit, a pinned data release and a bundle hash, so a canonical result is verifiable rather than asserted.

Where this goes next

The useful part of this work is not our pass@1 leaderboard. It is the environment.

If you are evaluating an agent that touches real records:

1. Inspect a failure. Find a run that terminated cleanly and still failed, and look at what actually changed in the database. It reframes what your own evals measure.
2. Reproduce one task through OpenEnv with your own model.
3. Report a repeat metric, and define it. Whatever k your use case justifies; say whether you are reporting best-of-k or every-of-k, and how you computed it.

Further details can be found in the following links:

ThinkingBox code is MIT-licensed; the benchmark data is CDLA-Permissive-2.0; the OpenEnv environment ships under OpenEnv's BSD-3-Clause.

Disclaimer: every task in the public benchmark is a synthetic reconstruction. The workflows and policies are modeled on real AI agentic enterprise patterns; the customers are not real.

ThinkingBox and ThinkingBox-Bench is built by the Microsoft Copilot Studio team in partnership with Toloka with collaborators from the University of Pittsburgh, Northwestern University, Columbia University and UC Irvine who interned at Microsoft. Questions are welcome in the comments section below or ongithub

• OpenRouter cost snapshot taken on Sept 20th 2026; Opus 5.5 pricing as per the Anthropic site.