将数据库判定转化为 RL 奖励：利用 ThinkingBox 对 Agent 进行后训练

本文是我们第一篇博客文章《Agent 声称已完成任务，但数据库表示反对》的后续。ThinkingBox-Training 现已在 GitHub 开源。

我们构建 ThinkingBox 的初衷，是通过将最终数据库状态与预期结果进行比对来评估 Agent 的工作成效。随后，我们将这种可执行信号作为奖励，应用于 ThinkingBox-Bench 之外的训练任务中。对于 Qwen3.8-27B 模型，在仅进行了 50 步 GRPO 训练后，其平均成功率就从 51.70% 攀升至 60.78%，在全部 20 次尝试中均正确完成的任务数也从 38 个激增至 107 个。

图 1. ThinkingBox 将对终端数据库状态的可执行检查转化为奖励，用于在 ThinkingBox-Bench 之外的任务上对 Agent 进行后训练。

这是微软与 Hugging Face 的联合博客，特别鸣谢微软的 Mirco Milletari 与 Xiang Jiang、Hugging Face 的 Sergio Paniego，以及我们曾经的实习生 Zhuochun Li（匹兹堡大学）在共同撰写与审阅方面做出的努力。

我们构建 ThinkingBox，是为了揪出那些明明没完成却自称已完成的 Agent。

ThinkingBox-Bench（我们的基准数据集）包含 507 个任务，每个任务都会为 Agent 提供一个真实的后端系统和一组工具，允许其自主运行，随后检查数据库中到底发生了哪些实际变更。在第一篇文章中，我们抓到了一个模型将客户的工单标记为已解决（resolved），然而背后的物流异常记录仍处于开启状态。它进行了 9 次看似专业合理的工具调用，最终却留下了一条错误的数据库记录。

接着我们意识到了自己所构建系统的潜在价值。对于绝大多数任务，针对最终数据库状态的可执行检查可以直接返回 1 或 0。这不仅仅是一个评测器（grader），更可以直接充当奖励函数（reward function）。

因此，我们从基准测试之外的 ThinkingBox 格式任务中构建了一个训练池，并将它们的可执行结果作为奖励。其中的关键挑战在于选出哪些任务真正适合用于 Agent 训练：大多数任务对于你眼前的模型而言，要么早已被轻松解决，要么毫无希望攻克，而这两类任务都无法让模型学到任何东西。

难度是相对于策略而言的（Difficulty is policy-relative）：一个任务对于当前模型来说是困难的，但只要该模型学会了它，它就不再困难。

我们将此称为离线策略相对难度（offline policy-relative difficulty）。在训练开始前，我们使用基模型对每个任务的成功概率进行了一次估计，并在整个训练过程中保持所选训练池固定不变。

训练后发生了什么

我们利用离线策略相对难度构建了训练池，随后使用 GRPO 对 Qwen3.8-27B 进行了 50 个 step 的微调。回顾一下，我们对 507 个基准任务中的每一个都进行了 20 次评估。pass@1 衡量所有 10,140 次尝试的平均成功率；pass@20 衡量至少被成功解决一次的任务比例；Observed 20/20 则衡量在记录的全部尝试中每一次都通过的任务比例。

表 1. Qwen3.8-27B 在离线选定的训练池上进行 50 步 GRPO 前后的表现结果。

指标	Base 基模型		RL 微调后	变化
Observed 20/20	38 (7.50%)	→	107 (21.10%)	+69 个任务
pass@1（成功尝试占比）	5,242 (51.70%)	→	6,163 (60.78%)	+9.08 个百分点
pass@20（至少解决一次的任务）	453 (89.35%)	→	446 (87.97%)	−7 个任务

训练显著提高了可重复的稳定性成功率，但至少解决一次的任务覆盖面略有收缩。本文余下部分将详细解释我们如何选择练习任务、构建奖励闭环，以及如何衡量这种权衡取舍。

这种提升在单个任务上是怎样的体现？ 例如一位旅客请求降低房间规格，但公司政策要求必须先获得经理批准。在训练之前，该 Agent 在 20 次尝试中有 15 次在未经批准的情况下擅自更改了预订并退款。而在经过强化学习微调后，它在全部 20 次尝试中均妥善保留了预订和付款状态。

图 2. 一个企业酒店预订任务，每个模型尝试 20 次。基模型 Qwen3.8-27B 在 15 次尝试中均在未获得必需的经理批准的情况下更改了预订。强化学习微调后的模型在全部 20 次中均保留了预订与支付状态；其中 19 次的所有预期记录均完全吻合。

目录

1. 60 秒读懂基准测试的发现
2. 评测结果：可靠性维度的跃升
3. 收益主要落在何处
4. 可靠性的成本代价
5. 训练循环架构
6. 离线策略相对难度
7. 怎样的表现才能获得奖励
8. 运行训练技术栈
9. 下一步走向何方

赶时间？核心成果已在上方列出。具体实现请参阅「运行训练技术栈」章节。

60 秒读懂基准测试的发现

ThinkingBox-Bench 根据 Agent 遗留在数据库中的记录来进行评分。其 507 个任务中的每一个都会启动一个真实后端，为 Agent 提供一套 MCP 工具，并配备一个拥有私有上下文的模拟 LLM 用户，最后对终端数据库状态运行可执行检查。这些检查针对的是需要保持正确的后端记录，而非特定的工具调用序列。这激励了模型去探索可能全新的解决路径，而不是机械地去“重现”那些由人类背书但可能次优（即便正确）的旧方案。

在第一篇博客中，我们让 18 个模型在该基准上各自运行了 20 次，并揭示了被表面综合准确率掩盖的巨大鸿沟。Kimi-K3 解决了 93.89% 的任务_至少一次_，但在全部 507 个任务中，只有 68 个（13.41%）能够在全部 20 次记录的尝试中都完成。Claude Opus 5 则相反：覆盖广度更窄，但一致性要高得多。

而且更大或更优的模型并没有消除这一差距。Claude Opus 5.5 在 pass@1 上比 Claude Opus 5 高出 0.66 个百分点，然而在全部 20 次尝试中均通过的任务数量却完全相同，均为 241 个。表面准确率的微弱提升并没有带来额外的可靠性。

如果你正在挑选一款用于处理生产级数据库记录的模型，绝不能孤立地看待 pass@1。Observed 20/20 反映了哪些任务能够在每一次尝试中都保持成功，这更贴近生产环境对可靠性的硬性要求。

终端状态的可执行判定，同时也充当了训练奖励。在拆解训练循环之前，先来看看在封闭基准测试集上发生了怎样的变化。

评测结果：可靠性维度的跃升

在完成 Qwen3.8-27B 的训练后，我们在包含 507 个从未参与过训练的任务基准上对所选的第 50 步 checkpoint 进行了评估。这 507 个任务属于分布外（OOD）测试集：它们的业务领域、数据库结构以及绝大部分 MCP 工具在强化学习训练期间都从未出现过。

表 2. 各领域的 ThinkingBox-Bench pass@1 (%)，数据选自我们第一篇博客的表 2 中的代表性模型以及本次微调后的模型。Observed 20/20 数据基于全部 507 个基准任务。每个模型均在所有 507 个任务上进行 20 次重复试验。粗体代表该列最优，下划线代表次优。

模型	零售	汽车保险	差旅	新型数字银行	咨询服务	综合总体	Observed 20/20
Claude Opus 5.5	80.97	68.40	54.28	71.25	61.58	67.16	241 (47.53%)
Claude Opus 5	80.71	65.80	49.95	70.62	66.19	66.50	241 (47.53%)
GPT-5.4	76.33	62.65	68.12	65.34	54.60	65.36	128 (25.25%)
GPT-5.6 Sol	67.65	65.30	60.34	59.09	57.52	61.91	82 (16.17%)
Qwen3.8-27B RLFT	67.91	56.45	62.69	62.40	54.51	60.78	107 (21.10%)
Claude Sonnet 4.6	72.35	54.40	58.94	56.39	54.31	59.19	102 (20.12%)
GPT-6 Astra	71.73	46.55	55.87	60.87	56.83	58.31	231 (45.56%)
Kimi-K3	82.24	50.80	61.83	41.35	51.63	57.37	68 (13.41%)
Qwen3.8-27B	64.03	47.85	53.41	47.88	45.69	51.70	38 (7.50%)

最引人瞩目的结果是表 1 中的 Observed 20/20 这一行。 一个原本在 ThinkingBox-Bench 上仅有 38 个任务（7.5%）能在每次尝试中都完成的 27B 模型，现在能在每次尝试中完成 107 个任务（21.1%），提升了近三倍。作为对比，Kimi-K3 的 Observed 20/20 为 68 个，GPT-5.4 为 128 个，GPT-6 Astra 为 231 个，而 Claude Opus 5.5 以 241 个位列榜首。

平均成功率与可重复稳定性给出了截然不同的排序。 在表 2 中我们注意到，GPT-5.6 Sol 在 pass@1 上的得分略高于经过 RL 微调的 Qwen3.8-27B（61.91% 对比 60.78%），但在记录的全部 20 次尝试中均通过的任务数上却更少：82 对比 107。Claude Sonnet 4.6 在 pass@1 上的得分更低，为 59.19%，然而每次都通过的任务数却几乎相当，达到 102 个。任何单一指标都无法完整反映全局。

观察到的模式与概率锐化（sharpening）一致：可重复的成功率上升，而“至少解决一次”的广度覆盖面略有下降。这与先前的研究工作是一致的。一种解释是概率分布朝向能够反复奏效的轨迹汇聚，而一些低概率路径则随之消失。

总体而言，模型并没有变得更宽泛，而是变得更具可重复性。接下来的问题是，这些提升具体落在了哪些地方。

收益主要落在何处

图 3. 各领域的基模型分数、RLFT 分数以及百分点变化。绿色单元格表示增长，红色单元格表示下降；蓝色深浅反映每个指标内的绝对分值水平。

Pass@1 和 Observed 20/20 在全部五个领域中均有所上升。Pass@20 的表现喜忧参半：新型数字银行和咨询服务领域有所上升，但在零售、差旅和汽车保险领域出现了回落。汽车保险领域的广度收缩幅度最大（−7.0 个百分点），而零售领域在 Observed 20/20 上的增幅最为可观（+24.5 个百分点）。

可靠性的成本代价

第一篇文章通过一个简单的数学公式来衡量一致性的成本：将完整 20 轮运行评测活动的预估总成本除以模型每一次均成功通过的任务数。

单次可靠任务成本 = 507 × 20 评测活动的预估成本 ÷ 达到 20/20 通过的任务数

将相同的成本计算公式应用于微调 checkpoint 实际测得的 Token 消耗：

表 3. Qwen3.8-27B 基模型与 RL 微调模型的预估评估成本与可靠性。单次成功尝试的成本等于总预估活动成本（507 × 20）除以成功尝试的总次数。

指标	Base 基模型		RL 微调后	变化
活动成本（20 次运行）	$925.80	→	$855.58	−7.6%
单次成功尝试成本	$0.177	→	$0.139	−21.4%
达到 20/20 通过的任务数	38	→	107	达到 2.8 倍
单次可靠任务成本	$24.36	→	$8.00	−67.2%

达到每个可靠任务 8.00 美元时，该微调模型已经非常接近对照组中的 GPT-5.4（$6.80）、GPT-6 Astra（$7.45）以及 Claude Opus 5.5（$7.80）。

单次尝试视角的成本分析

图 4 展示了_单次成功尝试成本_与 pass@1 之间的对比分布。

图 4. 金色箭头标出了 Qwen3.8-27B 基模型到 RLFT（菱形）的帕累托成本位移：pass@1 从 51.70% 提升至 60.78%，同时单次成功尝试的预估成本从 $0.177 降至 $0.139。带圆环的圆点标示出帕累托成本前沿。

这一跃迁使得微调后的 checkpoint 性能与成本非常逼近 GPT-5.6 Sol，不过 GPT-5.6 Sol 的成本依然更低且 pass@1 略高一些。我们论文中测算的另外四款模型落在了绘图区域之外。

上述美元数值属于相对维度的 Serving 成本指数，并非发票或实际云账单。部署微调后的 checkpoint 与基模型的推理成本是相同的。该计算包含了模型推理成本，未计入训练运行以及筛选训练数据集时的策略探测成本。

这些结果呈现了具体指标的变化。接下来的章节将解析训练循环，以及为何选对任务至关重要。

训练循环架构

图 5. 六个阶段的整体流程：候选任务输入，前三个阶段离线执行一次，后三个阶段在训练运行的每个组中不断重复。训练后的 checkpoint 最终在 ThinkingBox-Bench 上进行评测。

筛选流程针对基策略模型（在本案例中即 Qwen3.8 27B）在离线状态下仅运行一次；随后训练全程使用这一精选好的固定任务池。图 5 中的琥珀色虚线箭头代表了我们并未采用的路径：即在训练运行期间动态重新探测难度。

每次 Rollout 都是一个完整的 Episode，而非单纯的文本补全。它会分配到一个全新的后端环境、一个独立的 MCP 会话，以及由 GPT-5 Chat 充当的模拟用户。大多数奖励均来自确定性的终端状态检查；少数狭窄的评分细则要求可调用已配置的裁判员模型（GPT-5 Chat）。

训练与评估对 Qwen 输出的解析逻辑必须保持严格一致。 这包括思维链推理提取、工具调用、参数类型校验、渲染处理以及隔离的任务状态管理。否则，奖励信号评判的 Agent 将与最终接受评估的 Agent 发生偏差。共享的解释代码位于 trainer/core/tb_interpret.py。

This is the follow up to our first blog post, The Agent Said It Was Done. The Database Disagreed. ThinkingBox-Training is now available on GitHub.

We built ThinkingBox to evaluate an agent's work by comparing the final database state with the expected outcome. We then used the same kind of executable signal as a reward on training tasks outside ThinkingBox-Bench. For Qwen3.8-27B, average success rose from 51.70% to 60.78%, while tasks completed correctly on all twenty attempts rose from 38 to 107 with 50 GRPO steps.

Figure 1. ThinkingBox turns executable checks of terminal database state into rewards for post-training agents on tasks outside ThinkingBox-Bench.

This is a joint blog by Microsoft and Hugging Face, special thanks to Mirco Milletari and Xiang Jiang from Microsoft, Sergio Paniego from Hugging Face and our former intern Zhuochun Li (University of Pittsburgh) for co-authoring/reviewing efforts.

We built ThinkingBox to catch agents that say they are done when they are not.

Each of its 507 tasks in ThinkingBox-Bench (our benchmark dataset) hands an agent a real backend and a set of tools, lets it work, then checks what really changed in the database. In the first post, we caught a model closing a customer's ticket as resolved while the carrier exception behind it was still open. Nine competent-looking tool calls, one wrong database row.

Then we noticed what we had built. For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function. +

So we constructed a training pool from ThinkingBox-format tasks outside the benchmark and used their executable outcomes as rewards. The catch was choosing which tasks are suitable for agent training: most tasks are either already solved or hopeless for the model in front of you, and both teach it nothing.

Difficulty is policy-relative: a task is hard for the current model only until that model learns it.

We call this offline policy-relative difficulty. We estimated each task's success probability once using the base policy before training and kept the selected training pool fixed throughout the run.

What happened after training

We used offline policy-relative difficulty to build the training pool, then fine-tuned Qwen3.8-27B with GRPO for 50 steps. To recap, we evaluate each of the 507 benchmark tasks 20 times. pass@1 measures average success across all 10,140 attempts. pass@20 measures the share of tasks solved at least once. Observed 20/20 measures the share of tasks that pass every recorded attempt.

Table 1. Qwen3.8-27B results before and after 50 steps of GRPO on the offline-selected training pool.

Metric	Base		RL fine-tuned	Change
Observed 20/20	38 (7.50%)	→	107 (21.10%)	+69 tasks
pass@1 (successful attempts)	5,242 (51.70%)	→	6,163 (60.78%)	+9.08 pt
pass@20 (tasks solved at least once)	453 (89.35%)	→	446 (87.97%)	−7 tasks

Training sharply increased repeatable success, with a small decline in at-least-once coverage. The rest of the post explains how we selected practice tasks, built the reward loop, and measured that tradeoff.

What does that improvement look like on one task? A traveler requests a room downgrade, but company policy requires manager approval first. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After RL fine-tuning, it preserved the booking and payment state in all 20.

Figure 2. One corporate hotel-booking task, 20 attempts per model. The base Qwen3.8-27B model changed the booking without required manager approval in 15 attempts. The RL fine-tuned model preserved booking and payment state in all 20; all expected records matched in 19.

Contents

1. What the benchmark found, in sixty seconds
2. Results: the dependability column moved
3. Where the gains landed
4. What dependability costs
5. The training loop
6. Offline policy-relative difficulty
7. What earns a reward
8. Run the training stack
9. Where this goes next

In a hurry? The headline result is above. The implementation is inRun the training stack.

What the benchmark found, in sixty seconds

ThinkingBox-Bench grades agents on the database records they leave behind. Each of its 507 tasks starts a real backend, hands the agent a set of MCP tools and a simulated LLM user with private context, and at the end runs executable checks against the terminal database state. Checks are against the backend record that needs to be correct, not the specific tool calls. This incentivizes the model to discover possibly new solution paths rather than “rediscover” the human-backed solution, that might be suboptimal albeit correct.

In our first post we ran eighteen models through that benchmark twenty times each and reported a gap that headline accuracy hides. Kimi-K3 solved 93.89% of tasks at least once but completed only 68 of 507 tasks (13.41%) on all twenty recorded attempts. Claude Opus 5 inverted it: narrower coverage, far more consistency.

And bigger or better model did not close it. Claude Opus 5.5 scored 0.66 percentage points higher than Claude Opus 5 on pass@1, yet passed exactly the same number of tasks, 241 , on all twenty attempts. That gain in headline accuracy bought zero additional dependability.

If you are choosing a model for work that touches production database records, pass@1 should not be read alone. Observed 20/20 shows which tasks succeed on every attempt, which is closer to what production reliability requires.

The executable terminal-state verdict was also the training reward. Before unpacking the training loop, here is what changed on the sealed benchmark.

Results: the dependability column moved

After training Qwen3.8-27B, we evaluated the selected step-50 checkpoint on the 507-task benchmark it had never trained on. The 507 tasks represent an out-of-distribution test set: their domains, databases, and most MCP tools were not encountered during RL training.

Table 2. ThinkingBox-Bench pass@1 (%) by domain, for selected models fromour first blog's Table 2 and our fine-tuned model for different domains and overall. Observed 20/20 numbers are over the entire benchmark with 507 tasks. Each model is evaluated on all 507 tasks for 20 repeated trials. Bold marks the column leader; underline marks the runner-up.

Model	Retail	Auto insurance	Travel	Neobank	Consulting	Overall	Observed 20/20
Claude Opus 5.5	80.97	68.40	54.28	71.25	61.58	67.16	241 (47.53%)
Claude Opus 5	80.71	65.80	49.95	70.62	66.19	66.50	241 (47.53%)
GPT-5.4	76.33	62.65	68.12	65.34	54.60	65.36	128 (25.25%)
GPT-5.6 Sol	67.65	65.30	60.34	59.09	57.52	61.91	82 (16.17%)
Qwen3.8-27B RLFT	67.91	56.45	62.69	62.40	54.51	60.78	107 (21.10%)
Claude Sonnet 4.6	72.35	54.40	58.94	56.39	54.31	59.19	102 (20.12%)
GPT-6 Astra	71.73	46.55	55.87	60.87	56.83	58.31	231 (45.56%)
Kimi-K3	82.24	50.80	61.83	41.35	51.63	57.37	68 (13.41%)
Qwen3.8-27B	64.03	47.85	53.41	47.88	45.69	51.70	38 (7.50%)

The headline result is the observed-20/20 row in Table 1. A 27B model that initially completed 38 (7.5%) of ThinkingBox-Bench on every attempt now completes 107 (21.1%) on every attempt, almost thrice. For comparison, observed 20/20 for Kimi-K3 is 68, GPT-5.4 with 128, GPT-6 Astra has 231, and Claude Opus 5.5 leads with 241.

Average success and repeatability produce different orderings. In Table 2, we notice that GPT-5.6 Sol scored slightly higher on pass@1 than the RL fine-tuned Qwen3.8-27B (61.91% versus 60.78%) but passed fewer tasks on all 20 recorded attempts: 82 versus 107. Claude Sonnet 4.6 scored lower on pass@1, at 59.19%, yet passed nearly the same number of tasks every time, at 102. Neither metric alone captures the full picture.

The observed pattern is consistent with sharpening: repeatable success rose while at-least-once coverage fell slightly. This is in-line with prior work. One explanation is that probability distribution moved toward trajectories that work repeatedly while some low-probability paths disappeared.

Overall, the model did not become broader. It became more repeatable. The next question is where those gains landed.

Where the gains landed

Figure 3. Domain-level base scores, RLFT scores, and percentage-point changes. Green change cells show gains and red cells show declines; blue shading indicates the score level within each metric.

Pass@1 and observed 20/20 rose in all five domains. Pass@20 was mixed: it increased in neobank and consulting, but declined in retail, travel, and auto insurance. Auto insurance had the largest breadth loss (−7.0 points), while retail had the largest observed-20/20 gain (+24.5 points).

What dependability costs

The first post priced consistency using simple math: divide the estimated cost of the full 20-run campaign by the number of tasks the model passed every single time.

Cost per dependable task = estimated cost of the 507 × 20 campaign ÷ tasks passing 20/20

Applying the same cost math to the fine-tuned checkpoint's measured token usage:

Table 3. Estimated evaluation cost and dependability for base and RL fine-tuned Qwen3.8-27B. Cost per successful attempt divides total estimated campaign cost (507 x 20) by the number of passing attempts.

Measure	Base		RL fine-tuned	Change
Campaign cost, 20 runs	$925.80	→	$855.58	−7.6%
Cost per successful attempt	$0.177	→	$0.139	−21.4%
Tasks passing 20/20	38	→	107	2.8× as many
Cost per dependable task	$24.36	→	$8.00	−67.2%

At $8.00 per dependable task, the fine-tuned model came close to GPT-5.4 ($6.80), GPT-6 Astra ($7.45), and Claude Opus 5.5 ($7.80) in the companion comparison.

The single-attempt view

Figure 4 prices cost per successful attempt against pass@1.

Figure 4. The golden arrow traces Qwen3.8-27B base pareto cost position to the RLFT (diamond): pass@1 rises from 51.70% to 60.78% while estimated cost per successful attempt falls from $0.177 to $0.139. Ringed circles mark the Pareto cost frontier.

The move places the fine-tuned checkpoint close to GPT-5.6 Sol, although GPT-5.6 Sol remains cheaper and slightly higher on pass@1. Four models priced in our paper fall outside the plotted range.

The dollar values are a comparative serving index, not invoice or cloud bills. Serving the fine-tuned checkpoint would be the same as the base model. The calculation includes model inference cost, and excludes training run and policy-probe cost for curating the training dataset.

These results tell us what moved. The next sections explain the training loop and why selecting the right tasks matters the most.

The training loop

Figure 5. The six stages, now in context: candidate tasks feed in, the first three run once offline, the last three repeat for every group in the training run. The trained checkpoint is evaluated against ThinkingBox-Bench.

Selection runs once, offline, against the base policy model, i.e in this case Qwen3.8 27B; training then uses that fixed pool that has been curated. The dashed amber arrow in Figure 5 marks the path we did not take: re-probing difficulty during the training run.

Every rollout is a complete episode, not a text completion. It gets a fresh backend, an isolated MCP session, and GPT-5 Chat as the simulated user. Most rewards come from deterministic terminal-state checks; a minority of narrow rubric requirements can invoke the configured judge (GPT-5 Chat).

Training and evaluation must interpret Qwen output identically. That includes reasoning extraction, tool calls, argument typing, rendering, and isolated task state. Otherwise the reward measures a different agent from the one evaluated. The shared interpretation code is in trainer/core/tb_interpret.py.

For the reference run, we ran full-parameter RL fine-tuned Qwen3.8-27B on 187 tasks using token-mean GRPO. Each update generated 18 groups of 8 rollouts, or 144 episodes, across 24 H100s. Training ran synchronously with parameter and optimizer offload; AdamW used a 1e-6 learning rate and zero weight decay. We ran the training for 50 steps.

Zero-variance groups were replaced DAPO-style during training. System errors were handled separately: a backend timeout is not an incorrect agent outcome, so affected groups were refilled rather than scored as failures.

That explains how retained tasks were trained. The remaining question is which tasks entered the training pool.

Offline policy-relative difficulty

Before diving deep into the selection of training tasks, a disclaimer:

No ThinkingBox-Bench task was included in the RL training pool. The benchmark was reserved for evaluation.

ThinkingBox-Bench is out-of-distribution relative to the RL training pool: its domains, databases, policies, and most tools were not encountered during RL training. The two sets still share the ThinkingBox framework, simulated-user machinery, executable terminal-state grading, a few common tools, and a similar task style. These shared elements limit the scope of the transfer claim.

Why selection matters comes down to one arithmetic fact about GRPO. If all eight rollouts fail, the reward vector is [0, 0, 0, 0, 0, 0, 0, 0]. If all eight succeed, it is [1, 1, 1, 1, 1, 1, 1, 1]. In either case, centering within the group produces no task-reward advantage because no trajectory did better than another.++

A mixed group such as [0, 0, 0, 1, 1, 1, 1, 1] supplies the contrast GRPO can learn from. That distinction is expensive for a stateful agent: every rollout provisions an isolated backend, runs a real tool session, and calls a separate model to play the user.

The question that shaped this work was therefore not only which model or optimizer? It was: which candidate tasks should enter the fixed training pool?

We used an offline pass-rate filter on the posterior to select the training pool. It estimates which tasks are likely to produce contrasting terminal rewards under the base policy.

So we probe. Each candidate task is run repeatedly against the base policy model. The plain-language rule is simple: tasks the model always fails are unlikely to teach it yet; tasks it always passes are already learned; tasks it solves sometimes are where outcomes can provide contrast.

Estimating difficulty from limited probes

Each candidate task is run n times, producing k successes. We model its latent success probability with Jeffreys posterior probability:

success probability | k successes in n probes ~ Beta(k + ½, n − k + ½)

Figure 6. Offline policy-relative difficulty is estimated from repeated base-policy probes and used to retain tasks likely to produce contrasting rewards.

We partition the unit interval into five buckets with boundaries at 0.08, 0.40, 0.60, and 0.90, chosen as hyperparameters for these runs. Each task is assigned to the bucket containing the greatest posterior mass:

• Rarely solved [0, 0.08): drop
• Difficult [0.08, 0.40): keep
• Intermediate [0.40, 0.60): keep
• Frequently solved [0.60, 0.90): keep for full-parameter RLFT
• Nearly always solved [0.90, 1]: drop

Further details are in Appendix C.4 of our ThinkingBox paper.

One task, end to end

Figure 7. Zooming in, one task through all six stages: GPT-5.4 screening, base-model probes, difficulty classification, a rollout group, terminal-state grading, and a GRPO update.

Figure 7 follows one candidate task that can potentially enter the training pool. For the Qwen3.8 27B full-parameter pool, GPT-5.4 first served as a solvability screen. The base policy then succeeded on few of the probes e.g 3 of 8, producing a Beta(3.5, 5.5) posterior. Its largest mass, 54.0%, fell in the Difficult band.

When sampled for training, the task produced the mixed reward group [0, 0, 0, 0, 1, 1, 1, 1]. Successful trajectories therefore received above-mean task advantage and failed trajectories below-mean advantage, giving GRPO contrast to learn from. A different policy model could place the same task in a different difficulty band.

The final Qwen3.8-27B curated training pool contained 187 tasks.

What earns a reward

Nothing about the trajectory. The reward is the task's own executable verdict.

When an episode ends, the sandbox extracts what actually changed in the backend and runs the task's deterministic checks against the terminal state. Any trajectory that produces the correct end state passes. Wrong, missing, or extra side effects fail. The reward is 1 or 0.

In the benchmark, the database grader can be thought of as the “examiner”: it caught agents that reported success and left the wrong record. In training, the same kind of check can be seen as the “teacher” on a different, disjoint set of tasks.

Two reward paths, one binary verdict

• Executable check (the large majority). When the required outcome is represented in terminal state or side effects, a deterministic test accepts or rejects it; no LLM judge call is needed.
• Judge-assisted check (a minority). When a narrow requirement cannot be determined cleanly from database state, the configured LLM judge answers a binary rubric question used by the executable test.

Run the training stack

We release the data-selection, rollout, reward, and Verl integration used to train Qwen3.8-27B in microsoft/thinkingbox-training. The repository README is the entry point. The detailed guides are:

Before you start

Tested on Linux and WSL with Python 3.12 and uv. You need NVIDIA GPUs with a compatible CUDA/NCCL stack, a local Qwen3.8-27B checkpoint, and OpenAI-compatible endpoints for the simulated user and judge. The validated reference configuration uses 3 nodes x 8 H100 GPUs; smaller custom configurations depend on model size, context length, and rollout concurrency.

The public sample task list exercises the interface. It is not the private 187-task training pool used for the reported run. Supply an authorized training source that remains disjoint from the public benchmark.

Install

Keep the three repositories side by side:

mkdir -p ~/thinkingbox-workspace
cd ~/thinkingbox-workspace

git clone https://github.com/microsoft/thinkingbox.git
git clone https://github.com/microsoft/thinkingbox-data.git
git clone --branch main https://github.com/microsoft/thinkingbox-training

git -C thinkingbox checkout 892964e7226044e5463ad188da119df838f4bec1
git -C thinkingbox-data checkout thinkingbox-bench-v1.0

Create the validated environment

The released stack targets Linux, Python 3.12, Verl 0.9.0, and the pinned package lock:

cd ~/thinkingbox-workspace/thinkingbox-training

uv venv --python 3.12
source .venv/bin/activate

uv sync --frozen --extra fast --extra tracking --extra eval-proxy
uv pip install -e ../thinkingbox
uv pip install --config-settings editable-mode=compat \
  -e ../thinkingbox-data/servers/thinkingbox_tools
uv pip install --config-settings editable-mode=compat \
  -e ../thinkingbox-data/servers/tb_business_ops_servers_202606

python scripts/prepare_verl.py --dest .deps/verl --install
python scripts/verify_verl_install.py
uv pip check

The final two commands should report a verified patched Verl checkout and no broken requirements. Do not run another automatic package sync afterward: it can replace the patched source installation with the vanilla Verl wheel.

Prepare the training data

Start with the publicly provided samples, or add your own authorized dataset in ThinkingBox format. Then run the repository's preparation script:

cd ~/thinkingbox-workspace/thinkingbox-training
source .venv/bin/activate
python data/prepare_data.py --help

The script aggregates policy probes, applies offline policy-relative difficulty filters and benchmark exclusions, and writes the selected train.yaml and test.yaml files. docs/data-preparation.md documents the input fields, loader contract, and complete command.

Configure and inspect the launch

Follow docs/reference-q38-run.md to start Typesense and the MCP servers, configure the simulated user and judge, set the model, task-list, checkpoint, and topology variables, and prepare the Ray cluster. Keep credentials in runtime secret injection rather than committed files or shell history.

Before allocating a training job, use the guide's dry-run command:

DRY_RUN=true ./scripts/run_train.sh \
  trainer.total_training_steps=50 \
  trainer.save_freq=5

It must exit zero without starting Ray or allocating GPUs. Read the emitted python -m verl.trainer.main_ppo command and every resolved override before launching the run.

Train

The launcher configures training; it does not provision hosts. On the reference three-node topology, export the same environment on every node, mount CHECKPOINT_DIR at the same absolute path, then start Ray:

ROLE=head NODE_IP="$HEAD_IP" GPUS_PER_NODE=8 \
  ./scripts/start_ray.sh


ROLE=worker NODE_IP="$WORKER_IP" \
  HEAD_ADDRESS="$HEAD_IP:6379" GPUS_PER_NODE=8 \
  ./scripts/start_ray.sh

After ray status reports all 24 GPUs, launch once from the head node:

export RAY_ADDRESS=auto
export GPUS=0,1,2,3,4,5,6,7

./scripts/run_train.sh \
  trainer.project_name=thinkingbox-training \
  trainer.experiment_name=q38-full-rlft \
  trainer.total_training_steps=50 \
  trainer.save_freq=5

Evaluate the checkpoint through OpenEnv

The first ThinkingBox post covers Typesense, the MCP Session Proxy, and OpenEnv setup. Merge the trained checkpoint following docs/evaluation.md, then serve it with vLLM.

From the OpenEnv repository root, set MERGED_MODEL to the merged model directory and THINKINGBOX_DATA to the pinned benchmark checkout. The historical checkpoint evaluation used this eight-A100 serving configuration:

vllm serve "$MERGED_MODEL" \
  --served-model-name q38-rlft \
  --host 127.0.0.1 \
  --port 8001 \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --language-model-only \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

In a separate terminal, start the OpenEnv compatibility proxy for the Qwen chat template from the ThinkingBox Training checkout. It listens on port 8014 and forwards to vLLM on 8001:

cd ~/thinkingbox-workspace/thinkingbox-training
source .venv/bin/activate
bash scripts/launch_sysmerge.sh

Keep it running during evaluation. From another terminal, check readiness:

curl --fail http://127.0.0.1:8014/readyz

Update the agent entry in the existing thinkingbox.yaml; retain its MCP, simulated-user, and judge configuration:

orchestrator:
  agent_model:
    type: aoai
    endpoint_url: http://127.0.0.1:8014/v1/chat/completions
    deployment: q38-rlft
    credential:
      type: api-key
      api_key: <your_api_key>
    is_reasoning: true
    temperature: 1.0
    top_k: 20
    top_p: 0.95
    max_completion_tokens: 8192
    timeout: 1800

Start OpenEnv as in the first post, with OPENENV_TB_CONFIG pointing to this thinkingbox.yaml and OPENENV_TB_DATASET to $THINKINGBOX_DATA. Once /ready succeeds, run the full protocol of 507 tasks × 20 attempts:

uv run --project envs/thinkingbox_env --frozen \
  thinkingbox-eval \
  --config "$PWD/thinkingbox.yaml" \
  --base-url http://127.0.0.1:8000 \
  --dataset "$THINKINGBOX_DATA" \
  --output q38-rlft-full.jsonl \
  --errors-output q38-rlft-full-errors.jsonl \
  --repeat 20 \
  --coverage-profile full \
  --aggregate-output q38-rlft-full-aggregate.json \
  --message-timeout 1800

The full profile validates all 10,140 expected task/repetition keys before aggregation.

Where this goes next

The useful part of this work is not one Qwen3.8-27B checkpoint. It is the training loop and training data curation that turns executable outcomes into rewards.

If you are training or evaluating an agent that touches real database records:

1. Probe before you train. Take your own agent task set, run the base policy against it n times, and look at the band histogram. In our experience teams often have not inspected this distribution, and it is usually more lopsided than expected.
2. Check your parity. Replay the same recorded model output through the training and evaluation parsers, and verify that both extract identical tool names and arguments. Otherwise, training may reward behavior that evaluation interprets differently.
3. Report both coverage and repeatability. State how many tasks pass at least once in k attempts and how many pass all k. A model that occasionally finds a solution is not necessarily one that repeats it consistently.

Further details can be found in the following links:

Cite the paper

@article{li2026thinkingbox,
  title   = {One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows},
  author  = {Li, Zhuochun and Ko, Youngmin and Keramati, Ali and Kundu, Tuhin and
             Tsai, Liang-Chun and Ferri, Nicola and Milletari, Mirco and Liu, Jiaxiang and
             Lopez Pelaez, Susana Palmaz and Wang, Yuepeng and Smolyakov, Vadim and
             Jiang, Xiang and Olafsson, Kjartan and Guy, Tommy},
  journal = {arXiv preprint arXiv:2608.19741},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.19741}
}

ThinkingBox code is MIT-licensed; the benchmark data is CDLA-Permissive-2.0.

Disclaimer: every task in the public benchmark is a synthetic reconstruction. The workflows and policies are modeled on real enterprise agent patterns; the customers are not real.

ThinkingBox, ThinkingBox-Bench and ThinkingBox-Training is built by the Microsoft Copilot Studio team in partnership with Toloka with collaborators from the University of Pittsburgh, Northwestern University, Columbia University and UC Irvine who interned at Microsoft. Questions are welcome in the comments section below or ongithub or by email at firstnamelastname@microsoft.com.

• A minority of tasks also include narrow binary rubric requirements for outcomes that the database cannot capture cleanly; those checks can invoke the configured LLM judge.

++ That zero-gradient problem has a standard batch-time fix: oversample, discard the degenerate groups, refill. The full-parameter run did exactly that, similar toDAPO with replacement of zero-variance groups. But refilling still pays full price for every wasted rollout before throwing it away.