AutoSynthData：为企业 Agent 生成训练数据

企业需要能够在自身环境中良好运行的 Agent。这些 Agent 所承担的工作，取决于企业使用的系统、遵循的规则以及数据当前所处的状态。一个模型可能具备广泛的通用能力，却仍会在某个特定环境中遇到困难：可能无法妥善处理某个工作流、错误使用某组工具，或未能遵守某项约束。这些正是企业需要改进的薄弱环节。

难点在于如何将这些薄弱环节转化为训练数据。一次单独的失败能告诉我们一些信息，但训练模型需要大量新任务，以便在不同情境中检验同一种能力。这些任务还必须能够在环境中完成，类似于真实用户会提出的工作请求，并且具备可靠的方法来验证 Agent 是否成功。

在 ServiceNow CoreAI，我们构建了 AutoSynthData，用于将这些能力缺口转化为训练数据。它利用目标模型的失败和更强教师模型的成功，判断模型下一步应学习什么，然后生成并验证能够锻炼这些能力的新任务。随着模型能力提升，课程会转向它仍然感到困难的内容。我们使用 EnterpriseOps Gym（Malay et al., 2026）展示该流程，并采用其已发布数据集。首先，我们说明 Agent 所运行的环境，以及什么样的任务适合用于训练。

什么样的 Agent 任务有用？

Agent 环境定义了 Agent 所在的世界：它能够观察和修改的状态、可以调用的工具和 API，以及其行动所产生的状态转换。

任务在这一环境中被实例化。我们使用以下抽象：

task = (system specification, user prompt, verifier)

系统规范

系统规范定义了 Agent 运行时必须遵守的约束，包括系统指令、环境策略，以及在适用情况下的任务特定初始化内容，例如设定初始状态的数据库或一组知识文章。

该规范必须与环境的工具、状态和支持的操作兼容。其指令应当清晰，并避免仅为人为制造难度而加入任意约束。

面向 Agent 的任务

用户提示规定了用户希望 Agent 完成的事项，以及任何用户层面的约束。生成的任务应满足三项属性。

可行性。 在当前环境中，应至少存在一条轨迹，能够在遵守系统规范的前提下满足用户提示。这排除了依赖不可用工具、无法访问的知识、不可能的状态转换或被策略禁止的操作的任务。

真实性。 用户提示应当类似于用户在目标环境中合理可能提出的请求。可执行行为的空间通常远大于真实工作流的空间。

难度。 对于训练而言，任务应暴露当前 Agent 的薄弱点。已经能够被稳定解决的任务几乎不提供新的训练信号。因此，有价值的区域是那些可行、真实，但尚未被稳定解决的任务。

验证器

验证器决定最终轨迹是否成功完成任务。它应满足三项属性。

一致性。 它应与用户提示、系统规范和任务特定的环境状态保持一致。

健全性。 它应拒绝未满足任务要求或违反相关约束的轨迹。

完备性。 它应接受有效解，而不是将某一条特定参考轨迹硬编码为唯一答案。

这些属性会直接影响训练。宽松的验证器可能奖励错误行为，而过于严格的验证器则可能惩罚有效解。

概述

给定一个环境和目标模型，AutoSynthData 会生成由系统规范、用户提示和验证器构成的训练任务。生成的任务以环境为基础，并被选择为当前模型提供有用的训练信号。

AutoSynthData 首先使用诊断任务，在环境中评估目标模型，并识别其难以完成的任务模式。更强的教师模型帮助判断其中哪些任务可解，以及成功行为应当是什么样子。AutoSynthData 将由此得到的能力缺口转化为新的可执行任务，在环境中检查每个任务，并将通过审核的样本用于后训练。对更新后模型的评估将揭示哪些缺口仍然存在，并可指导下一轮生成。

从模型失败到课程体系

AutoSynthData 利用目标环境中的评估运行结果，识别模型下一步需要学习的内容。在我们的 EnterpriseOps Gym 实验中，我们让目标模型和更强的教师模型均执行评估任务。我们分析这些运行结果，以识别：

• 被测试的能力；
• 涉及的工具和工作流结构；
• 目标模型在哪些地方失败，以及教师模型如何成功；
• 正确最终状态必须满足的属性；
• 在保留被测试能力的情况下可以变化的维度。

我们将这些发现提炼为经过净化处理的能力规范卡。评估任务用于指导模型应学习什么，但生成器不会接收其原始提示、实体、轨迹或验证器细节。它接收的是这些规范卡，并利用它们创建具有不同提示、状态和解题路径的新任务。

生成并扩展任务

识别能力缺口告诉我们应当教授什么，但训练需要大量多样化的任务来锻炼这些能力。AutoSynthData 利用规范卡生成这些任务。

假设目标模型难以处理需要以下工作流的任务：

生成器会创建锻炼这一工作流的新任务，并改变实体、初始环境状态、工作流组合、工具组合、措辞和难度。随后，更强的教师模型为每项任务演示一条成功轨迹。对于监督微调（SFT）而言，这些演示会教导目标模型如何在新情境中应用该能力。

AutoSynthData 分两个阶段构建数据集：首先生成并验证核心样本，随后将其扩展为新的变体。

Target

Target 阶段根据能力规范创建核心训练样本集。工作单元会并行生成独立任务，并在完成后领取新的目标。每个候选样本在被接受前，均需经历验证、执行、求解器评估和修复。最终产出是一批围绕目标模型学习需求构建的、经过严格审查的样本。

Multiply

Multiply 阶段通过创建已接受 Target 样本的新变体来扩展数据集。每个变体都具有自己的用户请求、环境状态、实体配置、参考轨迹和验证器，并且必须通过相同的验证与执行检查。扩展生成的样本不能再作为另一个扩展样本的种子。这使扩展过程锚定于经过审查的 Target 集合，并限制多轮生成中的漂移。

实现细节

为支持这两个阶段，AutoSynthData 将生成控制与环境特定执行分离。一个共享控制器负责协调生成、质量控制、覆盖度和数据集构建；适配器则负责处理环境执行、任务与状态管理、参考轨迹重放、确定性验证、求解器执行和任务剖析。

并行 Target 生成与 Multiply 扩展共同为构建训练规模的数据集提供了路径。它们是否有用，取决于施加于每个候选样本的检查：任务必须可执行，解法必须有效，验证器必须能够区分成功与失败。

高质量合成数据需要的不只是生成

生成一个看似合理的请求，并不足以产出有用的训练数据。任务在目标环境中可能无法完成；其参考解在执行时可能失败；或者验证器可能会奖励错误的最终状态。AutoSynthData 会在接受任务用于训练前检查这些属性。

AutoSynthData 从两个层级审核质量：单个候选样本必须通过验证，批次则必须提供有用的覆盖度和多样性。

样本级验证与修复

每个候选样本在进入训练数据集前，必须通过一轮质量控制。我们先从求解器评估开始，以衡量难度。在这里使用的配置中，我们偏好目标模型在三次试验中最多成功一次、而更强求解器在三次试验中至少成功两次的任务。候选样本还会经历正向验证、负向验证和有次数上限的修复流程。

正向验证

正向关卡提出的问题是：预期解法是否能够解决生成的任务？

该流程会在目标环境中执行参考轨迹，并依据候选样本的验证器检查最终状态。这能够发现提示、初始状态、解法与成功标准之间的不匹配。

负向验证

负向关卡提出的问题是：相关的错误结果是否会失败？

例如，它可以改变预期结果的一部分，并确认这些状态不再能够通过验证。这样可以发现宽松的验证器：即使没有要求预期行为，它们仍会判定成功。

评审与修复

失败的候选样本不会被立即丢弃，而是先交由评审器处理。评审器会检查样本及其失败原因，寻找不一致的状态、不可能的工作流、不正确的任务构造、有问题的参考轨迹、薄弱的验证器逻辑，或与预期能力不匹配等问题。评审结果将指导修复，并设置固定的重试次数上限：

candidate
↓
failure
↓
critique / diagnosis
↓
targeted repair
↓
run the gates again
↓
accept or retry

修复后的任务必须再次通过相关检查。诊断结果会指导对现有候选样本的修复，而无需从头开始重新生成。

通过这些检查后，样本便具备训练资格；但单个有效样本仍可能组成一个重复或失衡的数据集。因此，AutoSynthData 还会在批次层面审查生成结果。

批次级审查

一个批次可能过度集中于少数简单任务类别、遗漏某项能力，或反映出过多生成资源被花费在低产出的模式上。

元审查会检查每个批次中已接受样本、被拒绝样本和生成行为。它会提出以下问题：

• 哪些任务类别被过度代表，哪些能力维度缺失？
• 是否反复出现相同类型的示例？
• 某些特定目标是否持续生成失败？
• 评审中是否出现系统性问题？
• 下一批次应如何调整指导策略？

控制器会追踪已接受数据集的覆盖度，减少在过度代表区域中的生成，并将更多工作导向缺口。当某一区域反复产生低质量候选样本时，评审和元审查会指导生成策略作出调整。这些调整会在可用生成预算和数据集规模要求下，平衡有用的学习信号、任务质量、覆盖度、多样性和低冗余性。

这些反馈循环共同提升单个任务及其所构成的数据集质量：样本级检查指导候选样本修复，批次级审查则指导未来生成。

推进训练前沿

随着模型能力提升，有用的训练分布也会发生变化。AutoSynthData 将合成数据生成视为对目标模型能力边界附近任务的搜索：任务要足够困难以暴露薄弱点，但也要足够可解，以便教师模型能够提供可靠演示。

后训练完成后，我们会在同一环境中评估更新后的模型。模型如今能够稳定解决的任务，对下一轮训练的价值会降低；持续失败则指向仍需关注的能力。这些结果可以指导下一轮生成。

[![图 07](https://cdn-uploads.huggingface.co/production/uploads/66a41f3c1d

Enterprises need agents that work well in their own environments. The work they ask these agents to do is shaped by the systems they use, the rules they follow, and the state of their data. A model may be broadly capable and still struggle with a particular environment: a workflow it handles poorly, a combination of tools it misuses, or a constraint it fails to respect. Those are the weaknesses an enterprise needs to improve.

The difficulty is turning those weaknesses into training data. An individual failure tells us something, but training a model requires many new tasks that exercise the same capability in different situations. Those tasks must also be possible to complete in the environment, resemble work someone would actually request, and have a reliable way to check whether the agent succeeded.

At ServiceNow CoreAI, we built AutoSynthData to turn those capability gaps into training data. It uses a target model’s failures and a stronger teacher’s successes to decide what the model should learn next, then generates and validates new tasks that exercise those capabilities. As the model improves, the curriculum shifts toward what it still finds difficult. We illustrate the pipeline with EnterpriseOps Gym (Malay et al., 2026), using the released dataset. We begin by describing the environment an agent operates in and what makes a task useful for training.

What makes a useful agentic task?

An agentic environment defines the world in which an agent operates: the state it can observe and modify, the tools and APIs it can invoke, and the state transitions produced by its actions.

A task is instantiated within this environment. We use the following abstraction:

task = (system specification, user prompt, verifier)

System specification

The system specification defines the constraints under which the agent operates, including system instructions, environment policies, and, when applicable, task-specific initialization such as a seeded database state or a set of knowledge articles.

The specification must be compatible with the environment’s tools, state, and supported actions. Its instructions should be clear and avoid arbitrary constraints introduced solely to manufacture difficulty.

Agent-facing task

The user prompt specifies what the user wants the agent to accomplish, together with any user-level constraints. A generated task should satisfy three properties.

Feasibility. There should exist at least one trajectory in the current environment that satisfies the user prompt while respecting the system specification. This rules out tasks that depend on unavailable tools, inaccessible knowledge, impossible state transitions, or actions prohibited by policy.

Realism. The user prompt should resemble something a user would plausibly ask in the target environment. The space of executable behaviors is usually much larger than the space of realistic workflows.

Difficulty. For training, the task should expose a weakness of the current agent. Tasks that are already solved reliably provide little new training signal. The useful region is therefore tasks that are feasible and realistic, but not yet consistently solved.

Verifier

The verifier determines whether the resulting trajectory successfully completes the task. It should satisfy three properties.

Consistency. It should agree with the user prompt, the system specification, and the task-specific environment state.

Soundness. It should reject trajectories that fail to satisfy the task or violate relevant constraints.

Completeness. It should accept valid solutions rather than encode one particular reference trajectory.

These properties matter directly during training. A lax verifier can reward incorrect behavior, while an overly restrictive verifier can penalize valid solutions.

Overview

Given an environment and a target model, AutoSynthData generates training tasks consisting of a system specification, user prompt, and verifier. The generated tasks are grounded in the environment and selected to provide useful training signal for the current model.

AutoSynthData first evaluates the target model in the environment using diagnostic tasks and identifies patterns in the tasks it struggles to complete. A stronger teacher helps characterize which of those tasks are solvable and what successful behavior looks like. AutoSynthData turns the resulting capability gaps into new executable tasks, checks each task in the environment, and uses accepted samples for post-training. Evaluating the updated model reveals which gaps remain and can guide the next round of generation.

From model failures to a curriculum

AutoSynthData uses evaluation runs in the target environment to identify what the model needs to learn next. In our EnterpriseOps Gym experiment, we run both the target model and a stronger teacher on the evaluation tasks. We examine those runs to identify:

• the capability being tested;
• the tools and workflow structure involved;
• where the target model fails and how the teacher succeeds;
• the properties that a correct final state must satisfy;
• the dimensions that can vary while preserving the capability being tested.

We distill these findings into sanitized capability specification cards. The evaluation tasks guide what the model should learn, but the generator does not receive their original prompts, entities, trajectories, or verifier details. It receives the cards and uses them to create new tasks with different prompts, states, and solution paths.

Generating and scaling tasks

Identifying a capability gap tells us what to teach, but training requires many varied tasks that exercise it. AutoSynthData uses the specification card to generate those tasks.

Suppose the target model struggles with tasks that require the following workflow:

The generator creates new tasks that exercise this workflow, varying the entities, initial environment state, workflow composition, tool combinations, wording, and difficulty. The stronger teacher then demonstrates a successful trajectory for each task. For supervised fine-tuning (SFT), these demonstrations teach the target model how to apply the capability in new situations.

AutoSynthData builds the dataset in two phases: first generating and validating core samples, then expanding them into novel variants.

Target

The target phase creates the core set of training samples from the capability specifications. Workers generate independent tasks in parallel, picking up a new target when they finish. Each candidate goes through validation, execution, solver evaluation, and repair before acceptance. The result is a batch of vetted examples built around what the target model needs to learn.

Multiply

The multiply phase expands the dataset by creating novel variants of accepted target samples. Each variant has its own user request, environment state, entity configuration, reference trajectory, and verifier, and must pass the same validation and execution checks. A multiplied sample cannot seed another multiplied sample. This anchors expansion to the vetted target set and limits drift across generations.

Implementation details

To support both phases, AutoSynthData separates generation control from environment-specific execution. A shared controller coordinates generation, quality control, coverage, and dataset construction, while an adapter handles environment execution, task and state management, reference replay, deterministic verification, solver execution, and task profiling.

Together, parallel target generation and multiplication provide a path to training-scale datasets. Their usefulness depends on the checks applied to every candidate: the task must be executable, the solution must work, and the verifier must distinguish success from failure.

High-quality synthetic data needs more than generation

Generating a plausible request is not enough to produce useful training data. A task may be impossible in the target environment, its reference solution may fail when executed, or its verifier may reward the wrong final state. AutoSynthData checks these properties before accepting a task for training.

AutoSynthData reviews quality at two levels: individual candidates must pass verification, and batches must provide useful coverage and diversity.

Sample-level verification and repair

Each candidate must clear a quality-control loop before entering the training dataset. We begin with solver evaluation to measure difficulty. In the configuration used here, we favor tasks the target model solves on no more than one of three trials and the stronger solver solves on at least two of three trials. Candidates also undergo positive and negative verification and a bounded repair process.

Positive verification

The positive gate asks: Does the intended solution solve the generated task?

The pipeline executes the reference trajectory in the target environment and checks the resulting state against the candidate’s verifier. This reveals mismatches among the prompt, initial state, solution, and success criteria.

Negative verification

The negative gate asks: Do relevant incorrect outcomes fail?

For example, it can mutate parts of the expected outcome and confirm that those states no longer pass verification. This catches weak verifiers that award success without requiring the intended behavior.

Critique and repair

Failed candidates go to a critic before being discarded. The critic examines the sample and its failure, looking for inconsistent state, impossible workflows, incorrect task construction, bad reference trajectories, weak verifier logic, or a mismatch with the intended capability. The critic’s findings guide repairs, with a fixed limit on retries:

candidate
↓
failure
↓
critique / diagnosis
↓
targeted repair
↓
run the gates again
↓
accept or retry

A repaired task must pass the relevant checks again. The diagnosis guides repairs to the existing candidate rather than requiring generation to start over.

Passing these checks makes a sample eligible for training, but individually valid samples can still form a repetitive or unbalanced dataset. AutoSynthData therefore also reviews generation at the batch level.

Batch-level review

A batch may overrepresent a few easy task families, miss a capability, or reflect too much generation effort spent on a low-yield pattern.

A meta-review examines accepted samples, rejected samples, and generation behavior across each batch. It asks:

• Which task families are overrepresented, and which capability dimensions are missing?
• Are the same kinds of examples appearing repeatedly?
• Do particular targets keep failing generation?
• Are systematic problems appearing in critiques?
• What guidance should change for the next batch?

The controller tracks coverage in the accepted dataset, reduces generation in overrepresented regions, and directs more work toward gaps. When a region repeatedly produces poor candidates, critiques and meta-review guide changes to the generation strategy. These adjustments balance useful learning signal, task quality, coverage, diversity, and low redundancy within the available generation budget and dataset size requirements.

Together, these feedback loops improve both individual tasks and the dataset they form: sample-level checks guide candidate repair, while batch-level review guides future generation.

Moving the training frontier

The useful training distribution changes as the model improves. AutoSynthData treats synthetic data generation as a search for tasks near the target model's capability boundary: difficult enough to expose weaknesses, but solvable enough for the teacher to provide reliable demonstrations.

After post-training, we evaluate the updated model in the same environment. Tasks it now solves reliably are less useful for the next training round; persistent failures point to capabilities that still need attention. Those results can guide the next generation round.

Our experiments focus on SFT, but the same mechanism could support reinforcement learning (RL): generate tasks that challenge the current policy and provide reliable learning signal, train, then move the generation target with the updated policy. We plan to test this moving, difficulty-calibrated frontier beyond SFT.

EnterpriseOps Gym experiments

We use EnterpriseOps Gym to test whether this approach improves a model on tasks in a stateful enterprise environment. We generate training tasks in the Gym’s Hybrid and ITSM environments, fine-tune the target model on accepted samples, and evaluate the resulting checkpoints.

Hybrid

We tested the pipeline on the Hybrid domain of EnterpriseOps Gym using Gemma-4-26B-A4B-it as the target model and Qwen3.8-27B as the teacher.

AutoSynthData generated 2,000 synthetic training samples in about 18 hours. We fine-tuned Gemma on this dataset and evaluated the resulting checkpoints on the benchmark. The best checkpoint was epoch 5.

Hybrid results

The synthetic SFT checkpoint improves mean Pass@1 by 7.2 percentage points, a 35% relative improvement, and raises verifier success from 63.01% to 68.55%. It closes 59% of the original Pass@1 gap between Gemma and the reference model.

The training tasks were newly generated from capability specifications; the generator did not receive the original evaluation tasks. The result demonstrates improvement in EnterpriseOps Gym Hybrid, the environment used for this experiment.

ITSM

We also applied AutoSynthData to the ITSM domain of EnterpriseOps Gym, using Gemma-4-26B-A4B-it as the target model and DeepSeek-V4.1-Flash as the teacher. AutoSynthData generated 1,994 synthetic training samples in 66 hours. Generation took longer than in the subsequent Hybrid run described above, primarily because the ITSM run used a larger teacher model and preceded pipeline optimizations that improved throughput.

On ITSM, synthetic SFT raises mean Pass@1 from 18.77% to 27.18%, showing that the approach also improves performance in a second domain.

Closing the loop

The tasks most useful for training depend on both the environment and the model working in it. AutoSynthData uses the model’s failures to choose what to generate, validates new tasks against the environment, and makes those tasks available for post-training. Our EnterpriseOps Gym results show the value of that approach in a controlled setting. As the model changes, the same process can focus on the gaps that remain.