工程化 Agent 质量，而非凭感觉检查

你已经交付了一个 Agent。它运行正常。你调整了一句提示词，修复了某个用户抱怨的问题，并且在你试过的三个例子上看起来更好了。但让你夜不能寐的问题是：我是不是刚刚弄坏了另外十个？

这种“在几个例子上看起来更好”与“在生产环境中真正更好”之间的差距，正是构建 Agent 过程中每天都在面对的现实。大多数团队或多或少都有一些评估用例，大多数团队都会调整提示词，但很少有人以足够的纪律将两者联系起来，从而知道某个改动到底改变了指标，还是仅仅改变了观感。

最可怕的失败并不是那些明显的崩溃，而是那些看似正在正常工作的 Agent——给出自信的回答，读起来像模像样的计划——却悄悄地搞错了用户的实际目标。在 Cloud Next '26 上，我们把 Agent 质量描述为一个三阶段飞轮——构建与测试 → 交付与监控 → 学习与优化——并展示了基本的构建模块。今天，我们在此基础上增加了面向开发者的路径：一项让你的编码 Agent 安装并为你驾驶的技能。

这个飞轮——其方法论以及核心的 AutoRater——基于我们用来评估和改进我们自己的模型和第一方 Agent 的相同原则，其中的 AutoRater 是与 Google DeepMind 紧密合作开发的。

飞轮的放大视图

这项技能以 构建与测试 这一快速迭代循环为中心，并将其扩展为五个具体阶段。它并不局限于该阶段：同样的阶段也会针对生产链路运行，随着时间推移，飞轮的更多部分将逐步触达。首次使用时可顺序运行，然后循环阶段 2–5，直到达到质量目标：

1. 准备数据：从现有的 OTel 链路、手工编写的用例或合成场景中构建评估数据集。
2. 运行推理：在数据集上执行 Agent 以产生链路；如果你已经有链路，可跳过此步。
3. 打分：使用 Google 的自适应 AutoRater（基于模型的裁判，对链路进行评分并解释原因）或你自己的自定义指标对链路评分。这是唯一始终运行的阶段。
4. 分析失败：阅读评分裁决，理解某个用例 为什么 会失败；若有十个或更多失败用例，可使用自动损失分析对它们进行聚类。
5. 优化与迭代：应用有针对性的修复，重新运行阶段 2–4，并与之前的基线进行比较。

大多数失败的用例都需要经过多次迭代才能让指标真正移动，而这项技能将这种纪律固化为流程。

优化器从不给自己的作业打分

优化器与评估器保持解耦：任何提出修复方案的东西（你的编码 Agent、自动化优化器或你自己）都绝不为其打分。Gemini 企业 Agent 平台 GenAI 评估服务独立进行评分。一个自己给自己打分的优化器会学会钻营指标，而不是真正改进 Agent。这个小小的架构选择，其影响比表面看起来要大得多。

这项技能是什么（和不是什么）

它是 运行在 你的编码 Agent 内部 的方法论加编排：它为目标选择正确的指标，运行 GenAI 评估服务，阅读评判结果，提出修复方案，并对比前后变化。

它不是：

• 全自动的。 它提出建议，你来批准。人参与其中，而非放手不管。
• 真值的来源。 内置的 AutoRater 并不仅仅是一个模型给一个答案打分。对于多轮 Agent，它会从对话中提取用户意图，生成针对该用例特定情况的评分量规，逐条验证链路是否达到每条标准，并对多次采样结果进行多数投票。它虽然很精细，但仍然基于模型：请将分数视为强方向性信号，并更信任不同运行之间的 差值，而不是将任何单一的绝对分数当成最终成绩。
• 真实流量的替代品。 该技能使用用户模拟器（GenAI 评估服务的一项功能）生成合成场景，但这只是一种冷启动引导手段。合成场景能让你起步，而生产数据才是使循环变得锐利的关键。

它针对同一 GenAI 评估服务提供了两种方式的安装包，选择适合你技术栈的方式即可：

一个真实的循环：一个看起来像成功的失败

以下是对一个真实 Agent 运行的一次循环。在阅读过程中请注意：开发者从未接触评估命令行界面，也从未指定过指标名称。他们安装了这项技能，用自然语言描述了一个担忧，批准了一项计划，然后阅读了结果。这项技能决定了 如何做，而它做的最有趣的决定，是选择哪个指标甚至能够检测到这种失败。

测试对象是 _travel-concierge_，一个来自 google/adk-samples 的 ADK 多 Agent 旅行规划器（灵感激发 → 规划 → 预订 → 出行前/中/后）。它将当前行程保留在会话状态中，这为一种特定而隐蔽的失败埋下了伏笔。

你告诉你的编码 Agent：

“在这个仓库上使用‘质量飞轮’技能。我担心 travel-concierge 是否能正确处理对话中途的更改。当用户中途修改旅行计划（不同的日期、不同的酒店、不同的出行人数）时，最终的计划是否反映了更改，还是继续基于最初的信息运作？请先弄清楚如何测试，并在运行任何东西之前提出一个计划。”

这就是整个交互界面。没有标志位，没有指标名称。这项技能的任务就是将这个目标转化为正确的评估方案，这正是它发挥价值的地方。它首先阅读了 Agent 的代码，然后回传了一个计划。它选择了两个内置的多轮 AutoRater，随后做了一件固定脚本做不到的事情：它在这两个指标之上添加了一个自定义的量规，专门用于将前后对比精确锁定在你所关注的那种行为上：

编码 Agent · quality-flywheel 技能

“multi_turn_task_success 和 multi_turn_trajectory_quality 是自适应的：它们会为每个用例生成一个量规，并且会在其中反映出修订遗漏的问题。但它作为多个标准中的一条出现，每次运行生成的量规都不相同，被揉合在一个混合分数中；并没有一个稳定的‘修订遵守情况’的数值可供设定阈值或观察趋势。因此，我将把这个关注点提升为它自己的指标：一个自定义量规 revision_honored，使用分类判定（HONORED / IGNORED / PARTIAL / NO_REVISION），这样我可以进行计数、设置门禁（如果超过 20% 返回 IGNORED 就采取行动），并在每个周期中进行跟踪。我将使用用户模拟器引导生成 25 个场景，并按五种修订类型进行划分。继续？”

你批准了。在那段话背后，技能运行了用户模拟器来合成这些场景，并使用两个内置指标加上它自己的 revision_honored 量规对链路进行评分。只有少数几个用例失败，低于需要聚类的阈值，因此它直接阅读了这些评判结果，而没有使用自动损失分析。以下是它实际运行的命令：

每种修订类型运行一次用户模拟器（×5：party_size, destination, dates, hotel, dropped_stop）

agents-cli eval dataset synthesize -n 5 --max-turns 8 --model gemini-3.5-flash
--instruction "$(cat instr_party_size.txt)"
--environment-context "$(cat synthesize_env_context.txt)"
-o traces_party_size.json

合并五个链路文件，然后一次性对所有 25 个进行评分

agents-cli eval grade --traces traces_merged.json --config eval_config_revisions.yaml

你一行代码也没写。技能选择了指标、模拟器和划分方式；你只描述了目标。

第一次运行。 三个指标全部触发了。内置的 AutoRater 已经显示出明显的低质量（各项均值在 0.6 左右，相对于严格标准的通过率较低），而自定义量规则隔离出其中有多少是专门由修订问题引起的：

在这个量规中，IGNORED 表示 修订被丢弃了（其他判定是 HONORED、PARTIAL 和 NO_REVISION）。这 21% 的比例超过了技能自己设定的行动阈值。而且这些判定精确地定位了失败所在。它不是你猜的那样：Agent 并没有信心满满地确认一个错误的行程。在四例失败中有三例，其内部状态是正确的（正确的值被存储了，正确的工具被调用了），但它的最终回复给用户的消息却仍然回显了过时的值。 Agent 在内部做了正确的事，却在对外表述时自相矛盾。其中一个评判结果将其具体化了：

“虽然 Agent 在第 3 轮中对 start_date 和 end_date 的 memorize 调用正确存储了 ‘2027-04-15’ 和 ‘2027-04-19’，但在用户明确纠正后，它在最终输出中未能向用户提供正确的日期。”

这就是“看起来像在工作”的失败缩影：没有崩溃，快速浏览时计划看起来没问题，Agent 听起来 像是按你说的做了，但用户实际收到的答案是错的。这些用例的共同原因是：根 Agent 的指令中没有告诉它在发送最终响应之前要对照用户的最新消息进行检查。

你可能会想，自定义量规到底有没有必要：毕竟内置指标是自适应的。事实证明，检测不是问题，但 隔离 失败才是。拿那个 IGNORED 而内置任务成功度仍能给出舒适 0.80 分的用例 party_size_02 来说：用户将其酒店需求改为某特定青年旅舍的宿舍房。下面的注解显示了原因：评分员 确实 生成了针对该确切要求的评判标准，并将其标记为未满足（它抓住了遗漏并作出了解释），但那条标准只是通过的四条标准旁边的一条，因此混合分数仍然很高。内置指标无法提供的是一个在所有 25 个用例上统一的“是否遵守修订？”这个单一数值；将关注点提升为自己的分类指标，才使得 21%→5% 的前后对比变得可量化。

一个用例，三个评分员：party_size_02

用户要求规划 5 人从柏林到阿姆斯特丹的经济旅行，然后中途将酒店需求改为“阿姆斯特丹世界青年旅舍的宿舍房”。

revision_honored (自定义) → IGNORED。 “Agent 承认了这一请求，但没有按照修订后的标准重新搜索，而是重新提供了之前的结果，并且从未记住新的偏好。”

multi_turn_task_success (内置) → 0.80。 生成了五项标准，四项通过：✓ 5 人经济行程 · ✓ 航班选项 · ✓ 确认 easyJet 选择 · ✓ 提供了酒店选项。第五项未通过：✗ “提供‘阿姆斯特丹世界青年旅舍’的宿舍房选项”：“Agent 未能提供所请求的具体信息…因为它声称缺乏工具能力。” 修订遗漏是真实的而且被指出，但它只是五项中的一项，因此混合分数仍然很高。

multi_turn_trajectory_quality (内置) → 0.67。 它的失分在于评估配置造成的痕迹，而非 Agent 缺陷：Agent 的工具模式没有暴露给评分员，因此它将合法的调用（flight_search_agent, _memorize_impl）标记为“不允许的工具”。这就是为什么我们依赖自定义量规和任务成功度，而非轨迹质量，来进行前后对比。

修复与重新运行。 你批准了一个有针对性的修改：在根 Agent 的指令中添加了三句话，要求它在发送最终响应之前与用户的最新修订进行核对。技能重新运行了相同的评估：

你甚至不需要一个特定的目标

上述循环是从一个明确的目标开始的。但这项技能在你还没有目标——当你无法说出哪里出了问题——时同样有效。将它指向一个未接触过的 Agent，对它说“ 找一个真实的问题并修复它”，它就会在广泛的范围内运行：合成多样化的场景，用内置的多轮指标进行评分，并自行浮现出最主要的失败簇。

我们恰好在一个不同的 Agent 上这样试过：来自 google/adk-samples 的 software-bug-assistant，一个接入了 真实 工具（MCP 工具箱背后的 Postgres 工单数据库，以及网页和 StackExchange 搜索）的缺陷分诊助手。在没有任何假设的情况下，这项技能立刻浮现出一个簇：在 15 个用例中有 14 个，Agent 正确地完成了工作，但从未告诉用户它调用了哪些工具。它自己的指令要求这样做，而模型悄悄地将其视为可选。一段简要的修复，强制要求每个响应末尾现在都附上类似“使用的工具：search-tickets, get-ticket-by-id”的页脚，在一个周期内就将所有 15 个用例中响应的符合率从 0% 提升到了 96%。

同样的技能，更宽松的提示。“这是我的目标”和“帮我找出一个问题”都能成功。

在这两个循环中，技巧是一样的：为你所改变的行为选择一个稳定的度量（一个自定义量规或一个简单的计数），并将自适应的内置指标视为整体健康信号，因为它们的量规在每次运行之间会发生变化。

从内循环到生产循环

上述循环使用了用户模拟器，因为我们还没有真实的使用数据——这是循环中按需开发的这一侧。随着 Agent 逐渐成熟并服务于真实流量，生产会话将成为最有价值的输入：每一个都是一个真实的请求——来自用户、其他 Agent 或上游服务，而每一个失败都是下一个循环现成的测试用例。此时，相同的几个阶段将由真实使用情况而不是模拟来驱动。

同样的技能可以针对生产流量运行；你只需将其指向真实链路，而不是合成的。告诉它对上周的生产会话进行评分，因为这些链路已经完成，它完全跳过“运行推理”阶段，直接在原地对它们进行评分。

Engineering agent quality, not vibe-checking it

You've shipped an agent. It works. You tweak a prompt to fix the one thing a user complained about, and it looks better on the three examples you tried. The question that keeps you up at night: did I just break ten others?

That gap, between "looks better on a few examples" and "actually better in production", is the daily reality of building agents. Most teams have eval cases somewhere. Most teams tweak prompts. Few connect the two with enough discipline to know whether a change moved the metric or just moved the vibe.

The scariest failures aren't the loud ones. They're the agents that look like they're working (confident answers, a plan that reads fine) while quietly getting the user's actual goal wrong. At Cloud Next '26 we talked about agent quality as a three-phase flywheel — Build & Test → Ship & Monitor → Learn & Refine — and showed the building blocks. Today we're adding the developer-facing path: a skill your coding agent installs and then drives on your behalf.

This flywheel — the methodology and the AutoRaters at its core — is built on the same principles we use to evaluate and improve our own models and first-party agents, with the AutoRaters developed in close partnership with Google DeepMind.

The flywheel, zoomed in

The skill is centered on Build & Test, the fast iteration loop, and expands it into five concrete stages. It isn't confined to that phase: the same stages run against production traces too, with more of the flywheel coming into reach over time. Run them in order on the first pass, then loop stages 2–5 until quality targets are met:

1. Prepare Data : build an evaluation dataset from existing OTel traces, hand-crafted cases, or synthesized scenarios.
2. Run Inference : execute the agent over the dataset to produce traces; skip this if you already have traces.
3. Grade : score traces with Google's adaptive AutoRaters (model-based judges that grade a trace and explain why) or your own custom metrics. This is the only stage that always runs.
4. Analyze Failures : read the rubric verdicts to understand why a case failed; for ten or more failures, cluster them with Automatic Loss Analysis.
5. Optimize & Iterate: apply a targeted fix, re-run stages 2–4, compare against the previous baseline.

Most failing cases take several iterations before metrics actually move, and the skill encodes that discipline.

The optimizer never grades its own work

The optimizer and the evaluator stay decoupled: whatever proposes a fix (your coding agent, an automated optimizer, or you) never grades it. The Gemini Enterprise Agent Platform GenAI evaluation service scores it independently. An optimizer that grades itself learns to game the metric instead of improving the agent. A small architectural choice matters more than it looks.

What this skill is (and isn't)

It is methodology plus orchestration running inside your coding agent : it chooses the right metric for the goal, runs the GenAI evaluation service, reads the verdicts, proposes fixes, and compares before/after.

It isn't:

• Autonomous. It proposes; you approve. Human-in-the-loop, not hands-off.
• A source of ground truth. The built-in AutoRaters are more than just a model scoring an answer. For a multi-turn agent they extract the user's intent from the conversation, generate rubrics specific to that case, validate the trace against each criterion, and majority-vote across samples. Sophisticated, but still model-based: treat the scores as a strong directional signal, and trust the deltas between runs more than any single number as an absolute grade.
• A substitute for real traffic. The skill generates synthetic scenarios with the User Simulator (a feature of the GenAI evaluation service), but it is a cold-start bootstrap. Synthetic scenarios get you moving; production data is what makes the loop sharp.

It ships in two packages against the same GenAI evaluation service. Pick whichever fits your stack:

A real cycle: a failure that looks like success

Here's one cycle on a real agent. The thing to notice as you read: the developer never touches the eval CLI, and never names a metric. They install the skill, describe a worry in plain language, approve a plan, and read the results. The skill decides how , and the most interesting decision it makes is which metric can even detect the failure.

The agent under test is _travel-concierge_, an ADK multi-agent trip planner from google/adk-samples (inspiration → planning → booking → pre/in/post-trip). It keeps the working itinerary in session state, which sets up a specific, subtle failure.

What you tell your coding agent:

"Use the quality-flywheel skill on this repo. I'm worried about whether travel-concierge honors mid-conversation changes. When a user revises the trip partway through (different dates, a different hotel, a different number of travelers), does the final plan reflect the change, or does it keep operating on the original? Figure out how to test it and propose a plan before running anything."

That's the whole interface. No flags, no metric names. The skill's job is to translate that goal into the right evaluation , and here's where it earns its keep. It reads the agent's code first, then comes back with a plan. It reaches for the two built-in multi-turn AutoRaters, then does something a fixed script wouldn't: it adds a custom rubric on top of them , purpose-built to pin the before/after to exactly the behavior you asked about:

coding agent · quality-flywheel skill

"_multi_turn_task_success_ and _multi_turn_trajectory_quality_ are adaptive: they'll generate a rubric for each case and will surface a revision miss inside it. But it lands as one criterion among several that are regenerated differently every run, folded into a blended score; there's no stable 'revision-honoring' number to threshold or trend. So I'll promote that one concern to its own metric: a custom rubric, _revision_honored_ , with a categorical verdict (HONORED / IGNORED / PARTIAL / NO_REVISION) that I can count, gate on (act if more than 20% come back IGNORED), and track cycle over cycle. I'll bootstrap 25 scenarios with the User Simulator, partitioned across the five revision types. Proceed?"

You approve. Behind that paragraph the skill runs the User Simulator to synthesize those scenarios and grades the traces with the two built-ins plus its own revision_honored rubric. Only a handful of cases fail, below the threshold where clustering pays off, so it reads those verdicts directly instead of reaching for Automatic Loss Analysis. Here's what it actually ran:

# one User Simulator pass per revision type (×5: party_size, destination, dates, hotel, dropped_stop)
agents-cli eval dataset synthesize -n 5 --max-turns 8 --model gemini-3.5-flash \
    --instruction "$(cat instr_party_size.txt)" \
    --environment-context "$(cat synthesize_env_context.txt)" \
    -o traces_party_size.json

# merge the five trace files, then grade all 25 in one pass
agents-cli eval grade --traces traces_merged.json --config eval_config_revisions.yaml

Shell

You wrote none of it. The skill chose the metrics, the simulator, and the partitioning; you described the goal.

First pass. All three metrics fired. The built-in AutoRaters already showed clearly sub-par quality (means in the mid-0.6s, with low pass-rates against a strict bar), and the custom rubric isolated how much of that was the revision problem specifically:

On this rubric, IGNORED means the revision was dropped (the other verdicts are HONORED, PARTIAL, and NO_REVISION). That 21% cleared the skill's own action threshold. And the verdicts located the failure precisely. It isn't what you'd guess: the agent doesn't confidently confirm a wrong itinerary. In three of the four failures, its internal state was correct (the right value was stored, the right tool was called), but its final message to the user echoed the stale value anyway. The agent did the right thing internally and contradicted itself out loud. One verdict makes it concrete:

"While the agent's internal _memorize_ calls for _start_date_ and _end_date_ in Turn 3 correctly stored '2027-04-15' and '2027-04-19', it failed to provide the correct date in its final output to the user after the explicit correction."

That's the "looks like it's working" failure in miniature: nothing crashes, the plan reads fine on a quick skim, the agent sounds like it did what you asked, but the answer the user actually receives is wrong. The common cause across the cases: nothing in the root agent's instruction told it to check its final response against the user's most recent message before sending.

You might wonder whether the custom rubric was needed at all: the built-ins are adaptive, after all. It turns out detection is not the problem, but isolating the failure is. Take the one IGNORED case where built-in task-success still scored a comfortable 0.80: party_size_02, where the user revised their hotel request to dorm rooms at a specific hostel. The annotation below shows why: the rater did generate a criterion for that exact request and marked it unmet (it caught the miss and explained it), but that one criterion sat among four that passed, so the blended score stayed high. What the built-in couldn't give you is a single "did it honor the revision?" number across all 25 cases; promoting the concern to its own categorical metric is what made the 21%→5% before/after countable.

One case, three raters: party_size_02

The user asked for a cheap Berlin→Amsterdam trip for 5, then mid-conversation revised the hotel request to "dorm rooms at Hostel World Amsterdam."

revision_honored (custom) → IGNORED. "The agent acknowledged the request but re-offered the earlier results instead of searching the revised criteria, and never memorized the new preference."

multi_turn_task_success (built-in) → 0.80. Five generated criteria, four passed: ✓ cheap trip for 5 · ✓ flight options · ✓ easyJet selection confirmed · ✓ hotel options provided. The fifth failed: ✗ "provides dorm room options at 'Hostel World Amsterdam'" : "the agent failed to provide the specific information requested … because it claimed a lack of tool capability." The revision miss is real and named; it's just one line in five, so the blended score stays high.

multi_turn_trajectory_quality (built-in) → 0.67. Its misses here are an eval-config artifact, not a defect: the agent's tool schemas weren't surfaced to the rater, so it flagged legitimate calls (flight_search_agent, _memorize_impl) as "tools not permitted." That's why we lean on the custom rubric and task-success, not trajectory, for the before/after.

The fix and the re-run. You approve a targeted change: three sentences added to the root agent's instruction, telling it to reconcile its final response with the latest user revision. The skill re-runs the same evaluation:

You don’t even need a specific goal

That cycle started from a clear goal. But the skill works just as well when you don’t have one yet, when you can’t name what’s wrong. Point it at an agent cold and say “ find a real failure and fix it ,” and it runs broad: synthesizing varied scenarios, grading them on the built-in multi-turn metrics, and surfacing the dominant failure cluster on its own.

We tried exactly that on a different agent: software-bug-assistant from google/adk-samples, a bug-triage assistant wired to real tools (a Postgres ticket database behind an MCP toolbox, plus web and StackExchange search). With no hypothesis, the skill surfaced one cluster immediately: in 14 of 15 cases the agent did the work correctly but never told the user which tools it had called. Its own instruction asked for it, and the model had quietly treated it as optional. A one-paragraph fix mandating that every response now ends with a footer like “Tools used: search-tickets, get-ticket-by-id” took that from 0% to 96% of responses across all 15 cases, in a single cycle.

Same skill, looser prompt. “Here’s my goal” and “find me a problem” both land.

In both cycles the trick was the same: pick one stable measure for the behavior you changed (a custom rubric or a plain count) and treat the adaptive built-ins as your broad-health signal, since their rubrics shift from run to run.

From the inner loop to the production loop

The cycle above used the User Simulator because we had no real usage yet — the on-demand, dev side of the loop. As the agent matures and serves real traffic, production sessions become the most valuable input: each one is a genuine request — from a user, another agent, or an upstream service and each failure is a ready-made test case for the next cycle. The same stages are now fed by real usage instead of simulation.

The same skill runs against production traffic; you just point it at real traces instead of synthesized ones. Tell it to grade last week's production sessions, and because those traces are already complete, it skips Run Inference entirely and grades them in place with the same raters. Online Monitors continuously evaluate live traffic and write quality scores to Cloud Monitoring; when scores drift, you hand the failing traces to the same skill: the eval-fix loop you just saw. Same flywheel, different cadence: continuous in production, on-demand in dev, with the same AutoRaters grading both.

Today the skill runs the inner loop on demand and grades production traces when you point it at them. The direction is to let it drive more of that outer loop on its own: watching the monitors, surfacing regressions, and proposing fixes as your traffic shifts.

Get started today

You will need: a GCP project with the Agent Platform GenAI Evaluation Service enabled, an agent to evaluate (ADK or any framework), and a coding agent to drive the skill. To grade production traffic, your agent should also emit OpenTelemetry traces (ADK does by default).

Install the skill your coding agent will drive:

# CLI-driven (ADK + agents-cli):
npx skills add https://github.com/google/agents-cli --skill google-agents-cli-eval

# SDK-driven (any framework):
npx skills add https://github.com/google/skills --skill agent-platform-eval-flywheel

Shell

Then start a cycle: point it at your agent and describe what you want measured. Your coding agent picks up the rest.

Your agent doesn't have to be perfect. It has to be improvable.

Credits: Quality Flywheel skills and the underlying services built by Jason Dai, Ludwik Trammer, Iwo Naglik, Xi Liu,Aleksandra Grzegorczyk, and the broader Cloud AI Agent Platform team. The talk this builds on was delivered at Cloud Next '26 with Alex Martin (Google) and Daniel J. Lewis (Geotab).

Learn more: Cloud Next '26 talk · Agent Evaluation docs · agents-cli on GitHub · google/skills on GitHub.