评估 AI Agent：使用 Strands 与 AgentCore 的生产级蓝图

本文由 Motorway、AWS Prototyping and AI Customer Engineering（PACE）团队联合撰写。

---

英国在线汽车交易平台 Motorway 每天运营一场拍卖，最多有 8,000 家经销商为最多 2,500 辆汽车竞价。Motorway 与 AWS Prototyping and AI Customer Engineering（PACE）合作，构建了一款由 AI 驱动的经销商库存搜索 Agent。该 Agent 通过自然语言查询取代了数小时的手动筛选，改变了经销商寻找车辆的方式。

挑战

Agent 可以给出听起来很自信的回答，但在涉及真实资金时，如何证明它能够可靠运行？

• 工具选择错误会导致错误的搜索结果，损害经销商信任。
• 语义搜索误解会返回不相关结果。例如，“车龄不超过 5 年的汽油、混动和电动车”这类查询，要求 Agent 正确解析多项约束。
• 多轮对话中的上下文漂移会丢失经销商后续补充的筛选条件。
• 非确定性输出使得单次试验测试并不可靠。

解决方案

Motorway 与 AWS 共同构建了一条端到端评估流水线，将错误结果从每 8 次查询中 1 次降低至每 50 次查询中 1 次，并将问题发现时间从数小时缩短至数分钟。

该流水线结合了 Strands Agents SDK 和 Amazon Bedrock AgentCore；后者是一项用于大规模部署和运营 AI Agent 的全托管服务。本文将介绍如何为自己的 Agent 构建这条流水线：

• 两阶段评估策略：涵盖构建期测试和生产监控。构建期使用 strands-agents-evals（Strands Agents 的开源评估库），生产环境使用 Amazon Bedrock AgentCore Evaluations。
• 用于评估工具使用、推理和输出质量的三层框架。
• 具有质量门槛的五阶段部署流水线：当指标低于阈值时阻止发布。

一个配套代码仓库提供了可部署的蓝图，你可以将其调整用于自己的 Agent。尽管该蓝图使用 AWS 服务，但其中的核心原则是所有生产就绪 AI Agent 都必须满足的、与系统无关的要求。这些原则包括三层评估框架，以及使用 pass^k 指标衡量一致性。

前提条件

你必须具备以下前提条件，才能跟随本文操作。

**完成时间：**初次部署需 30–45 分钟；根据你的业务领域进行定制需 2–3 小时。

**预估成本：**运行示例评估套件的 Amazon Bedrock 推理费用约为 5–10 美元。生产监控成本会因采样率而异。

**安全说明：**配套代码仓库实现了最小权限 AWS Identity and Access Management（IAM）角色，将 API 密钥存储在 AWS Systems Manager Parameter Store 中（而非环境变量），并使用类型化参数来帮助防止注入攻击。详情请参阅代码仓库 README。

实例：经销商库存搜索 Agent

Motorway 基于 Strands Agents SDK 和 Amazon Bedrock AgentCore 构建了经销商库存搜索 Agent。该 Agent 提供八项工具，结合针对超过 89 项车辆属性的结构化筛选，以及由 LanceDB 和 Amazon Titan Text Embeddings V2 驱动的向量相似度搜索。

经销商通常需要通过 CSV 文件和固定筛选器浏览房源，耗费数小时。引入对话式 AI Agent 后，经销商现在可以直接与 Agent 对话，例如：“帮我找经销店附近价格低于 2.5 万英镑的柴油 SUV”，或者“适合家庭、偏运动风格且为自动挡的车型”。

高峰时段约有 1,500 名并发用户，因此让 Agent 行为正确并非可有可无。工具选择错误或语义搜索误解会直接影响用户信任。图 1 展示了端到端请求流程：经销商通过 Web 界面提交自然语言查询，查询被路由至 Amazon Bedrock AgentCore Runtime。该运行时在使用 Amazon Bedrock 模型（Claude 用于推理，Amazon Titan 用于嵌入）的同时，编排对八种不同工具的调用。工具响应经由运行时返回，并生成最终面向经销商的结果。

为什么 Agent 评估有所不同

大语言模型（LLM）评估关注文本生成质量：连贯性、事实准确性和回答相关性。Agent 评估考察的是本质上不同的内容。可以这样理解：LLM 评估考察的是发动机性能；Agent 评估则考察整辆车在车流中、雨天或后座坐满乘客时的驾驶表现。

传统 LLM 指标无法告诉你，Motorway 的 Agent 是否针对“Grade 1 Suzuki models”调用了正确的搜索工具。它们无法揭示 Agent 是否将正确的筛选参数传递给 LanceDB，也无法发现经销商在前一轮基础上进一步细化结果时，是否获得了正确响应。

评估维度	为何对 Agent 很重要
任务完成度	Agent 运行多步骤工作流，部分完成的情况很常见
工具使用正确性	错误的工具或不当参数可能使整个工作流偏离轨道
推理连贯性	有缺陷的推理会在条件变化时导致不可预测的失败
可靠性与一致性	非确定性意味着相同输入可能产生不同结果
安全性与合规性	自主 Agent 可能执行具有现实后果的操作
成本与效率	每个任务需要 50 次 API 调用的 Agent 可能不具备经济可行性

像“车龄 7–12 年的大众 Golf”这样的精确查询可能运行得非常完美，但“我在找一辆老款 VW”这种口语化表述，如果没有对语义搜索层进行适当评估，就可能失败。

使用 strands-agents-evals 在部署前发现问题

该蓝图在两个阶段实施评估，对应 GenAIOps 生命周期。构建期评估在部署前发现问题，生产评估则发现合成测试遗漏的问题。下图（图 2）展示了工具使用、推理和输出质量三层必须全部通过，才能部署。该框架从三个层面评估 Agent：

• 第 1 层（工具使用）验证正确的工具选择和参数传递，阈值为高于 95%。
• 第 2 层（推理）评估逻辑决策能力，阈值为高于 85%。
• 第 3 层（输出质量）衡量回答的有用性和准确性，阈值为高于 90%。三层都必须通过，部署才能继续。

在开发和持续集成、持续部署（CI/CD）期间，该流水线使用 strands-agents-evals 框架，在部署前捕捉问题。该框架提供输出验证、轨迹评估、多轮对话模拟和自动化实验生成。每项能力都专为基于 Strands Agents SDK 构建的 Agent 原生设计。该框架提供三种基础构件：

• Experiment：针对 Agent 运行的一组测试用例。
• Case：输入查询、预期输出和预期工具轨迹。
• Evaluator：评分逻辑，可为确定性逻辑或基于 LLM 的逻辑。

应将测试按层组织。第 1 层采用基于代码的确定性评分器来衡量工具选择准确性；第 2 层和第 3 层使用 LLM-as-judge 评估器（使用 LLM 为 Agent 输出评分）来评估推理和输出质量。

自定义 Evaluator 子类用于处理特定领域的问题。对于 Motorway Agent，这些问题包括数据新鲜度、经销商范围限定和安全护栏。你的 Agent 也会有其自身的领域约束。

三类评分器

评估框架使用三种评分器类型，每种类型适用于不同的评估需求。

评分器类型	层级	衡量内容	权衡
基于代码的确定性评分器	第 1 层	工具选择、参数传递、轨迹顺序	快速、成本低、可复现
LLM-as-judge（Claude Sonnet 4.6）	第 2–3 层	推理质量、输出有用性、目标成功率	灵活；存在非确定性（通过 pass^k 控制）
人工审核	校准	边缘案例与安全性	成本高；用于校准 LLM 评审提示词

在实践中，对 Agent 产出内容进行评分比评估其执行路径能发现更多问题。你关心的是用户是否获得相关结果，而不是 Agent 最先调用了哪个工具。

三层评估框架

构建期评估横跨三个不同层级，每层都有明确的通过/失败阈值。

**第 1 层：工具使用（阈值 > 95%）。**Agent 是否使用正确参数调用了正确工具？

• “价格介于 7,000 至 20,000 英镑的柴油车”应使用带有类型化筛选器的 search_vehicles
  (fuel_type=diesel, min_price=7000, max_price=20000)。
• “低里程的现代掀背车”应触发 hybrid_search，将语义嵌入与结构化筛选结合。

你可以通过确定性方式衡量这一点：ToolSelectionGrader 检查调用了哪些工具，TrajectoryOrderGrader 验证调用顺序。

**第 2 层：推理（阈值 > 85%）。**决策过程是否合乎逻辑？strands-agents-evals 中的 HelpfulnessEvaluator 和 TrajectoryEvaluator 使用 LLM-as-judge 评分，评估 Agent 的推理是否前后一致。通过不合逻辑推理得到正确回答的 Agent，会在条件变化时出现不可预测的失败。

**第 3 层：输出质量（阈值 > 90%）。**回答是否有用、准确且可执行？strands-agents-evals 中的 OutputEvaluator 和 GoalSuccessRateEvaluator 使用 LLM-as-judge 评估，以判断用户是否获得了有用且格式良好的回答。

三个层级都必须通过才能部署。任一层失败都会阻塞流水线。

处理非确定性

由于 LLM 输出在不同运行之间会发生变化，单次试验的结果可能具有误导性。配套代码仓库中的 run_all_layers() 函数接受 num_trials 参数来应对这一问题。代码生成研究社区中的两项指标有助于衡量可靠性：

• pass@k 衡量在 k 次尝试中至少成功一次的可能性。当找到一个正确解决方案就足够时，该指标很有用。
• pass^k（pass 的 k 次方）衡量在连续 k 次试验中均成功的概率。当用户期望每次都获得可靠行为时，该指标很有用。

对于面向客户的 Agent，pass^k 最为重要。单次试验成功率为 75% 的 Agent，连续三次试验全部通过的概率仅为 42%（0.75³）。用户期望每次交互都具备一致的质量。

在配套代码中，run_all_layers(task_fn, registry, num_trials=5) 以多次试验支持运行评估层，并根据 pass^k 决定是否允许部署。请参阅完整实现。

测试用例管理

测试用例按类别组织：

• 常规路径：应当成功的常见查询。
• 边缘案例：模糊查询、俚语和多轮细化查询。
• 安全/护栏：Agent 应拒绝或重定向的查询。

当生产监控发现问题时，该交互就会成为一个新的测试用例。Motorway 的测试套件在三个月内从最初的 50 个用例增长到 150 个，每个用例都基于真实用户行为。可以从 20 至 50 个用例开始，并让生产数据推动测试套件增长。

应包含 Agent 不应 调用某些工具的负向用例。例如，个人资料查询应调用个人资料工具，而不是搜索工具；结构化查询应使用结构化搜索，而不是原始 SQL 回退方案。单边评估会造成单边优化。

多轮对话测试

单轮评估遗漏了一个关键维度：对话连贯性。经销商会自然地在多轮对话中不断细化搜索：

• 第 1 轮：“帮我找柴油 SUV。”
• 第 2 轮：“现在只显示自动挡的。”
• 第 3 轮：“换成旅行车怎么样？”

strands-agents-evals 框架提供 ActorSimulator 来生成真实的多轮交互，并提供 InteractionsEvaluator 对跨轮次的上下文保留进行评分。多轮测试能够发现单轮测试遗漏的上下文漂移、筛选条件累积错误和代词指代解析失败。

使用 AgentCore Evaluations 监控生产行为

将 Strands Agent 部署到 Amazon Bedrock AgentCore Runtime 后，AgentCore Evaluations 可提供持续监控。它通过 OpenTelemetry 插桩与 Strands Agents 集成（业界标准的可观测性

This post was co-written with Motorway and the AWS Prototyping and AI Customer Engineering (PACE) team.

---

Motorway, a UK-based online car marketplace, runs a daily auction where up to 8,000 dealers bid on up to 2,500 vehicles. Motorway worked with AWS Prototyping and AI Customer Engineering (PACE) to build an AI-powered dealer stock search agent that transforms how dealers find vehicles, replacing hours of manual filtering with natural language queries.

The challenge

The agent gives a confident-sounding response, but how do you prove it works reliably with real money on the line?

• Tool selection errors cause wrong search results, eroding dealer trust.
• Semantic search misinterpretations return irrelevant results. A query like “Petrol, Hybrid and electric cars up to 5 years old” requires the agent to correctly parse multiple constraints.
• Context drift in multi-turn conversations loses dealer refinements.
• Non-deterministic outputs make single-trial testing unreliable.

The solution

Together, Motorway and AWS built an end-to-end evaluation pipeline that reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from few hours to few minutes.

The pipeline combines the Strands Agents SDK with Amazon Bedrock AgentCore, a fully managed service for deploying and operating AI agents at scale. In this post, you will learn how to build this pipeline for your own agents:

• A two-phase evaluation strategy spanning build-time testing with strands-agents-evals (the open source evaluation library for Strands Agents) and production monitoring with Amazon Bedrock AgentCore Evaluations.
• A three-layer framework for assessing tool usage, reasoning, and output quality.
• A five-stage deployment pipeline with quality gates that block releases when metrics fall below thresholds.

A companion repository provides a deployable blueprint that you can adapt for your own agents. Although the blueprint uses AWS services, the core principles are essential and system-agnostic requirements for any production-ready AI agent. These principles include the three-layer evaluation framework and the use of the pass^k metric for consistency.

Prerequisites

You must have the following prerequisites to follow along with this post.

Time to complete: 30–45 minutes for initial deployment and 2–3 hours to customize for your domain.

Estimated costs: Running the sample evaluation suite costs approximately $5–10 in Amazon Bedrock inference charges. Production monitoring costs vary based on sampling rate.

Security note: The companion repository implements least-privilege AWS Identity and Access Management (IAM) roles, stores API keys in AWS Systems Manager Parameter Store (not environment variables), and uses typed parameters to help prevent injection attacks. See the repository README for details.

The worked example: A dealer stock search agent

Motorway built the dealer stock search agent on the Strands Agents SDK and Amazon Bedrock AgentCore. The agent exposes eight tools that combine structured filtering across over 89 vehicle attributes with vector similarity search powered by LanceDB and Amazon Titan Text Embeddings V2.

Dealers typically spent hours browsing through listings using CSVs and rigid filters. By introducing a conversational AI agent, dealers can now talk to the agent: “Find me diesel SUVs under 25k near my dealership” or “something sporty and automatic for a family.”

With around 1,500 concurrent users during peak hours, getting agent behavior right isn’t optional. A tool selection error or semantic search misinterpretation directly impacts user trust. Figure 1 shows the end-to-end request flow: dealers submit natural language queries through a web interface, which routes to Amazon Bedrock AgentCore Runtime. The runtime orchestrates calls to eight different tools while using Amazon Bedrock models (Claude for reasoning, Amazon Titan for embeddings). Tool responses flow back through the runtime to generate the final dealer-facing results.

Why agent evaluation is different

Large language model (LLM) evaluation focuses on text generation quality: coherence, factual accuracy, response relevance. Agent evaluation assesses something fundamentally different. Think of it this way: LLM evaluation examines engine performance. Agent evaluation assesses how the whole car drives in traffic, in rain, or with a backseat full of passengers.

Traditional LLM metrics don’t tell you whether the Motorway agent called the right search tool for “Grade 1 Suzuki models.” They don’t reveal if the agent passed correct filter parameters to LanceDB. And they miss whether a dealer refining results from a previous turn gets the right response.

Evaluation dimension	Why it matters for agents
Task completion	Agents run multi-step workflows where partial completion is common
Tool use correctness	Wrong tools or bad parameters can derail entire workflows
Reasoning coherence	Flawed reasoning leads to unpredictable failures as conditions change
Reliability and consistency	Non-determinism means the same input can produce different outcomes
Safety and compliance	Autonomous agents can take actions with real-world consequences
Cost and efficiency	An agent requiring 50 API calls per task may not be economically viable

A precise query like “Volkswagen Golf 7-12 years old” might work perfectly, but a colloquial variant like “I’m looking for an older VW” could fail if the semantic search layer isn’t properly evaluated.

Catch issues before deployment with strands-agents-evals

The blueprint implements evaluation across two phases that map to the GenAIOps lifecycle. Build-time evaluation catches issues before deployment, and production evaluation catches what synthetic tests miss. The following diagram (Figure 2) shows how the tool usage, reasoning, and output quality layers must pass before deployment. The framework evaluates agents across three layers:

• Layer 1 (Tool Usage) validates correct tool selection and parameter passing with a greater than 95 percent threshold.
• Layer 2 (Reasoning) assesses logical decision-making with a greater than 85 percent threshold.
• Layer 3 (Output Quality) measures response helpfulness and accuracy with a greater than 90 percent threshold. All three layers must pass before deployment proceeds.

During development and continuous integration and continuous deployment (CI/CD), the pipeline uses the strands-agents-evals framework to catch issues before deployment. It provides output validation, trajectory evaluation, multi-turn conversation simulation, and automated experiment generation. Each is designed to work natively with agents built on the Strands Agents SDK. The framework provides three primitives:

• Experiment: A collection of test cases run against the agent.
• Case: Input query, expected output, and expected tool trajectory.
• Evaluator: Scoring logic (deterministic or LLM-based).

Structure your tests into layers. Layer 1 runs deterministic code-based graders for tool selection accuracy. Layers 2 and 3 use LLM-as-judge evaluators (using an LLM to score agent outputs) for reasoning and output quality.

Custom Evaluator subclasses handle domain-specific concerns. For the Motorway agent, these cover data freshness, dealer scoping, and safety guardrails. Your agent will have its own domain constraints.

Three types of graders

The evaluation framework uses three grader types, each suited to different evaluation needs.

Grader type	Layer	What it measures	Trade-off
Code-based deterministic	Layer 1	Tool selection, parameter passing, trajectory ordering	Fast, cheap, reproducible
LLM-as-judge (Claude Sonnet 4.6)	Layers 2–3	Reasoning quality, output helpfulness, goal success	Flexible; non-deterministic (controlled via pass^k)
Human review	Calibration	Edge cases and safety	Expensive; used to calibrate LLM judge prompts

In practice, grading what the agent produced catches more issues than grading the path it took. You care that the user got relevant results, not which tool the agent called first.

Three-layer assessment framework

Build-time evaluation operates across three distinct layers, each with specific pass/fail thresholds.

Layer 1: Tool Usage ( > 95 percent threshold). Did the agent call the right tools with correct parameters?

• “Diesel vehicles from £7,000 to £20,000” should use search_vehicles with typed filters
  (fuel_type=diesel, min_price=7000, max_price=20000).
• “Modern hatchback with low mileage” should trigger hybrid_search , combining semantic embeddings with structured filters.

You measure this deterministically: ToolSelectionGrader checks which tools were called, TrajectoryOrderGrader verifies the call sequence.

Layer 2: Reasoning ( > 85 percent threshold). Was the decision-making process logical? The HelpfulnessEvaluator and TrajectoryEvaluator from strands-agents-evals use LLM-as-judge scoring to assess whether the agent’s reasoning holds together. An agent that arrives at the right response through illogical reasoning will fail unpredictably as conditions change.

Layer 3: Output Quality ( > 90 percent threshold). Was the response helpful, accurate, and actionable? The OutputEvaluator and GoalSuccessRateEvaluator from strands-agents-evals use LLM-as-judge evaluation to assess whether the user got a useful, well-formatted response.

The three layers must pass before deployment. A failure in a layer blocks the pipeline.

Handling non-determinism

Because LLM outputs vary between runs, single-trial results can be misleading. The run_all_layers() function in the companion repository accepts a num_trials parameter to address this. Two metrics from the code generation research community help measure reliability:

• pass@k measures the likelihood of succeeding at least once in k attempts. This metric is useful when finding one correct solution is sufficient.
• pass^k (pass to the power of k) measures the probability of succeeding in k consecutive trials. This metric is useful when users expect reliable behavior every time.

For customer-facing agents, pass^k matters most. An agent with a 75 percent per-trial success rate has only a 42 percent chance of passing three consecutive trials (0.75³). Users expect consistent quality on every interaction.

In the companion code, run_all_layers(task_fn, registry, num_trials=5) runs the evaluation layers with multi-trial support and gates deployment on pass^k. See the full implementation.

Test case management

Test cases are organized by category:

• Happy path : Common queries that should succeed.
• Edge cases : Ambiguous queries, slang, multi-turn refinements.
• Safety/Guardrails : Queries the agent should refuse or redirect.

When production monitoring detects an issue, that interaction becomes a new test case. Motorway’s suite grew from an initial 50 cases to 150 in three months, each grounded in real user behavior. Start with 20 to 50 cases and let production data grow the suite.

Include negative cases where the agent should not call certain tools. For example, a profile query should call the profile tool, not the search tool. A structured query should use structured search, not a raw SQL fallback. One-sided evals create one-sided optimization.

Multi-turn conversation testing

Single-turn evaluation misses a critical dimension: conversational coherence. Dealers naturally refine searches across multiple turns:

• Turn 1: “Find me diesel SUVs.”
• Turn 2: “Now show me only the automatics.”
• Turn 3: “What about estates instead?”

The strands-agents-evals framework provides ActorSimulator to generate realistic multi-turn interactions and InteractionsEvaluator to score context retention across turns. Multi-turn tests catch context drift, filter accumulation errors, and pronoun resolution failures that single-turn tests miss.

Monitor production behavior with AgentCore Evaluations

After you deploy your Strands Agent to Amazon Bedrock AgentCore Runtime, AgentCore Evaluations provides continuous monitoring. It integrates with Strands Agents through OpenTelemetry instrumentation (the industry-standard observability framework). Figure 3 illustrates the architecture in production, with observability traces sampled at 1–5 percent and with metrics aggregating to Amazon CloudWatch.

Two monitoring approaches

AgentCore Evaluations offers two complementary modes:

On-demand evaluation analyzes specific agent interactions by selecting spans from Amazon CloudWatch logs. This is useful for debugging issues or validating fixes.

Online evaluation automatically samples live traffic and applies evaluators in the background. Configure a sampling rate (1–5 percent is recommended), select up to 10 evaluators, and let it run.

Built-in and custom evaluators

AgentCore provides pre-configured evaluators for common scenarios. The following table shows the built-in evaluators and what they measure.

Evaluator	Level	What it measures
Builtin.Helpfulness	TRACE	How helpful the agent’s response is (0–1 score across 7 levels)
Builtin.GoalSuccessRate	SESSION	Whether the user’s overall goal was achieved
Builtin.ToolSelection	TOOL_CALL	Whether the agent selected appropriate tools
Builtin.Correctness	TRACE	Factual accuracy of the response

For domain-specific requirements, you can create custom evaluators using an LLM-as-a-judge configuration. Every agent has domain constraints that built-in evaluators don’t cover, including data freshness, access scoping, forbidden actions, latency budgets, and cost limits.

The companion repository includes five custom Evaluator subclasses you can adapt:

Evaluator	What it validates
DataFreshnessEvaluator	Validates auction-cycle timestamps so the agent doesn’t surface stale inventory
SafetyGuardrailEvaluator	Blocks the agent from attempting automated bidding actions
DealerDataScopingEvaluator	Enforces dealer-scoped queries for data isolation

See the companion repository for LatencyEvaluator and CostEvaluator examples.

Key metrics to track

The companion repository includes an AWS CDK stack that provisions CloudWatch dashboards and alarms. Track these metrics to monitor agent health.

Metric	Target	Alert threshold
Task completion rate	>95%	<80%
Tool selection accuracy	>95%	<90%
Helpfulness score (0-1)	>0.83	<0.58
Response latency P50 / P99	<2s / <10s	>5s / >15s
Hallucination rate	<2%	>5%
Cost per interaction	Monitor trend	>2x baseline

Make quality checks a deployment gate

Evaluation should be a quality gate in your deployment pipeline, not an afterthought. Figure 4 shows the deployment pipeline with evaluation gates, spanning build-time evaluation, staging validation, shadow mode, A/B testing, and production rollout, with failures blocking deployment.

The deployment pipeline follows five phases:

1. Build-time evaluation : Unit tests, tool correctness (ToolSelectionGrader >95 percent), trajectory tests, and LLM-as-judge scoring (HelpfulnessEvaluator >85 percent).
2. Staging validation : On-demand AgentCore evaluation with synthetic traffic against staging data.
3. Shadow mode : Real production traffic processed in parallel without user impact. Run this for at least 4 hours with a 2 percent deviation threshold.
4. A/B testing : 5 percent of live traffic routes to the candidate agent for real outcome measurement.
5. Production rollout : 100 percent traffic with continuous online evaluation and monitoring.

You define thresholds for each phase: tool selection accuracy below 95 percent or task completion below 80 percent blocks deployment. For major releases, multi-trial evaluation with num_trials=5 gates on pass^k to catch non-deterministic failures.

Shadow mode

Between staging and production, run the candidate agent against real queries without affecting users. Shadow mode receives a copy of production traffic, processes it through the candidate in parallel, and compares outcomes. Run shadow mode for at least 4 hours before proceeding to A/B testing. Define a deviation threshold (2 percent is a good starting point) that pauses deployment automatically.

Shadow mode catches issues that the other stages miss:

• Timeout handling under concurrent load.
• Domain terminology missing from synthetic tests.
• Tool call ordering that causes latency spikes under real traffic patterns.

Getting started: A phased approach

Phase 1: Build your test suite. Start with 20–50 test cases drawn from real user queries. Include positive and negative cases. Test what the agent should do and what it should refuse.

Phase 2: Configure build-time evaluation. Match grader types to what you’re measuring: deterministic for tool selection, LLM-as-judge for reasoning and output quality.

Phase 3: Enable production monitoring. Configure AgentCore Evaluations with a 1–5 percent sampling rate. Start low, then increase once you’ve confirmed evaluator costs are acceptable.

Phase 4: Close the feedback loop. Turn production failures into test cases. When failures become regression tests, this feedback loop has consistently improved evaluation quality across teams.

Results and impact

Before implementing this evaluation pipeline, the agent had an 87% tool selection accuracy- which meant 1 in 8 dealer queries returned wrong results. The team was seeing 12 production incidents per month, and it took an average of 4 hours to detect issues after they started affecting dealers.

After implementing the pipeline:

Metric	Before	After
Tool selection accuracy	87%	98%
Task completion rate	82%	96%
Context retention (multi-turn)	71%	94%
Production incidents (monthly)	12	2
Mean time to detect issues	few hours	few minutes

The business impact: dealers now complete vehicle searches in minutes instead of few hours, with confidence that results are accurate and current.

Troubleshooting

See the companion repository README for solutions to common issues including trajectory grader failures, LLM-as-judge variance, empty evaluation results, and cost threshold tuning.

Key takeaways

Keep these principles in mind when building your evaluation pipeline.

• Layer your evaluation : Tool usage (>95 percent), reasoning (>85 percent), and output quality (>90 percent) catch different failure modes.
• Gate on pass^k, not single trials : A 75 percent per-trial success rate means only 42 percent reliability across three consecutive runs.
• Turn production failures into test cases : Let real user behavior grow your evaluation suite.
• Shadow mode catches what synthetic tests miss : Real traffic reveals timeout handling, rare terminology, and latency patterns.
• Start monitoring at 1 percent sampling : Scale gradually to manage evaluator costs.

Conclusion

You’ve now seen how to build an evaluation pipeline for production AI agents on AWS, using Motorway’s dealer stock search agent as a worked example.

The core lesson: a fluent response doesn’t mean the agent did the right thing. You must verify tool selection, parameter correctness, reasoning coherence, and consistency across repeated runs. By combining build-time testing with strands-agents-evals and production monitoring with AgentCore Evaluations, you can deploy agents with evidence that they work as designed.

These patterns apply to most multi-tool, customer-facing agents: whether you’re building customer service agents that query knowledge bases and ticketing systems, financial advisory agents that pull portfolio data and market feeds, or healthcare triage agents that access patient records and scheduling tools.

To get started:

1. Clone the companion repository:

   git clone https://github.com/aws-samples/sample-evaluating-agents-on-aws-with-strands-and-agentcore
   

2. Navigate to the CDK project directory:

   cd sample-evaluating-agents-on-aws-with-strands-and-agentcore/examples/vehicle-auction-agent/cdk
   

3. Deploy the sample infrastructure:
   Follow the repository README deployment steps to deploy the sample infrastructure. After deployment completes, verify the infrastructure by checking that all AWS CloudFormation stacks show CREATE_COMPLETE status in the AWS CloudFormation console. You should see the CloudWatch dashboard and Lambda functions listed in their respective consoles.

4. Run the sample evaluation suite against the included test cases to see the three-layer framework in action. Verify success by confirming: Layer 1 (Tool Usage) shows greater than 95% pass rate, Layer 2 (Reasoning) shows greater than 85% pass rate, and Layer 3 (Output Quality) shows greater than 90% pass rate. If a layer fails, check the CloudWatch logs for detailed error messages.

5. Customize the evaluators for your domain. Start with the DataFreshnessEvaluator and SafetyGuardrailEvaluator as templates.

To dive deeper, explore the Amazon Bedrock AgentCore documentation and the Strands Agents SDK on GitHub.

Clean up resources

To avoid incurring ongoing charges, delete the resources created in this walkthrough:

1. Navigate to the CDK project directory:

   cd sample-evaluating-agents-on-aws-with-strands-and-agentcore/examples/vehicle-auction-agent/cdk
   

2. Destroy all deployed stacks:

3. Confirm the deletion when prompted.

4. Verify in the AWS Management Console that the cleanup process removed all resources, including Lambda functions, S3 buckets, DynamoDB tables, CloudWatch log groups and dashboards, EventBridge rules, and SNS topics.

Note: S3 buckets that contain objects may require manual deletion. Export any evaluation data or logs that you need to retain before running the cleanup.

Resources

---

About the authors

Amit Deol

Amit is a Senior Prototyping Architect at AWS Prototyping and Cloud Engineering (PACE). He partners with AWS customers to experiment with new ideas and build production-ready solutions across generative AI, data and analytics, and real-time streaming. When he’s not prototyping, you’ll find him on long walks through the woods.

Hin Yee Liu

Hin Yee is a Senior Prototype Engagement Manager at AWS. She leads customer engagements that accelerate the path from AI prototypes to production, helping teams adopt best practices for building and operating generative AI workloads on AWS.

Ryan Cormack

Ryan Principal Engineer at Motorway and an AWS Community Builder. He leads development of AI-powered tools for dealers and has spoken at AWS Summit London on Motorway’s adoption of agentic AI.