使用 Strands Evals 和 Amazon Bedrock AgentCore 评估具备技能的 Agent

通用型 Agent 能够处理广泛的任务，但你仍需要它们遵循支撑业务运行的流程：合规检查、文档处理工作流、升级策略和工程规范。将所有这些内容编码在一条系统提示词或应用逻辑中，会变得难以维护和更新。Skills 是一种模块化替代方案。Skill 是一组可复用的指令，通常存储在 SKILL.md 文件中，用于教会 Agent 执行特定领域任务，例如脱敏合同、核对发票，或遵循团队的 pull request 规范。由于 Skills 遵循开放的 Agent Skills 标准，它们可在兼容的运行框架之间移植；Agent 只会在运行时加载所需的 Skill，而无需在核心指令中携带全部流程。

一个 Skill 将一个或多个工具与 Agent 正确使用这些工具所需的上下文打包在一起：

• 指令：注入 Agent 上下文中的领域专属指导和约束。
• 工具绑定：Skill 依赖的 API、Model Context Protocol（MCP）服务器或本地命令。
• 知识：参考材料和已完成的示例。
• 工作流：Skill 所遵循的多步骤流程或决策逻辑。
• 护栏：格式要求、范围限制和验证规则。

这种模块化方法可帮助团队更快地专门化 Agent，在不同 Agent 和工作流间复用经过验证的流程，保持行为一致，并且无需微调底层模型或重写 Agent 核心逻辑，即可更新领域专属指导。

Agent 中这种 Skill 可组合性引入了两种通用输出质量指标可能遗漏的失败模式：Agent 调用了不适合该任务的 Skill；以及 Agent 调用了正确的 Skill，却跳过或仅部分遵循其指令。这两种失败都可能生成流畅、看似合理的回答，却没有使用你预先确定的领域知识。因此，评估不能只检查最终回答。

为了使这些失败可被度量，Strands Evals SDK 和 Amazon Bedrock AgentCore Evaluations（Amazon Bedrock AgentCore 的一项功能）新增了面向 Skill 的评估器：

• Skill Selection Accuracy 确定每个已调用的 Skill 是否适合该任务。它会为每个已调用的 Skill 返回二元结果。
• Skill Instruction Following 确定 Agent 在多大程度上遵循了已调用 Skill 的指令。它会依据每个规定步骤的证据返回五级评分。
• 此外，在 Strands Evals 中，Skill Invoked 是一项确定性检查，用于验证指定名称的 Skill 是否已成功加载。

在本文中，你将了解如何通过 AgentCore CLI：从 Strands Evals 中的录制轨迹评估 Skill 选择和指令遵循情况、向测试套件添加确定性的路由检查、使用 AgentCore Evaluations 从 OpenTelemetry 追踪中评估 Skill 行为，以及解读按 Skill 划分的结果以选择正确的修复措施。

了解每个评估器衡量的内容

Agent 接收任务和 Skill 目录，选择一个 Skill，将其加载并执行操作。此次运行会在 Strands Evals 中记录为轨迹，或在可观测性层中记录为 OpenTelemetry 追踪。现在，这一记录可用于全部三个 Skill 评估器。Skill Selection Accuracy 会检查每个已调用的 Skill 是否适合任务，以及 Agent 是否调用了正确的 Skill。下图展示了 Agent 如何从提供给它的 1:n 个 Skills 中选择一个 Skill。随后，Skill Selection Accuracy 会评估所选 Skill 对该任务是否正确。你可以在提示词模板文档中查看该评估器的提示词模板和评分标准。

图 1：Skill Selection Accuracy 检查 Agent 是否为任务选择了合适的 Skill

Skill Instruction Following 用于判断 Agent 在多大程度上遵循了该 Skill 规定的步骤。你可以在提示词模板文档中查看该评估器的提示词模板和评分标准。

图 2：Skill Instruction Following 衡量 Agent 对 Skill 步骤的遵循完整度

SkillInvoked 是确定性的。它不会调用模型，并且仅适用于 Strands Evals。

下图以图示方式概述了这些 Skill 评估器。

图 3：三个 Skill 评估器概览

以一个具备带薪休假（PTO）规划和员工福利咨询 Skills 的 HR 助理 Agent 为例。一名员工询问其牙科和视力福利。如果 Agent 调用了福利 Skill，它或许能生成更完善的回答。若工具调用成功，但 Agent 选择了错误的操作手册，Skill Selection Accuracy 就能将这一路由决策单独识别出来。

再假设 Agent 针对一项相关请求正确调用了 PTO 规划 Skill。该 Skill 指示 Agent 识别 employee_id、检查 PTO 余额、依据最新 HR 政策检查结转规则，并在条件允许时提交 PTO 申请。如果 Agent 检查了 PTO 余额却跳过结转规则，它可能仍会返回看似合理的回答，但会违反规定流程。Skill Instruction Following 会将这一执行失败单独识别出来，并指出被跳过的步骤。

这些失败需要不同的修复方式。不恰当的选择通常意味着 Skill 描述之间存在重叠或歧义。指令遵循不完整则可能需要更清晰的步骤、不同的 Skill 结构，或能力更强的 Agent 模型。

评估器	可用性	评分	回答的问题
Skill Selection Accuracy	Strands Evals 和 AgentCore Evaluations	二元，每个已调用 Skill 一项	调用此 Skill 是否适合该任务？
Skill Instruction Following	Strands Evals 和 AgentCore Evaluations	五级，每个已调用 Skill 一项	Agent 在多大程度上遵循了该 Skill 规定的步骤？
Skill Invoked	Strands Evals	二元，确定性	是否成功加载了这一指定名称的 Skill？

由于基于裁判模型的评估器会返回按已调用 Skill 划分的结果，多 Skill 运行仍然可诊断：你可以确定是哪项选择或指令遵循结果拉低了聚合得分。如果没有调用任何 Skill，基于裁判模型的评估器不会产生评分。当回归测试具有已知的路由要求时，可将它们与 SkillInvoked 搭配使用。

前提条件

• Python 3.10 或更高版本。
• 一个具有 Amazon Bedrock 访问权限的 AWS 账户，以及拥有裁判模型 InvokeModel 权限的凭证。

如需跟随 Strands Evals 部分操作，请安装 SDK：

pip install strands-agents-evals strands-agents

你还需要一条已录制的 Agent 运行记录。Skill 评估器可接受 Strands Evals Session 或原始消息列表作为轨迹。在发布时，Skill 提取支持识别来自 Strands AgentSkills 插件、Claude Code、Claude Agent SDK、OpenAI Agents SDK、Codex、Gemini CLI、OpenHands、Google ADK 以及通用 SKILL.md 文件读取的信号。

如需跟随 AgentCore Evaluations 部分操作，你需要：

• 一个托管在 Amazon Bedrock AgentCore runtime 或其他位置的 Agent。我们将使用 HR 助理 Agent 示例，你可以将其部署到自己的账户中。
• 为该 Agent 启用可观测性，使其将遥测数据发送至 Amazon CloudWatch。
• 在 CloudWatch 中启用 Transaction Search。

本文示例使用 AgentCore CLI：

npm install -g @aws/agentcore

使用 Strands Evals 评估录制轨迹

当你能够控制测试用例，并且可以在开发或持续集成期间重新运行 Agent 时，Strands Evals 非常有用。请在完整代码示例中查看为 HR 助理 Agent 创建的完整 Strands evals 代码示例。

1. 定义用例和评估器

from strands_evals import Case, Experiment
from strands_evals.evaluators import (
    SkillInstructionFollowingEvaluator,
    SkillInvoked,
    SkillSelectionAccuracyEvaluator,
)

case = Case(
    name="q3-revenue-tables",
    input="Summarize the revenue tables in q3-report.pdf",
)

evaluators = [
    SkillSelectionAccuracyEvaluator(),
    SkillInstructionFollowingEvaluator(),
    SkillInvoked(skill_name="pdf-table-extraction"),
]

2. 运行 Agent 并捕获其轨迹

Skill 评估器读取运行的轨迹。TracedHandler 会收集 Agent 的 spans，并将其作为轨迹附加：

from strands import Agent
from strands_evals import TracedHandler, eval_task

@eval_task(TracedHandler())
def task_function():
    return Agent(...) # your skill-equipped agent

experiment = Experiment(cases=[case], evaluators=evaluators)
report = experiment.run_evaluations(task_function)
report.run_display()

3. 解读报告

假设 Agent 加载了 pdf-table-extraction，运行了 pdftotext -layout，但从未打开提取出的文件，也没有定位表格边界。简化后的报告如下：

SkillSelectionAccuracyEvaluator: score=1.00, pass=True
pdf-table-extraction: The skill directly matches the request.
SkillInstructionFollowingEvaluator: score=0.50, pass=False
pdf-table-extraction: The extraction phase was completed,
the table-boundary phase was skipped.
Steps:
- Extract text with layout preservation: covered
- Locate table boundaries: skipped
- Summarize each table's headline figure: partial
SkillInvoked: score=1.00, pass=True
skill 'pdf-table-extraction' was invoked

Skill Instruction Following 使用五种评级：Fully Followed（1.0）、Mostly Followed（0.75）、Partially Followed（0.5）、Minimally Followed（0.25）和 Not Followed（0.0）。达到 Mostly Followed 或更高评级即为通过。

将检查转化为部署门禁

• 为特定回归用例必须调用的每个关键 Skill 添加 SkillInvoked。由于它不会调用模型，因此是一项快速的路由断言。
• 以 report.test_passes 作为构建门禁：if not all(report.test_passes): raise SystemExit(1)。
• 使用聚合得分追踪更广泛的趋势，但应在强制执行前，基于自己的用例校准阈值。

使用 AgentCore Evaluations 评估生产追踪

AgentCore Evaluations 可直接处理现有 OpenTelemetry 追踪，支持按需评估、存储 Session 的批量处理，以及对实时流量的持续采样。遥测数据按 session、trace 和 span 组织。由于 Skill 评估器在工具调用层级运行，每项结果均包含 spanContext，其中带有已识别 Skill 调用的 sessionId、traceId 和 spanId。

Skill 调用可通过两种方式识别：一种是读取 SKILL.md 文件系统文件，该方式适用于各种框架；另一种是在 Strands Agents、LangGraph Deep Agents、Google ADK 或 Claude Agent SDK 中使用原生的 Skill 加载工具。

用于自定义评估器的追踪占位符

AgentCore Evaluations 包含两个内置的、基于裁判模型的 Skill 评估器。若要评估内置评估器未涵盖的内容，请在 TOOL_CALL 层级创建自定义评估器。工具层级模板可以引用以下 Skill 占位符：

占位符	内容
{invoked_skill}	在此次工具调用中加载的 Skill 名称
{skill_content}	已加载的 SKILL.md 正文
{available_skills}	运行时提供的目录，或“(not recorded by this harness)”
{user_message}	触发此次调用的请求
{context}	对话记录

例如，以下模板检查一次 Skill 运行的一项特定属性：

## Skill instructions

{skill_content}

## Conversation record

{context}

## Evaluation Question

Did the agent complete every numbered step in the skill instructions above, in the order given? Answer Yes or No.

该占位

General-purpose agents handle a broad range of tasks, but you still need them to follow the procedures that run your business: compliance checks, document-processing workflows, escalation policies, engineering conventions. Encoding all of that in one system prompt or in application logic gets hard to maintain and update. Skills are a modular alternative. A skill is a reusable set of instructions, usually stored in a SKILL.md file, that teaches an agent a domain-specific task like redacting a contract, reconciling an invoice, or following a team’s pull-request conventions. Because skills follow the open Agent Skills standard, they are portable across compatible harnesses, and the agent loads only the skill it needs at runtime instead of carrying every procedure in its core instructions.

A skill packages one or more tools with the context an agent needs to use them correctly:

• Instructions : Domain-specific guidance and constraints injected into the agent’s context.
• Tool bindings : The APIs, Model Context Protocol (MCP) servers, or local commands the skill depends on.
• Knowledge : Reference material and worked examples.
• Workflow : The multi-step procedure or decision logic the skill follows.
• Guardrails : Format requirements, scope limits, and validation rules.

This modular approach helps teams specialize agents faster, reuse proven procedures across agents and workflows, keep behavior consistent, and update domain-specific guidance without fine-tuning the underlying model or rewriting the agent’s core logic.

This skill composability in agents introduces two failure modes that general output-quality metrics can miss: the agent invokes a skill that is not appropriate for the task and the agent invokes the right skill but skips or only partially follows its instructions. Both failures can produce a fluent, plausible response without having used your pre-determined domain knowledge. An evaluation therefore cannot examine the final response alone.

To make these failures measurable, Strands Evals SDK and Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, add skill-focused evaluators:

• Skill Selection Accuracy determines whether each invoked skill was an appropriate choice for the task. It returns a binary result for each invoked skill.
• Skill Instruction Following determines how fully the agent followed an invoked skill’s instructions. It returns a five-level rating grounded in evidence for each prescribed step.
• Additionally on Strands Evals, Skill Invoked is a deterministic check of whether a named skill has been loaded successfully.

In this post, you will learn how to evaluate skill selection and instruction following from a recorded trajectory in Strands Evals, add deterministic routing checks to a test suite, evaluate skill behavior from OpenTelemetry traces with AgentCore Evaluations, and interpret per-skill results to choose the right fix, all through the AgentCore CLI.

Understand what each evaluator measures

An agent receives a task and a catalog of skills, chooses a skill, loads it, and acts. The run is recorded as a trajectory in Strands Evals or an OpenTelemetry trace in your observability layer. This record can now be used for all three skill evaluators. Skill Selection Accuracy checks whether each invoked skill fits the task and whether the agent invoked the correct skill. The following figure shows how the agent chooses a skill from the 1:n skills provided to it. Skill Selection Accuracy then scores whether the selected skill is the correct one for the task. You can find the prompt template and the rubric of this evaluator in the prompt template documentation.

Figure 1: Skill Selection Accuracy checks whether the agent chose an appropriate skill for the task

Skill Instruction Following asks how fully the agent followed that skill’s prescribed steps. You can find the prompt template and the rubric for this evaluator in the prompt template documentation.

Figure 2: Skill Instruction Following measures how completely the agent followed the skill’s steps

SkillInvoked is deterministic. It calls no model and is specific to Strands Evals.

An overview of these skill evaluators is demonstrated diagrammatically in the following figure.

Figure 3: Overview of the three skill evaluators

Consider an HR assistant agent with skills for paid time off (PTO) planning and discussing employee benefits. An employee asks about their dental and vision benefits. If the agent invokes the benefits skill, it may produce a more polished response. If the tool call succeeded but the agent chose the wrong playbook, Skill Selection Accuracy isolates that routing decision.

Now suppose the agent correctly invokes the PTO-planning skill for a related request. The skill instructs the agent to identify the employee_id, check the PTO balance, check the rollover rules against the latest HR policy, and then submit a PTO request if the conditions allow. If the agent checks the PTO balance but skips the rollover rules, the agent might still return a plausible response while violating the prescribed process. Skill Instruction Following isolates that execution failure and identifies the skipped step.

The failures require different fixes. An inappropriate selection often points to overlapping or ambiguous skill descriptions. Incomplete instruction following might call for clearer steps, a different skill structure, or a more capable agent model.

Evaluator	Availability	Score	Question answered
Skill Selection Accuracy	Strands Evals and AgentCore Evaluations	Binary, per invoked skill	Was invoking this skill appropriate for the task?
Skill Instruction Following	Strands Evals and AgentCore Evaluations	Five levels, per invoked skill	How fully did the agent follow this skill’s prescribed steps?
Skill Invoked	Strands Evals	Binary, deterministic	Was this named skill successfully loaded?

Because judge-based evaluators return per-invoked-skill results, multi-skill runs remain diagnosable: you can identify which selection or instruction-following result lowered the aggregate score. If no skill is invoked, the judge-based evaluators don’t produce a score. Pair them with SkillInvoked when a regression test has a known routing requirement.

Prerequisites

• Python 3.10 or later.
• An AWS account with Amazon Bedrock access, and credentials with InvokeModel permission for the judge model.

To follow the Strands Evals section, install the SDKs:

pip install strands-agents-evals strands-agents

You also need a recorded agent run. The skill evaluators accept either a Strands Evals Session or a raw message list as the trajectory. At launch, skill extraction recognizes signals from the Strands AgentSkills plugin, Claude Code, Claude Agent SDK, OpenAI Agents SDK, Codex, Gemini CLI, OpenHands, Google ADK, and generic SKILL.md file reads.

To follow the AgentCore Evaluations section, you need:

• An agent hosted on Amazon Bedrock AgentCore runtime or elsewhere. We will use the example of the HR assistant agent which you can deploy in your account.
• Observability enabled for that agent, so it delivers telemetry to Amazon CloudWatch.
• Transaction Search enabled in CloudWatch.

The examples in this post use the AgentCore CLI:

npm install -g @aws/agentcore

Evaluate a recorded trajectory with Strands Evals

Strands Evals is useful when you control the test cases and can rerun the agent during development or continuous integration. Check out the complete Strands evals code sample created for the HR assistant agent in the complete code sample.

1. Define the case and evaluators

from strands_evals import Case, Experiment
from strands_evals.evaluators import (
    SkillInstructionFollowingEvaluator,
    SkillInvoked,
    SkillSelectionAccuracyEvaluator,
)

case = Case(
    name="q3-revenue-tables",
    input="Summarize the revenue tables in q3-report.pdf",
)

evaluators = [
    SkillSelectionAccuracyEvaluator(),
    SkillInstructionFollowingEvaluator(),
    SkillInvoked(skill_name="pdf-table-extraction"),
]

2. Run the agent and capture its trajectory

The skill evaluators read the run’s trajectory. TracedHandler collects the agent’s spans and attaches them as the trajectory:

from strands import Agent
from strands_evals import TracedHandler, eval_task

@eval_task(TracedHandler())
def task_function():
    return Agent(...) # your skill-equipped agent

experiment = Experiment(cases=[case], evaluators=evaluators)
report = experiment.run_evaluations(task_function)
report.run_display()

3. Interpret the report

Suppose the agent loaded pdf-table-extraction, ran pdftotext -layout, but never opened the extracted file or located the table boundaries. A simplified report:

SkillSelectionAccuracyEvaluator: score=1.00, pass=True
pdf-table-extraction: The skill directly matches the request.
SkillInstructionFollowingEvaluator: score=0.50, pass=False
pdf-table-extraction: The extraction phase was completed,
the table-boundary phase was skipped.
Steps:
- Extract text with layout preservation: covered
- Locate table boundaries: skipped
- Summarize each table's headline figure: partial
SkillInvoked: score=1.00, pass=True
skill 'pdf-table-extraction' was invoked

Skill Instruction Following uses five ratings: Fully Followed (1.0), Mostly Followed (0.75), Partially Followed (0.5), Minimally Followed (0.25), and Not Followed (0.0). It passes at Mostly Followed or better.

Turn the checks into a deployment gate

• Add SkillInvoked for every critical skill that a specific regression case must invoke. Because it doesn’t call a model, it is a fast routing assertion.
• Gate the build on report.test_passes: if not all(report.test_passes): raise SystemExit(1).
• Use aggregate scores to track broader trends, but calibrate the threshold on your own cases before enforcing it.

Evaluate production traces with AgentCore Evaluations

AgentCore Evaluations works directly with existing OpenTelemetry traces, supporting on-demand evaluation, batch processing of stored sessions, and continuous sampling of live traffic. Telemetry is organized into sessions, traces, and spans. Because skill evaluators operate at the tool-call level, each result includes a spanContext with the sessionId, traceId, and spanId of the recognized skill invocation.

A skill invocation is recognized through either a SKILL.md filesystem read, which works across frameworks, or a native skill-loading tool in Strands Agents, LangGraph Deep Agents, Google ADK, or the Claude Agent SDK.

Trace placeholders for custom evaluators

AgentCore Evaluations includes two built-in judge-based evaluators for skills. To score something the built-ins don’t cover, create a custom evaluator at the TOOL_CALL level. Tool-level templates can reference skill placeholders:

Placeholder	Contents
{invoked_skill}	Name of the skill loaded on this tool call
{skill_content}	The loaded SKILL.md body
{available_skills}	The catalog offered at runtime, or “(not recorded by this harness)”
{user_message}	The request that triggered the invocation
{context}	The conversation record

For example, a template that checks one specific property of a skill run:

## Skill instructions

{skill_content}

## Conversation record

{context}

## Evaluation Question

Did the agent complete every numbered step in the skill instructions above, in the order given? Answer Yes or No.

The placeholders you reference also decide when the evaluator runs. A template containing {invoked_skill} runs only on skill-invocation spans, and one containing {skill_content} additionally requires the loaded body.

The following commands target the separate skill-enabled runtime. Substitute the runtime name from skills-evaluation/agent_config.json (agent_id or agent_arn) for <skill-runtime>. Strands.SkillInvoked is client-side only and has no CLI equivalent.

Run an on-demand evaluation

Use on-demand evaluation to investigate a session, validate a recent change, or evaluate staged traffic:

agentcore run eval \
  --runtime <skill-runtime> \
  --evaluator Builtin.SkillSelectionAccuracy Builtin.SkillInstructionFollowing \
  --session-id <session-id>

Run a batch evaluation

Use batch evaluation to score many stored sessions at once, for example to establish a baseline before a skill-catalog change:

agentcore run batch-evaluation \
  --runtime <skill-runtime> \
  --evaluator Builtin.SkillSelectionAccuracy Builtin.SkillInstructionFollowing

Configure continuous online evaluation

Online evaluation samples live traffic so you can detect behavior that a curated test set didn’t anticipate:

agentcore add online-eval \
  --name HRSkillsProductionEval \
  --runtime <skill-runtime> \
  --evaluator Builtin.SkillSelectionAccuracy Builtin.SkillInstructionFollowing \
  --sampling-rate 100 \
  --enable-on-create

Then deploy to provision the online evaluation configuration:

Continuous evaluation is particularly useful for detecting catalog drift (when a new skill overlaps with an existing description), unanticipated phrasing (when real requests differ from curated test prompts), and long-session failures (when instruction following degrades as context grows).

Best practices

When you’re evaluating agent skills, the first thing to internalize is that routing and execution are two different failure modes, and your evaluation strategy needs to separate them cleanly. If you see a high selection score but a low instruction-following score, that indicates the router picked the correct skill but it was not completely executed. The opposite pattern means the skill would have worked fine if only it had been invoked. Running these two evaluations together, rather than collapsing them into one pass/fail number, is what lets you tell those two stories apart.

Before you build anything custom, start with built-in evaluators to establish a baseline, so any custom logic you add afterward can cover the gap the baseline actually missed. For requirements where you already know the correct routing behavior, don’t rely on a judge model to catch it. Add a deterministic SkillInvoked assertion for every skill that must fire.

After you’re running evaluations, resist the urge to only look at the aggregate score. Per-step evidence is where the real diagnosis happens. It tells you whether an instruction was fully covered, partially completed, or skipped outright, and that level of detail is what turns a failing eval into an actionable fix. This is also why you should evaluate at every lifecycle stage instead of waiting for the final output. A failure at the end doesn’t tell you whether the router sent the request to the wrong place or the right skill executed poorly, and you need both signals to know what to fix.

Skill-level and end-to-end evaluation should be paired, because a skill can execute perfectly and still be the wrong skill for the request in front of it. You will reduce a lot of this ambiguity upstream by writing skill descriptions that are genuinely discriminative. The same logic applies to scope of a skill. A skill built to handle six unrelated functions doesn’t have a single definition of correct behavior, which makes it nearly impossible to evaluate consistently.

State clearly that only the path actually taken in a given run should be evaluated, otherwise untaken branches get miscounted as skipped steps and quietly corrupt your pass rates. And before you trust any of these scores, validate that your extraction pipeline is actually working against live traces.

As you scale this across tools, thresholds may not transfer cleanly. Each evaluation surface needs to be calibrated on its own terms, since a passing score on Strands Evals and a passing score on AgentCore Evaluations aren’t guaranteed to mean the same thing. And ultimately, none of this should live outside your deployment pipeline. Skill quality regressions need to block a release the same way a failing unit test would, or the evaluation work you’ve done up to that point isn’t actually protecting production.

Conclusion

Skills make agents inexpensive to specialize, but a plausible final answer does not prove that the agent selected the right procedure or followed it. Skill Selection Accuracy and Skill Instruction Following separate those failure modes and return evidence for each invoked skill. In Strands Evals, SkillInvoked adds a deterministic guard for known routing requirements. In AgentCore Evaluations, the two judge-based metrics can run on demand for a specific session, as a batch over stored sessions, or continuously over sampled traffic.

Use Strands Evals when you have test cases and recorded trajectories you can rerun. Use AgentCore Evaluations when you want to evaluate OpenTelemetry traces from staged or live agents. Many teams will use both: deterministic and judge-based gates before deployment, followed by trace-based monitoring in production.

Get started

Acknowledgements

Thank you to Ritvika Pillai, Vincent Chen, Qiaoxuan Xue, and Shoaib Javed for the AgentCore Evaluations implementation, to Po-Shin Chen for the Strands Evals review, to Anwesan Pal for early discussions on skill evaluation, to Ben Coombs for product guidance, and to everyone else who helped make this work possible.

---

About the authors

Sangmin Woo

Sangmin is an Applied Scientist at AWS AI Labs, where he conducts research and develops machine learning solutions for agentic AI, with a focus on evaluation frameworks and advancing agent behavior and performance. His interests include agentic AI, generative models, and multimodal AI. Outside of work, he enjoys traveling and exploring new places.

Bharathi Srinivasan

Bharathi is a Generative AI Data Scientist at AWS. She is passionate about Responsible AI to increase the reliability of AI agents in real-world scenarios. Bharathi guides internal teams and AWS customers on their responsible AI journey.

Shruthi Rajoli

Shruthi is a Solutions Architect at AWS based in Chicago, Illinois. She works with startups in the US East region, helping early-stage and growth-stage companies design and build scalable cloud architectures on AWS, with a focus on generative AI, agentic AI workflows, data, and migration workloads. Outside of work, she enjoys walking, yoga, and discovering new food and coffee places.

Visakh Madathil

Visakh is a Solutions Architect at AWS, working with customers and internal teams to bring legibility, trust, and reliability to production artificial intelligence (AI). His work on agentic reliability and AI safety has been presented at machine learning conferences. Outside of work, he enjoys music, birding, and sports.

Renu Rozera

Renu is a Software Development Engineer at Amazon Web Services, where she works on Amazon Bedrock AgentCore. She previously helped build AgentCore Memory and now focuses on developing scalable systems for AgentCore Evaluations & optimization, helping customers assess and continuously improve the quality of their agentic applications.

Vinayak Arannil

Vinayak is a Sr. Applied Scientist at Amazon Web Services. With several years of experience, he has worked on various domains of AI like computer vision, natural language processing, recommendation systems etc. Currently, Vinayak helps build new capabilities on the AgentCore and Strands, enabling customers to evaluate their Agentic applications with ease, accuracy and efficiency.

Haibo Ding

Haibo is a Principal Applied Scientist and Manager working on agentic AI at Amazon. He holds a Ph.D. from the University of Utah. His work focuses on large language models (LLMs) and AI agents, where he leads research in areas such as agent evaluation, agent tool optimization, prompt optimization, and model routing. He has served as an area chair for conferences such as AAAI and ACL, and previously as Program Chair for KDD 2025 Workshop on Prompt Optimization.

Jonathan Buck

Jonathan is a Senior Software Engineer at AWS. He builds agent environments, evaluation frameworks, and post-training infrastructure that help turn advances in agentic AI into reliable production systems.