使用 Amazon Bedrock AgentCore Evaluations 评估任意 Agent 框架

构建生产级 Agent 的 AI 团队面临一种令人沮丧的不对称：Agent 框架的多样性持续增长，但评估工具却未能同步跟上。大多数评估系统假定你以某种特定方式构建 Agent：使用特定 SDK、特定大语言模型（LLM）客户端以及特定追踪模式。一旦超出这个狭窄的兼容范围，评估管道就会失效。

团队使用 LangGraph，是因为其工作流编排模型；使用 LlamaIndex，是因为它与检索管道紧密集成；而当组织标准化采用 GPT 模型时，则使用 OpenAI Agents SDK。他们使用 Google ADK 进行多 Agent 协调，或使用 Claude Agent SDK 获取原生 Anthropic 能力。他们选择 Strands Agents，是因为其模型驱动循环能在数分钟而非数天内，让可用的 Agent 在 Amazon Bedrock AgentCore 上运行起来。而且，越来越多团队将这些框架全部部署在 Amazon Bedrock AgentCore runtime 上，这是 Amazon Bedrock AgentCore 的一项能力。它负责处理托管、扩缩容、内存和可观测性基础设施，否则团队需要为每个项目重复构建这些能力。

Amazon Bedrock AgentCore evaluations 通过将评估与框架选择解耦，解决了这种碎片化问题。每个主流框架都支持 OpenTelemetry，无论是原生支持还是通过社区仪表化库支持。只要 Agent 的遥测数据通过 OpenTelemetry 流动，评估服务就可以对它进行评分，无论底层使用什么 SDK。本文将说明其工作方式：服务读取哪些遥测数据、如何决定读取 span 的方式、哪些属性承载评估数据，以及覆盖范围如何扩展至上述列表之外的框架。

OpenTelemetry：共同语言

OpenTelemetry 是一个供应商中立的仪表化框架，对分布式系统如何输出追踪、指标和日志进行了标准化。一个 trace 是一棵由 span 组成的树，每个 span 代表一次请求中的一个单独步骤：一个工作单元，包含其名称、时间戳、一组带类型的属性以及可选的 span 事件。span 通过 OpenTelemetry Protocol（OTLP）导出，并由遥测后端收集。在 AgentCore runtime 上，该后端是 AWS Distro for OpenTelemetry（ADOT），它将 span 和事件记录路由至 Amazon CloudWatch。

由于 Agent 会执行多种类型的工作，因此一次 Agent 执行会产生多种 span。一次用户交互可能生成模型调用、工具调用、从向量存储或数据库检索文档、对检索结果进行重排序、生成嵌入、护栏检查、提示词模板渲染、记忆读写，以及将这些操作串联起来的编排步骤等 span。这两套约定对这些操作进行了明确命名：OpenTelemetry GenAI conventions 定义了 chat、embeddings、retrieval、execute_tool、invoke_agent、create_agent、plan 以及一组记忆操作等操作。OpenInference 定义了 LLM、TOOL、RETRIEVER、RERANKER、EMBEDDING、AGENT、CHAIN、GUARDRAIL、EVALUATOR 和 PROMPT 等 span 类型。生产环境的 trace 通常会混合使用其中多种类型。

在这完整的 span 集合中，评估服务需要三种 span 角色来重建会话中发生的情况并进行评分：

• 一个 调用 Agent span（invoke agent span） 代表顶层的请求—响应周期，即一次对话中的一个用户交互。它携带用户提示词和最终 Agent 响应。
• 推理 span（inference spans） 代表各次独立的模型调用，每个 span 都携带传给模型的消息历史记录和模型回复。
• 执行工具 span（execute tool spans） 代表 Agent 调用的每个工具，携带工具名称、输入参数和结果。

这三类 span 就是评估器操作的对象。服务会对收到的每个 span 进行分类，从这三种角色中读取所需值，并跳过其余 span。即使更丰富的 trace 还包含检索、重排序、护栏或记忆 span，也无需进行任何特殊配置即可处理。这些 span 只是添加了评估器并不需要的上下文。随着框架和约定不断新增 span 类型，这一机制仍可向前兼容：不熟悉的 span 类型只是服务跳过的上下文，而不是错误。

Session（一个 runtimeSessionId）
└── Trace（一次用户交互，一个 trace_id）
    ├── 调用 Agent span ← 读取：用户提示词 + 最终 Agent 响应
    ├── 推理 span ← 读取：发送至模型的消息 + 模型回复
    ├── 执行工具 span ← 读取：工具名称 + 参数 + 结果
    ├── 检索器 span（上下文；评估器不需要）
    ├── 推理 span ← 读取：消息历史中包含工具结果的下一次模型调用
    └── ...（护栏、记忆、重排序器、编排 span 等）

服务读取标记出的三种 span 角色，并将其余 span 视为额外上下文。

不同框架和仪表化库会使用不同的属性名称、嵌套结构和 span 命名约定来记录这三种角色。OpenTelemetry GenAI 语义约定和 OpenInference 规范都定义了记录这三种 span 角色的模式，但它们使用不同的属性键和 span 类型词汇。AgentCore Evaluations 将两种模式桥接到相同的评估结果。

AgentCore Evaluations 如何读取你的 trace

当你按需运行评估，或通过在线评估配置运行评估时，服务会从 CloudWatch 获取 Agent 的 span 和事件记录，并重建会话。一个 会话（session） 按 session.id 分组。在会话内，每个 trace（一个 trace_id）对应一次用户交互。每次交互均由前文介绍的三种 span 类型构成。

服务对每个 span 进行分类，提取所需值，并将重建后的会话交给评估器。从此时起，评估完全与框架无关：相同的评估器，包括 GoalSuccessRate、Correctness、Helpfulness 和自定义的 LLM-as-a-judge，会以相同方式对每个框架进行评分。下图直观展示了这一过程。

图 1：从 AgentCore Runtime 经 Amazon CloudWatch 到 AgentCore Evaluations 的数据流

你无需配置其中任何内容。 每个 OpenTelemetry 仪表化库都会在其生成的 span 和事件记录上标记 scope.name，评估服务通过该值决定如何读取它们。系统会根据你安装的仪表化软件包自动启用正确的处理方式，无需修改 Agent 代码。每个受支持框架及其 scope 名称均列于 Supported agent frameworks 文档中；目前该文档覆盖 Strands Agents、LangGraph、OpenAI Agents SDK、LlamaIndex、Google ADK 和 Claude Agent SDK，其中大多数同时支持 OpenTelemetry 和 OpenInference 仪表化。

覆盖范围不止于这些已命名的框架。任何 scope 名称属于 opentelemetry.instrumentation.*（遵循 OpenTelemetry GenAI 语义约定）或 openinference.instrumentation.*（遵循 OpenInference 规范）的库，都会通过通用路径读取。在实践中，支持新框架通常只需要安装一个合规的仪表化软件包：scope 名称前缀就是库加入支持范围的方式。即使其 span 完全遵循约定，一个名为 mycompany.agent.tracing 的 scope 也不会被识别。该前缀是仪表化作者有意遵循已文档化模式的信号。

让一个会话完成端到端评估取决于两个要求。第一个是分组：Agent 的 span 必须携带 session.id 属性，其值必须与调用 Agent 时传入的 runtimeSessionId 一致。该属性让服务能够将 span 组装为 trace，并将 trace 组装为会话。在 AgentCore runtime 上，ADOT 会自动注入此属性，因此无需修改 Agent 代码。

第二个要求是，数据源必须包含消息内容，而不只是 span。对于启用了统一可观测性的 Agent（新建 Agent 默认启用），这一点会自动满足：消息内容与 span 位于同一个按 Agent 划分的日志组中，因此单个日志组即可满足要求。然而，对于仍在使用统一前配置的既有 Agent，span 会进入共享的 aws/spans 日志组。消息内容则作为相关联的事件记录，单独存储在 Agent 的日志组中。在这种配置下，如果数据源只覆盖 aws/spans，span 分类仍会成功，但消息内容会为空，任何用于评估响应质量的评估器都会返回错误。

OpenTelemetry GenAI 语义约定：驱动评估的属性

OpenTelemetry GenAI 语义约定定义了由 LLM 框架生成的 span 模式。AgentCore Evaluations 会读取其中一组特定属性，用于对 span 分类并提取评估数据。了解哪些属性携带这些数据，有助于你调试评估或编写自定义仪表化代码。

span 分类从 gen_ai.operation.name 开始：

• invoke_agent 标识调用 Agent span，即每次交互的顶层 span，携带用户提示词和最终响应。
• chat 标识推理 span。服务读取消息历史记录和模型回复。
• execute_tool 标识执行工具 span。服务读取工具名称、输入参数和结果。

当 gen_ai.operation.name 缺失时——这在某些 LlamaIndex trace 中较为常见——服务会回退使用 traceloop.span.kind：其中 workflow 映射为调用 Agent span，tool 映射为执行工具 span，llm 映射为推理 span。

工具标识来自 gen_ai.tool.name，而 gen_ai.tool.call.id 是关联 ID，用于将推理 span 上请求的工具调用与相应执行工具 span 上的结果关联起来。推理 span 上的 gen_ai.tool.definitions 属性包含经过 JSON 编码的工具模式数组，GoalSuccessRate 评估器使用它来验证 Agent 调用的工具是否属于它声明可用的工具。

消息内容会根据遥测数据的采集方式存放在两个位置之一。当遥测数据被拆分时，内容位于相关联事件记录的正文中（body.input.messages 和 body.output.messages）。当遥测数据未被拆分时，内容保留在 span 中，作为 gen_ai.input.messages 或 gen_ai.output.messages 属性，或者作为内联 span 事件。服务会从实际存放内容的位置读取数据，因此你无需知道某一部署采用了哪条路径。有关内容存放位置的完整说明，请参阅 Spans、事件记录和遥测信号。

OpenInference 约定：另一层语义抽象

OpenInference 是由 Arize AI 维护的一项开放规范，在 LlamaIndex、Haystack 和 Phoenix 生态系统中被广泛采用。它使用与 OpenTelemetry GenAI 语义约定不同的属性名称来表示 LLM 操作。包括使用 openinference-instrumentation-openai-agents 的 OpenAI Agents SDK、Google ADK 和 Claude Agent SDK 在内的多个框架都会生成 OpenInference span。

OpenInference 中的 span 分类使用 openinference.span.kind。其中，LLM、TOOL、AGENT 和 CHAIN 分别对应推理、执行工具、调用 Agent 和结构容器 span。

推理 span 的消息内容使用扁平化、索引式属性约定。输入消息遵循 llm.input_messages.{i}.message.role 和 llm.input_messages.{i}.message.content 模式，其中 i 从零开始，并随历史记录中的每条消息递增。工具结果消息还会携带额外的 llm.input_messages.{i}.message.tool_call_id，以将其关联到相应的工具调用。输出消息遵循相同的索引模式，但工具调用具有更深一层嵌套：

llm.output_messages.0.message.role = "assistant"
llm.output_messages.0.message.tool_calls.0.tool_call.function.name = "get_pto_balance"
llm.output_messages.0.message.tool_calls.0.tool_call.function.arguments = "{\"employee_id\":\"EMP-001\"}"
llm.output_messages.0.message.tool_calls.0.tool_call.id = "call_abc123"

推理 span 上的工具模式使用 llm.tools.{i}.tool.json_schema，其中每个值都是一个 JSON 字符串，可以编码普通函数模式，也可以编码 OpenAI 风格的 {"type": "function", "function": {...}} 包装结构。服务能够处理这两种形式，并读取 name、description 和 parameters 字段。

执行工具 span 使用三个属性：tool.name 作为标识符，input.value 作为参数（对于多参数工具，为 JSON 编码对象；对于单参数工具，则为普通字符串），以及 output.value 作为结果。output.value 的格式在不同版本的 OpenInference 仪表化库中不断演变，服务同时支持较旧和较新的结构，因此你无需绑定到特定版本。

评估任何合规框架

已命名的框架可得到直接支持，但该设计有意保持开放。对于该列表之外的任意框架，只要其仪表化遵循已文档化的约定，前文介绍的两条通用路径就能提供广泛覆盖：

• OpenInference 路径处理 openinference.instrumentation.* 下的任何 scope。它使用 openinference.span.kind 对 span 分类，从带索引的 llm.input_messages 和 llm.output_messages 属性读取推理内容，并读取 tool.name、input.value，以及

AI teams building production agents face a frustrating asymmetry: the diversity of agent frameworks keeps growing, but evaluation tooling has not kept pace. Most evaluation systems assume you built your agent in a specific way: a specific SDK, a specific large language model (LLM) client, a specific tracing pattern. The moment you step outside that narrow compatibility zone, the evaluation pipeline breaks.

Teams build on LangGraph for its workflow orchestration model, on LlamaIndex for its tight integration with retrieval pipelines, and on the OpenAI Agents SDK when their organization standardizes on GPT models. They use Google ADK for multi-agent coordination, or the Claude Agent SDK for native Anthropic capability. They reach for Strands Agents because its model-driven loop gets a working agent running on Amazon Bedrock AgentCore in minutes rather than days. And increasingly, they deploy all of these on Amazon Bedrock AgentCore runtime, a capability of Amazon Bedrock AgentCore. It handles the hosting, scaling, memory, and observability infrastructure they would otherwise rebuild for each project.

Amazon Bedrock AgentCore evaluations solves this fragmentation by decoupling evaluation from the framework choice. Every major framework supports OpenTelemetry, either natively or through a community instrumentation library. As long as an agent’s telemetry flows through OpenTelemetry, the evaluation service can score it, regardless of what SDK sits underneath. This post explains how that works: what telemetry the service reads, how it decides how to read your spans, which attributes carry the evaluation data, and how coverage extends to frameworks beyond the named list.

OpenTelemetry as the common language

OpenTelemetry is a vendor-neutral instrumentation framework that standardizes how distributed systems emit traces, metrics, and logs. A trace is a tree of spans, and each span represents a single step within a request: one unit of work with its name, timestamps, a set of typed attributes, and optional span events. Spans are exported over the OpenTelemetry Protocol (OTLP) and collected by a telemetry backend. On AgentCore runtime, that backend is the AWS Distro for OpenTelemetry (ADOT), which routes spans and event records to Amazon CloudWatch.

An agent’s execution produces many kinds of span, because an agent does many kinds of work. A single user turn can generate spans for model calls, tool calls, document retrieval from a vector store or database, reranking of retrieved results, embedding generation, guardrail checks, prompt-template rendering, memory reads and writes, and the orchestration steps that tie them together. The two conventions name these explicitly: the OpenTelemetry GenAI conventions define operations such as chat, embeddings, retrieval, execute_tool, invoke_agent, create_agent, plan, and a family of memory operations. OpenInference defines span kinds such as LLM, TOOL, RETRIEVER, RERANKER, EMBEDDING, AGENT, CHAIN, GUARDRAIL, EVALUATOR, and PROMPT. A production trace routinely mixes several of these.

Out of that full set, the evaluation service needs three span roles to reconstruct what happened in a session and score it:

• An invoke agent span represents the top-level request-response cycle, which is one user turn in a conversation. It carries the user prompt and the final agent response.
• Inference spans represent individual model calls, each carrying the message history passed to the model and the model’s reply.
• Execute tool spans represent each tool the agent called, carrying the tool name, input parameters, and result.

These three are what the evaluators operate on. The service classifies every span it receives, reads the values it needs from these three roles, and passes over the rest. Richer traces that also carry retrieval, reranking, guardrail, or memory spans are handled without any special configuration. Those spans simply add context the evaluators don’t require. As frameworks and the conventions add new span kinds over time, this stays forward-compatible: an unfamiliar span kind is context the service skips, not an error.

Session (one runtimeSessionId)
└── Trace (one user turn, one trace_id)
    ├── invoke agent span ← read: user prompt + final agent response
    ├── inference span ← read: messages to model + model reply
    ├── execute tool span ← read: tool name + parameters + result
    ├── retriever span (context; not required by evaluators)
    ├── inference span ← read: next model call with tool result in history
    └── ... (guardrail, memory, reranker, orchestration spans, and more)

The service reads the three marked span roles and treats the others as additional context.

Different frameworks and instrumentation libraries record these three roles using different attribute names, nesting structures, and span-naming conventions. The OpenTelemetry GenAI semantic conventions and the OpenInference specification both define schemas for recording these three span roles, but they use different attribute keys and different span-kind vocabularies. AgentCore Evaluations bridges both schemas to the same result.

How AgentCore Evaluations reads your traces

When you run an evaluation, on demand or through an online evaluation config, the service fetches your agent’s spans and event records from CloudWatch and reconstructs the session. A session is grouped by session.id. Within it, each trace (one trace_id) is one user turn. Every turn is made up of the three span types described earlier.

The service classifies each span, extracts the values it needs, and hands the reconstructed session to the evaluators. From that point on, evaluation is entirely framework-agnostic: the same evaluators, GoalSuccessRate, Correctness, Helpfulness, and custom LLM-as-a-judge, score every framework identically. This is shown visually in the following figure.

Figure 1: Data flow from AgentCore Runtime through Amazon CloudWatch to AgentCore Evaluations

You don’t configure any of this. Every OpenTelemetry instrumentation library stamps a scope.name on the spans and event records it produces, and the evaluation service uses that value to decide how to read them. The correct handling activates automatically from the instrumentation package you installed, with no changes to your agent code. Each supported framework and its scope name is listed in the Supported agent frameworks documentation, which today covers Strands Agents, LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, and the Claude Agent SDK, most with both OpenTelemetry and OpenInference instrumentation.

Coverage reaches beyond the named frameworks. Any library whose scope name falls under opentelemetry.instrumentation.* (following the OpenTelemetry GenAI semantic conventions) or openinference.instrumentation.* (following the OpenInference specification) is read through a generic path. In practice, supporting a new framework is usually only a matter of installing a compliant instrumentation package: the scope-name prefix is how a library opts in. A scope named mycompany.agent.tracing will not be picked up, even if its spans follow the conventions exactly. The prefix is the signal that the instrumentation author deliberately conformed to a documented schema.

Getting a session evaluated end to end comes down to two requirements. The first is grouping: your agent’s spans must carry a session.id attribute that matches the runtimeSessionId you invoked the agent with. That attribute is what lets the service assemble spans into traces and traces into a session. On AgentCore runtime, ADOT injects this attribute automatically, so no agent code changes are needed.

The second requirement is that your data source includes the message content, not only the spans. For agents with unified observability (the default for newly created agents), this is automatic: message content lives in the same per-agent log group as the spans, so a single log group is sufficient. For existing agents still running the pre-unified configuration, however, spans land in the shared aws/spans log group. Message content is stored separately as correlated event records in the agent’s log group. In that configuration, if your data source covers only aws/spans, span classification still succeeds, but message content comes back empty, and any evaluator that scores response quality returns an error.

OpenTelemetry GenAI semantic conventions: Attributes that drive evaluation

The OpenTelemetry GenAI semantic conventions define a schema for spans produced by LLM frameworks. AgentCore Evaluations reads a specific subset of these attributes to classify spans and extract evaluation data. Knowing which attributes carry that data is useful when you are debugging an evaluation or writing custom instrumentation.

Span classification starts with gen_ai.operation.name:

• invoke_agent marks the invoke agent span, the top-level span per turn, carrying the user prompt and final response.
• chat marks an inference span. The service reads the message history and the model’s reply.
• execute_tool marks an execute tool span. The service reads the tool name, input parameters, and result.

When gen_ai.operation.name is absent, common in some LlamaIndex traces, the service falls back to traceloop.span.kind, where workflow maps to the invoke agent span, tool to an execute tool span, and llm to an inference span.

Tool identity comes from gen_ai.tool.name, and gen_ai.tool.call.id is the correlation ID that links a tool call requested on an inference span to its result on the corresponding execute tool span. The gen_ai.tool.definitions attribute on inference spans carries a JSON-encoded array of tool schemas, which the GoalSuccessRate evaluator uses to verify that the tools the agent called were among those it declared available.

Message content lives in one of two places, depending on how telemetry was collected. When telemetry is split, the content is in the correlated event record body (body.input.messages and body.output.messages). When it is not split, the content stays on the span, as gen_ai.input.messages or gen_ai.output.messages attributes or as inline span events. The service reads from whichever location holds the content, so you don’t need to know which path a given deployment took. For the full picture of where content lives, see Spans, event records, and telemetry signals.

OpenInference conventions: An alternative semantic layer

OpenInference is an open specification maintained by Arize AI and widely adopted across the LlamaIndex, Haystack, and Phoenix ecosystems. It represents LLM operations with different attribute names from the OpenTelemetry GenAI conventions. Several frameworks, including the OpenAI Agents SDK with openinference-instrumentation-openai-agents, Google ADK, and the Claude Agent SDK, produce OpenInference spans.

Span classification in OpenInference uses openinference.span.kind. The values LLM, TOOL, AGENT, and CHAIN correspond to inference, execute tool, invoke agent, and structural container spans respectively.

Inference-span message content uses a flat, indexed attribute convention. Input messages follow the pattern llm.input_messages.{i}.message.role and llm.input_messages.{i}.message.content, where i starts at zero and increments for each message in the history. Tool result messages carry an additional llm.input_messages.{i}.message.tool_call_id to link them to the corresponding tool call. Output messages follow the same indexed pattern, with tool calls nested further:

llm.output_messages.0.message.role = "assistant"
llm.output_messages.0.message.tool_calls.0.tool_call.function.name = "get_pto_balance"
llm.output_messages.0.message.tool_calls.0.tool_call.function.arguments = "{\"employee_id\":\"EMP-001\"}"
llm.output_messages.0.message.tool_calls.0.tool_call.id = "call_abc123"

Tool schemas on inference spans use llm.tools.{i}.tool.json_schema, where each value is a JSON string encoding either a plain function schema or an OpenAI-style {"type": "function", "function": {...}} wrapper. The service handles both and reads the name, description, and parameters fields.

Execute tool spans use three attributes: tool.name for the ID, input.value for the parameters (a JSON-encoded object for multi-argument tools, or a plain string for single-argument tools), and output.value for the result. The output.value format has evolved across versions of the OpenInference instrumentation library, and the service reads both the older and newer shapes, so you are not tied to a specific version.

Evaluating any compliant framework

The named frameworks are covered directly, but the design is deliberately open-ended. For any framework outside that list, the two generic paths introduced earlier provide broad coverage whenever the instrumentation follows a documented convention:

• The OpenInference path handles any scope under openinference.instrumentation.*. It classifies spans with openinference.span.kind, reads inference content from the indexed llm.input_messages and llm.output_messages attributes, and reads tool.name, input.value, and output.value for tool spans. This covers most frameworks that adopt the OpenInference specification, including ones released after a given framework was named explicitly.
• The OpenTelemetry path handles any scope under opentelemetry.instrumentation.*. It classifies spans with gen_ai.operation.name, reads message content from the event record body or span, and uses gen_ai.tool.name for tool identity.

The scope-name prefix is what routes a library to one of these paths. For teams writing custom instrumentation that should be evaluated generically, naming your scope under opentelemetry.instrumentation.* or openinference.instrumentation.* is the way to opt in.

From agent code to evaluation score: A walkthrough

To make this concrete, the AgentCore samples repository provides complete working examples, for instance an HR assistant implemented with the OpenAI Agents SDK, Google ADK, LlamaIndex, and the Claude Agent SDK. Each deploys to AgentCore runtime and is evaluated with the same built-in and custom evaluators.

Instrumentation setup requires no explicit code in either agent. On AgentCore runtime, AWS Distro for OpenTelemetry (ADOT) discovers the installed instrumentation package at startup and activates it automatically. For the OpenAI Agents SDK sample, adding opentelemetry-instrumentation-openai-agents to requirements.txt is sufficient. For the LlamaIndex sample, opentelemetry-instrumentation-llamaindex does the same job. The one thing the OpenAI Agents SDK agent must avoid is calling set_tracing_disabled(True): the instrumentation hooks into the SDK’s own tracing pipeline, so disabling SDK tracing silences the evaluation spans.

For LlamaIndex specifically, agent structure matters. The agent must be built as a FunctionAgent or ReActAgent (workflow agents), not as a plain AgentExecutor. Workflow agents emit a top-level invoke agent span, which anchors the trace. Without it, there is no top-level span to reconstruct the turn from, and the session cannot be assembled from inference spans alone.

To reliably deliver telemetry to CloudWatch, flush at the end of the invocation handler before returning the response. AgentCore runtime suspends the execution environment after the handler returns, while the OTel SDK buffers telemetry in client-side batch processors that export on a periodic timer. As a result, unexported data may still be sitting in the buffers at return time, and there is no guarantee that an export completes before suspension. In practice, missing telemetry from an omitted flush is the most common source of evaluation failures. The OpenAI Agents SDK and LlamaIndex samples implement this with the following pattern:

def _flush_telemetry():
    from opentelemetry import trace as _trace
    from opentelemetry._logs import get_logger_provider as _get_lp
    for provider in (_trace.get_tracer_provider(), _get_lp()):
        flush = getattr(provider, "force_flush", None)
        if flush:
            flush()

@app.entrypoint
async def invoke(payload, context):
    prompt = payload.get("prompt", "")
    try:
        result = await run_agent(prompt)
    finally:
        _flush_telemetry()
    return str(result)

The flush must cover both the tracer provider (spans) and the logger provider (event records): spans and event records go through separate client-side batch processors, so flushing only one of them leaves the other’s buffer untouched.

AgentCore Evaluations usesEvaluationClient identically for both frameworks, after a CloudWatch ingestion wait of 90–150 seconds:

from bedrock_agentcore.evaluation import EvaluationClient
from bedrock_agentcore.evaluation.client import ReferenceInputs
from datetime import timedelta

ec = EvaluationClient(region_name=REGION)
# The client resolves each evaluator's level (SESSION, TRACE, or TOOL_CALL)
# automatically, so no level configuration is needed here.
results = ec.run(
    evaluator_ids=[
        "Builtin.GoalSuccessRate",
        "Builtin.Correctness",
        "Builtin.Helpfulness",
        CUSTOM_RESPONSE_QUALITY_ID,
        CUSTOM_SESSION_COMPLETENESS_ID,
    ],
    agent_id=AGENT_ID,
    session_id=SESSION_ID,
    look_back_time=timedelta(hours=1),
    reference_inputs=ReferenceInputs(
        assertions=ASSERTIONS,
        expected_trajectory=EXPECTED_TRAJECTORY,
        expected_response=EXPECTED_RESPONSES[-1],
    ),
)

The GoalSuccessRate evaluator operates at the session level and checks whether the agent’s tool-call history matched the expected trajectory across all turns. Correctness and Helpfulness are trace-level evaluators that score each individual turn. Built-in evaluators are configured for the session, trace, tool levels and available to use as templates. For custom evaluators that you configure, you define if it works on the session, trace or tool level. It is possible to combine evaluators at various levels into a single evaluation call. The evaluation code for the different framework samples is structurally identical. The only differences are the agent names and resource IDs.

On-demand and online evaluation modes

The sample repository demonstrates both evaluation modes, which serve distinct purposes in a production agent workflow.

On-demand evaluation is the natural fit for continuous integration and continuous delivery (CI/CD) pipelines and regression testing. You invoke the agent to generate a session, then call EvaluationClient.run() in the same script to score it. Because you control the invocation, you can supply ground truth through ReferenceInputs: expected responses, expected tool trajectories, and behavioral assertions. A pipeline that runs on every pull request can invoke a fixed set of test prompts, score them against ground truth, and fail the build if any evaluator drops below threshold.

Online evaluation monitors live agent traffic continuously. You create an OnlineEvaluationConfig once, pointing it at the agent’s CloudWatch log group and specifying a sampling rate. The service picks up new sessions automatically as they arrive:

_cp.create_online_evaluation_config(
    onlineEvaluationConfigName="hr_agent_online_eval",
    rule={"samplingConfig": {"samplingPercentage": 25.0}},
    dataSourceConfig={
        "cloudWatchLogs": {
            "logGroupNames": [CW_LOG_GROUP],
            "serviceNames": [OTEL_SERVICE_NAME],
        }
    },
    evaluators=[
        {"evaluatorId": "Builtin.GoalSuccessRate"},
        {"evaluatorId": "Builtin.Correctness"},
        {"evaluatorId": "Builtin.Helpfulness"},
    ],
    evaluationExecutionRoleArn=ONLINE_EVAL_ROLE_ARN,
    enableOnCreate=True,
)

Online evaluation can only use evaluators that do not require ground truth, since live traffic has no associated expected responses. The built-in evaluators work in both modes. Custom LLM-as-a-judge evaluators that reference {expected_response} or {assertions} placeholders in their instructions are on-demand only.

Online evaluation results land in a CloudWatch log group at /aws/bedrock-agentcore/evaluations/results/{config_id}. Building a CloudWatch alarm or dashboard from that data gives you continuous visibility into agent quality in production, across all frameworks, through a single consistent interface. Because the same evaluation pipeline processes on-demand and online traffic identically, scores from a CI test run and scores from production traffic are directly comparable when the agent’s behavior is stable.

Conclusion

AgentCore Evaluations is framework-agnostic. Whether you build with LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, the Claude Agent SDK, or Strands Agents, the same evaluators apply uniformly in on-demand and online modes, because the integration contract is built on OpenTelemetry standards. If your agent emits spans under a recognized scope name, includes tool schemas on inference spans, and surfaces message content through span attributes or event records, the evaluation service produces scores without any framework-specific code on your side. For frameworks beyond the named list, following the OpenTelemetry GenAI conventions or the OpenInference specification with the right scope prefix is all it takes. With one evaluation suite, across every agent that emits telemetry under a recognized scope, you can now assess quality consistently.

Ready to try it yourself? Get started with the AgentCore Evaluations feature samples, and refer to the Supported agent frameworks documentation for full details on connecting your framework of choice.

Acknowledgements

Thank you to the AgentCore evaluations engineering team led by Shoaib Javed and Qiaoxuan Xue, and the engineers who contributed to this work: Swarnim Singhal, Lefan Zhang, Athul Iddya, Irene Wang, Ritvika Pillai, Aditya Arepalli.

---

About the authors

Swarnim Singhal

Swarnim is a Software Development Engineer at AWS on the Amazon Bedrock AgentCore Evaluations team, where he has helped build the product’s evaluation capabilities from the ground up. He designs and builds the core services that let customers measure and improve the quality of their AI agents, with expertise spanning agent evaluation, optimization, and insights. Before AgentCore, he worked on Amazon OpenSearch Serverless, building the metadata services behind OpenSearch Serverless.

Bharathi Srinivasan

Bharathi is the technical go-to-market for Trustworthy AI at AWS, helping organizations safely move frontier AI concepts into real-world production environments. Passionate about AI reliability and developer enablement, she frequently publishes research and code samples on LLM evaluation, reliability, and security. In her free time, she enjoys exploring mountain trails, gearing up for backcountry adventures, and indulging in K-dramas.

Renya Kujirada

Renya is an AI/ML Specialist Solutions Architect at AWS Japan. He works with customers across industries to build AI agents, design agent platforms, and fine-tune LLMs. Before joining AWS, he worked as a Data Scientist developing deep learning models and building solutions powered by AI agents. He was selected as a 2025 Japan AWS Top Engineer and an AWS Community Builder.