使用 Amazon Bedrock AgentCore 评估多智能体系统的可解释性与有用性

随着多智能体系统从实验走向生产，一个关键挑战是确保这些系统在真实场景中始终具备有用性、准确性和可解释性。企业正越来越多地采用多智能体系统来解决需要跨数据源、工具和业务约束进行推理的复杂现实问题。从供应链规划到金融分析和客户运营，这些系统已经超越了简单的问答模式。它们会协调多个专业 Agent 进行决策、执行工作流，并生成可执行的建议。

虽然大语言模型能够生成流畅的回答，但企业应用需要更深层的保障：Agent 必须可靠地遵循指令、选择正确的工具、遵守约束，并清晰说明其输出背后的推理过程。

Amazon Bedrock AgentCore 是一个可使用任意框架或模型，大规模构建、连接和优化 Agent 的平台。Amazon Bedrock AgentCore Evaluations 是 Amazon Bedrock AgentCore 的一项能力，旨在应对这一挑战。它提供完全托管的能力，用于在开发和生产阶段评估 Agent 性能，使团队能够从多个质量维度衡量准确性、任务成功率和行为表现。传统仅关注模型响应质量的评估方法不足以应对 Agent 系统，因为系统正确性还取决于工具选择、工作流执行以及对业务约束的遵守。

除评估外，Agent 系统的生产部署还需要负责任 AI 控制。Amazon Bedrock Guardrails 提供可配置的安全防护措施，例如内容过滤、禁用主题检测和事实依据验证，可与评估框架形成互补。评估用于在执行后衡量 Agent 质量，而 Guardrails 则在执行过程中实施安全约束。本文重点介绍如何通过 Amazon Bedrock AgentCore Evaluations 对内置评估器和自定义评估器的支持，将这一评估框架落地运行。内置评估器为有用性、任务成功和指令遵循等常见质量维度提供预定义评估，使团队无需额外配置即可快速建立 Agent 性能基线。然而，企业用例还需要更深入、更具领域针对性的验证。自定义评估器能够满足这一需求，使你可以定义感知业务上下文的检查。

我们还特别将可解释性作为一项一级评估维度。我们展示内置评估器如何评估一般性的响应清晰度，并演示如何使用自定义评估器验证 Agent 是否明确阐述决策理由、引用支持性数据或工具输出，以及解释成本与服务水平等权衡。通过组合这些评估器，我们展示了 AgentCore Evaluations 如何超越表层响应质量，为 Agent 如何以及为何得出决策提供结构化、可量化的洞察。

为使这些概念更加具体，以下章节将介绍一个参考架构和实现，展示这些组件如何在实际场景中协同工作。

解决方案概述

本文使用一家名为 AnyCompany Retail 的虚构全球零售公司。该公司是一家跨国零售商，运营电商渠道、区域履约中心、配送中心以及数千家实体门店。AnyCompany 经常面临库存失衡问题：部分区域在促销期间出现缺货，而其他区域则持有过剩库存。运输团队还必须平衡配送速度、承运商运力和成本。该公司希望拥有一个 Agent 助手，帮助规划人员优化库存分配、推荐配送调整、分析库存健康状况，以及模拟路线或履约场景。

你将使用 Strands Agents SDK、Amazon Bedrock AgentCore MCP Server 和 Amazon Bedrock AgentCore Evaluations，构建并评估一个多智能体供应链决策系统。该解决方案使用 Strands Agents，并包含一个编排 Agent 和四个专业子 Agent：优化 Agent、配送 Agent、路线 Agent 和分析 Agent。每个 Agent 都运行在启用了 Amazon Bedrock AgentCore runtime、Amazon Bedrock AgentCore memory 和 Amazon Bedrock AgentCore Observability 的环境中。

编排 Agent 接收规划人员的请求，并将工作委派给以工具形式暴露的专业 Agent。优化 Agent 调用由模拟 Amazon API Gateway REST 接口支持的 MCP 工具，以返回优化决策。配送 Agent 调用推荐 API，为履约中心、门店和数字渠道之间的库存再平衡提供建议。路线 Agent 调用物流 API，以推荐承运商和路线选项；分析 Agent 则回答供应链诊断问题。该解决方案在 Agent 循环中使用 Amazon Bedrock 上的基础模型。有关模型在各区域的可用性，请参阅 Amazon Bedrock 中按 AWS 区域划分的支持模型。

该解决方案使用内置评估器评估有用性和任务完成等通用质量维度，同时也提供自定义评估器，用于评估供应链特定行为，例如约束满足、路线可行性、SQL 正确性、库存依据和解释质量。AnyCompany 可以同时评估响应的语言质量和 Agent 决策的业务有效性。

该解决方案通过 Amazon Bedrock AgentCore Evaluations 支持按需模式和在线模式。按需模式适用于开发基准测试、回归测试以及持续集成和持续交付（CI/CD）门禁。在线模式适用于持续性的生产监控和告警。这两种模式都可帮助你形成反馈闭环，并根据用户反馈采取行动。用于按需评估的同一批自定义评估器（例如供应链解决方案中的约束满足、路线可行性、SQL 正确性和可解释性评估器），可以通过 OnlineEvaluationConfig 对象复用。该对象引用评估器的 Amazon Resource Names（ARN），并指定采样率（例如生产追踪的 1–10%）以及可选的会话筛选条件。该服务随后会自动从 AgentCore Observability 读取追踪记录、对其评分，并将结果流式传输至 Amazon CloudWatch 仪表板和告警系统。本文将使用按需模式测试该解决方案。

下图展示了解决方案的各个组件。

图 1：多智能体供应链决策解决方案的架构

评估框架

本文将针对多智能体系统使用一种三层评估方法，以逐步建立企业信任。该方法遵循清晰的递进路径：首先使用内置评估器衡量通用质量，然后增加自定义评估器验证业务准确性，最后叠加可解释性评估器以实现可信和可审计。

第一层使用无需设置的内置评估器。我们将“有用性”作为通用基线，并为每个 Agent 添加第二个面向其主要失败模式的 Agent 专用评估器：编排 Agent 使用“工具选择准确性”，优化和配送 Agent 使用“响应相关性”，路线 Agent 使用“指令遵循”，分析 Agent 使用“忠实性”。

第二层添加对领域特定业务规则进行编码的自定义评估器：针对优化的约束满足、针对配送的数据依据、针对路线的路线可行性、针对分析的 SQL 正确性，以及针对编排的计划一致性。这些评估器验证业务有效性：该建议是否遵守预算限制、是否使用真实库存数据、是否生成了在运营上正确的输出？

下表映射了你将在本文中为每个 Agent 实现的两项内置评估器和自定义评估器。第二项内置评估器面向每个 Agent 的主要失败模式，而自定义评估器则对验证运营正确性的领域特定业务规则进行编码。

Agent	内置评估器	自定义评估器
编排 Agent	有用性；工具选择准确性	计划一致性评估器：它是否将子 Agent 输出组合为有效且不存在矛盾的建议？工具轨迹评估器：它是否路由至正确的子 Agent？
优化 Agent	有用性；响应相关性	约束满足评估器：预算、库存覆盖范围（需求 ≤ 数量 ≤ 2× 需求）和仓库容量约束。关键绩效指标（KPI）达成评估器：是否达到补货率/营收改善目标。
配送 Agent	有用性；响应相关性	推荐依据评估器：建议是否基于当前库存/需求数据。风险影响评估器：建议是否改善缺货/库存过剩风险。
路线 Agent	有用性；指令遵循	路线可行性评估器：路线是否遵守配送窗口、成本、承运商运力和区域约束。服务水平协议（SLA）评估器：预期配送是否达到目标服务水平。
分析 Agent	有用性；忠实性	SQL 正确性评估器：查询是否符合用户意图。数据依据评估器：响应是否得到 Amazon Relational Database Service（Amazon RDS）查询结果的支持。无未经支持声明评估器。

可解释性

我们评估方法的第三层将可解释性评估器作为跨 Agent 的独立检查项。这些评估器独立判断 Agent 是否阐述了决策理由、引用了工具输出中的支持性证据、解释了哪些约束影响了响应、说明了相互竞争目标之间的权衡、明确了为何调用特定子 Agent，以及在数据不完整时披露了假设。通过将可解释性划分为独立评估层，我们可以单独衡量透明度。一项建议可能准确但不可解释（通过自定义评估器但未通过可解释性评估），这会为团队提供可执行信号，帮助其判断 Agent 是否需要改进推理表达，而非决策逻辑。

下表定义了你将在本文中实现的六项独立可解释性评估器。这些评估器作为跨 Agent 的横向层应用，用于检查 Agent 是否阐述推理、引用证据、解释约束与权衡，并披露假设。它们与准确性分开衡量，使团队能够区分“不可解释但正确”的响应和“解释充分但错误”的响应。

评估器	Agent	检查内容
决策理由质量	全部	Agent 是否解释了为何提出该建议？
证据归因	分析、配送、路线	是否引用了所使用的数据字段、API 响应或 SQL 结果？
约束推理	优化、路线	是否解释了哪些约束影响了最终响应？
权衡说明	优化、配送、路线	是否解释了成本、服务水平和库存风险之间的权衡？
工具使用可解释性	编排	是否解释了为何调用每个子 Agent 或 MCP 工具？
假设披露	全部 Agent	当数据不完整时，是否清晰说明了假设？

前提条件

部署该解决方案前，请使用以下工具设置开发环境。

1. 安装 AWS Command Line Interface（AWS CLI）
2. 安装 AWS Serverless Application Model（AWS SAM）CLI v1.100.0+
3. 安装 Docker v20.x+
4. 安装 Node.js v18.x+
5. 安装 Python v3.11+

依赖项

Strands Agents 实现还需要以下依赖项，这些依赖项已打包在 DockerFile 中：

1. strands-agents # Strands Agents 多智能体框架
2. strands-agents-tools # Strands Agent 工具和实用程序
3. requests # 用于 API 调用的 HTTP 库
4. bedrock-agentcore # Amazon Bedrock Agent Core 功能
5. boto3 # AWS Python SDK（Boto3）

部署并运行解决方案

该解决方案可从我们的 GitHub repo 下载，并提供单步部署，可在你的 AWS 环境中部署和访问该解决方案：

# Edit terraform.tfvars: set vpc_id and runtime_subnet_azs
cd terraform
cp terraform.tfvars.example terraform.tfvars
terraform init
terraform apply

输出包括 runtime ARN、AnyCompany Retail API URL、评估器 API URL 和 memory ARN。

运行解决方案

按照以下步骤运行该解决方案：

test_client/ fo

A critical challenge that emerges as multi-agent systems move from experimentation to production is making sure that these systems are consistently helpful, accurate, and explainable in real-world scenarios. Enterprises are increasingly adopting multi-agent systems to solve complex, real-world problems that require reasoning across data sources, tools, and business constraints. From supply chain planning to financial analysis and customer operations, these systems go beyond simple question answering. They coordinate multiple specialized agents to make decisions, execute workflows, and generate actionable recommendations.

While large language models can generate fluent responses, enterprise applications require much deeper guarantees, where agents must follow instructions reliably, select the right tools, respect constraints, and provide clear reasoning behind their outputs.

Amazon Bedrock AgentCore is a platform to build, connect, and optimize agents at scale, with any framework or model. Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, is designed to address this challenge as a fully managed capability for assessing agent performance across development and production, so teams can measure accuracy, task success, and behavior across multiple quality dimensions. Traditional evaluation approaches that focus only on model response quality are insufficient for agentic systems, where correctness depends on tool selection, workflow execution, and adherence to business constraints. In addition to evaluation, production deployment of agentic systems requires responsible AI controls. Amazon Bedrock Guardrails provides configurable safeguards such as content filtering, denied topic detection, and grounding validation that complement the evaluation framework. While evaluations assess agent quality after execution, Guardrails enforce safety constraints during execution. In this post, we focus on operationalizing this evaluation framework with Amazon Bedrock AgentCore Evaluations support for both built-in evaluators and custom evaluators. Built-in evaluators offer pre-defined assessments for common quality dimensions such as helpfulness, task success, and instruction following, so teams can quickly baseline agent performance without additional setup. However, enterprise use cases require deeper, domain-specific validation. Custom evaluators address this, so you can define business-aware checks.

We also focus specifically on explainability as a first-class evaluation dimension. We demonstrate how built-in evaluators can assess general response clarity. We showcase how custom evaluators are used to verify that agents explicitly articulate decision rationale, reference supporting data or tool outputs, and explain tradeoffs such as cost versus service level. By combining these evaluators, we show how AgentCore Evaluations can move beyond surface-level response quality and provide structured, measurable insights into how and why agents arrive at their decisions.

To make these concepts concrete, the following sections walk through a reference architecture and implementation that demonstrates how these components work together in practice.

Solution overview

For this post, we use a fictitious global retail company called AnyCompany Retail, a multinational retailer operating ecommerce channels, regional fulfillment centers, distribution centers, and thousands of physical stores. AnyCompany experiences frequent inventory imbalances: some regions face stockouts during promotions, while others carry excess inventory. Transportation teams must also balance delivery speed, carrier capacity, and cost. The company wants an agentic assistant that can help planners optimize inventory allocation, recommend distribution adjustments, analyze inventory health, and simulate routing or fulfillment scenarios.

You will build and evaluate a multi-agent supply chain decisioning system using Strands Agents SDK, Amazon Bedrock AgentCore MCP Server and Amazon Bedrock AgentCore Evaluations. The solution uses Strands Agents with an orchestrator agent and four specialized sub-agents: an optimization agent, distribution agent, routing agent, and analytics agent. Each agent runs on Amazon Bedrock AgentCore runtime with Amazon Bedrock AgentCore memory and Amazon Bedrock AgentCore Observability enabled.

The orchestrator agent receives the planner’s request and delegates work to specialized agents exposed as tools. The optimization agent calls MCP tools backed by mock Amazon API Gateway REST interfaces that return optimization decisions. The distribution agent calls recommendation APIs to suggest inventory rebalancing across fulfillment centers, stores, and digital channels. The routing agent calls logistics APIs to recommend carrier and route options and the analytics agent answers supply chain diagnostics questions. This solution uses foundation models on Amazon Bedrock for the agent loop. For model availability by Region, refer to Supported models by AWS Region in Amazon Bedrock.

The solution uses built-in evaluators that assess general quality dimensions such as helpfulness and task completion. It also provides custom evaluators that assess supply-chain-specific behavior such as constraint satisfaction, route feasibility, SQL correctness, inventory grounding, and explanation quality. AnyCompany can evaluate both the language quality of the response and the business validity of the agent’s decision.

The solution supports both on-demand and online modes with Amazon Bedrock AgentCore Evaluations. The on-demand mode is meant for development benchmarking, regression testing, and continuous integration and continuous delivery (CI/CD) gates. The online mode is for continuous production monitoring and alerts. Both modes help you close the loop and act on feedback from your users. The same custom evaluators (such as the constraint satisfaction, route feasibility, SQL correctness, and explainability evaluators from your supply chain solution) used for on-demand evaluations are repurposed with an OnlineEvaluationConfig object that references the Amazon Resource Names (ARNs) of the evaluators and specifies a sampling rate (for example, 1–10% of production traces) along with optional session filters. The service then automatically reads traces from AgentCore Observability, scores them and streams results to Amazon CloudWatch dashboards and alarms. In this post, you will use the on-demand mode to test the solution.

The following architecture diagram illustrates the various components of our solution.

Figure 1: Architecture of the multi-agent supply chain decisioning solution

Evaluation framework

In this post, you will use a three-layer evaluation approach for multi-agent systems that progressively builds enterprise trust. The approach follows a clear progression starting with built-in evaluators for general quality then adding custom evaluators for business accuracy and finally layering explainability evaluators for trust and auditability.

The first layer uses built-in evaluators requiring no setup. We apply Helpfulness as a universal baseline plus a second agent-specific evaluator targeting each agent’s primary failure mode: Tool Selection Accuracy for the orchestrator, Response Relevance for optimization and distribution, Instruction Following for routing, and Faithfulness for analytics.

The second layer adds custom evaluators encoding domain-specific business rules: constraint satisfaction for optimization, data grounding for distribution, route feasibility for routing, SQL correctness for analytics, and plan coherence for orchestration. These validate business validity: did the recommendation respect budget limits, use real inventory data, and produce operationally correct outputs?

The following table maps the two built-in evaluators and custom evaluator selected for each agent that you will implement here. The second built-in evaluator targets each agent’s primary failure mode, while the custom evaluator encodes domain-specific business rules that validate operational correctness.

Agent	Built-in evaluators	Custom evaluators
Orchestrator agent	Helpfulness; Tool Selection Accuracy	Plan coherence evaluator: Did it combine sub-agent outputs into a valid, non-contradictory recommendation? Tool trajectory evaluator: Did it route to the correct sub-agent?
Optimization agent	Helpfulness; Response Relevance	Constraint satisfaction evaluator: budget, inventory coverage (demand ≤ qty ≤ 2× demand), and warehouse capacity constraints. Key performance indicator (KPI) attainment evaluator: fill-rate/revenue improvement target met.
Distribution agent	Helpfulness; Response Relevance	Recommendation groundedness evaluator: recommendation is grounded in current inventory/demand data. Risk impact evaluator: recommendation improves stockout/overstock risk.
Routing agent	Helpfulness; Instruction Following	Route feasibility evaluator: route respects delivery window, cost, carrier capacity, and region constraints. Service level agreement (SLA) evaluator: expected delivery meets target service level.
Analytics agent	Helpfulness; Faithfulness	SQL correctness evaluator: query matches user intent. Data-grounding evaluator: response is supported by Amazon Relational Database Service (Amazon RDS) query results. No unsupported claims evaluator.

Explainability

The third layer of our evaluation approach applies explainability evaluators as distinct, cross-cutting checks across the agents. These independently assess whether agents articulate decision rationale, cite supporting evidence from tool outputs, explain which constraints shaped the response, articulate trade-offs between competing objectives, clarify why specific sub-agents were invoked, and disclose assumptions when data is incomplete. By separating explainability into its own evaluation layer, we can independently measure transparency. A recommendation can be accurate but unexplainable (passing custom evaluators but failing explainability), giving teams actionable signals about whether agents need better reasoning articulation rather than better decision logic.

The following table defines the six independent explainability evaluators that you implement here and that are applied across agents as a cross-cutting layer. These assess whether agents articulate reasoning, cite evidence, explain constraints and trade-offs, and disclose assumptions. They are measured separately from accuracy so teams can distinguish unexplainable-but-correct responses from well-explained-but-wrong ones.

Evaluator	Agents	What it checks
Decision rationale quality	All	Did the agent explain why it made the recommendation?
Evidence attribution	Analytics, Distribution, Routing	Did it cite the data fields, API response, or SQL result used?
Constraint reasoning	Optimization, Routing	Did it explain which constraints shaped the final response?
Trade-off explanation	Optimization, Distribution, Routing	Did it explain the trade-offs among cost, service level, and inventory risk?
Tool-use explainability	Orchestrator	Did it explain why each sub-agent or MCP tool was invoked?
Assumption disclosure	All agents	Did it clearly state assumptions when data was incomplete?

Prerequisites

Before you deploy this solution, set up your development environment with the following tools.

1. Install the AWS Command Line Interface (AWS CLI)
2. Install the AWS Serverless Application Model (AWS SAM) CLI v1.100.0+
3. Install Docker v20.x+
4. Install Node.js v18.x+
5. Install Python v3.11+

Dependencies

The Strands Agents implementation also needs to have the following dependencies that are packaged in the DockerFile:

1. strands-agents # Strands Agents multi-agent framework
2. strands-agents-tools # Strands agent tools and utilities
3. requests # HTTP library for API calls
4. bedrock-agentcore # Amazon Bedrock agent core functionality
5. boto3 # AWS SDK for Python (Boto3)

Deploy and run the solution

The solution is available for download from our GitHub repo and provides a single step deployment to deploy and access the solution in your AWS environment:

# Edit terraform.tfvars: set vpc_id and runtime_subnet_azs
cd terraform
cp terraform.tfvars.example terraform.tfvars
terraform init
terraform apply

The outputs include runtime ARNs, AnyCompany Retail API URL, evaluator API URL, and memory ARN.

Run the solution

Follow these steps to run the solution:

The test_client/ folder contains a Python script that invokes the deployed Supply Chain agent with 20 sample queries (5 per sub-agent) to validate end-to-end functionality. Each category runs as a multi-turn session, and session IDs are printed at the end for use with the evaluators API.

cd test_client
pip install -r requirements.txt
cd terraform
terraform output supply_chain_arn

Run all queries (20 total, 4 sessions)

cd test_client
python test_agent.py --runtime-arn "<supply_chain_arn>" --region <your_region>

Run specific sub-agent categories

#Only optimization queries (1 session, 5 turns)
python test_agent.py --runtime-arn "<supply_chain_arn>" --category optimization
#Only routing and analytics (2 sessions, 5 turns each)
python test_agent.py --runtime-arn "<supply_chain_arn>" --category routing analytics

The test client prints each query and the full agent response. At the end it prints the session IDs for use with the evaluators.

Create and run evaluations

The test_evaluators/ folder contains scripts to run evaluations against agent sessions. These evaluations run asynchronously and the results are saved as markdown files in S3.

You must invoke the supply chain decisioning multi-agent solution first as outlined earlier to generate agent sessions with traces and note down the session IDs printed at the end of the test client run. Finally wait 3-5 minutes after running the solution for traces to propagate to CloudWatch.

cd test_evaluators
pip install -r requirements.txt
cd terraform
terraform output evaluators_api_url

Create custom evaluators

python test_evaluator.py --api-url "https://<evaluators-api-url>" create

This registers custom evaluators and prints their IDs. Save these for use with the run command.

Run evaluations

Pass a comma-separated list of evaluator IDs (custom or built-in). At least 1 is required:

# Run custom + built-in evaluators
python test_evaluator.py --api-url "https://<evaluators-api-url>" run \
--agent-id "supply_chain_orchestrator_agent-<id>" \
--session-id "<session-id-from-test-client>" \
--evaluators " sc_optimization_constraint-<id>,sc_distribution_groundedness-<id>,Builtin.Correctness,Builtin.GoalSuccessRate"

The API returns 202 immediately. Results are saved asynchronously to S3 at:

s3://<amzn-s3-demo-agent-source-bucket>/evaluations/<session-id>/<timestamp>/EvaluationResults.md

Delete evaluators

python test_evaluator.py --api-url "https://<evaluators-api-url>" delete \
--evaluator-ids "sc_optimization_constraint-<id>,sc_distribution_groundedness-<id>"

Run an optimization evaluation

Now that you understand the evaluation framework and have deployed the solution, let’s walk through a focused end-to-end evaluation of the optimization agent. This walkthrough demonstrates how to combine a custom business-accuracy evaluator (Layer 2) with explainability evaluators (Layer 3) to assess both the correctness and transparency of optimization decisions.

Step 1: Run optimization queries

First, invoke the test client with only the optimization category to generate a focused session:

cd test_client
python test_agent.py --runtime-arn "<supply_chain_arn>" \
--category optimization \
--region <your_region>

This runs 5 optimization queries as a multi-turn session. The agent processes requests such as “What is the optimal inventory level for prod-001 over the next 30 days?” Each query requires the optimization agent to call MCP tools, retrieve demand forecasts, and produce stocking recommendations that respect budget, inventory coverage, and warehouse capacity constraints. At the end of the run, the test client prints the optimization session ID.

Step 2: Run the Constraint Satisfaction evaluator (Layer 2: business accuracy)

With the optimization session generated, run the custom Constraint Satisfaction evaluator to validate whether the agent’s stocking recommendations respect business rules. This evaluator checks three constraints simultaneously:

• Budget: Does the incremental holding cost fit within remaining budget (budget_limit − budget_used)?
• Inventory coverage: Is the recommended level ≥ demand forecast (avoids stockout) and ≤ 2× demand (avoids excess)?
• Warehouse capacity: Does the recommended level fit within available warehouse space?

Run the evaluator along with the built-in Helpfulness and Response Relevance evaluators:

cd test_evaluators
python test_evaluator.py --api-url "https://<evaluators-api-url>" run \
--agent-id "supply_chain_orchestrator_agent-<id>" \
--session-id "<optimization-session-id>" \
--evaluators "sc_optimization_constraint-<id>,Builtin.Helpfulness,Builtin.ResponseRelevance"

The API returns HTTP 202 immediately. Evaluations run asynchronously. Results are saved to S3:

s3://<amzn-s3-demo-agent-source-bucket>/evaluations/<optimization-session-id>/<timestamp>/EvaluationResults.md

Step 3: Run explainability evaluators (Layer 3: trust and auditability)

After confirming that the optimization agent produces constraint-satisfying recommendations, the next question is: does it explain its reasoning? A recommendation can be accurate but opaque. Such a recommendation passes the constraint evaluator but fails to articulate why it chose a specific inventory level.

Using the same optimization session ID from Step 1, now run the two explainability evaluators that apply to the optimization agent:

• Decision Rationale Quality — Did the agent explain why it made the recommendation? (for example, “Recommending 1,500 units because demand forecast is 1,200 and we target a 1.25× safety factor”)

• Constraint Reasoning — Did the agent explain which constraints shaped the final response? (for example, “Budget allows up to 1,800 units but warehouse capacity limits us to 1,600, so we recommend 1,500”)

  python test_evaluator.py --api-url "https://" run
  --agent-id "supply_chain_orchestrator_agent-"
  --session-id ""
  --evaluators "sc_decision_rationale-,sc_constraint_reasoning-"

These evaluators independently assess transparency. They are measured separately from accuracy.

Interpreting the combined results

By running all three evaluators against the same optimization session, you get a complete picture of agent quality across two dimensions:

Dimension	Evaluator	Question answered
Business accuracy (Layer 2)	Constraint Satisfaction	Are the recommendations operationally correct?
Explainability (Layer 3)	Decision Rationale Quality	Why does the agent make this recommendation?
Explainability (Layer 3)	Constraint Reasoning	Which constraints shaped the response?

Table 3: Evaluation coverage across quality dimensions

This layered approach enables targeted improvements. If constraint satisfaction scores are high but explainability scores are low, the agent’s decision logic is sound, but its communication needs work. Conversely, if explainability is high but constraints are violated, the agent articulates reasoning well but applies incorrect logic. Each failure mode has a different remediation path, and the evaluation framework makes this distinction measurable.

Clean up

To avoid recurring charges, clean up your AWS account in a single step after trying the solution.

Conclusion

In this post, we demonstrated how to build and evaluate a multi-agent supply chain decisioning system using Amazon Bedrock AgentCore Evaluations, with a focus on verifying that agent behavior is not only functional, but also helpful, accurate, and explainable. Using the AnyCompany Retail Group scenario, we showed how an orchestrator agent and specialized sub-agents collaborate to solve complex problems such as inventory allocation, distribution planning, routing optimization, and supply chain diagnostics, while integrating with enterprise data sources and APIs. As illustrated throughout the architecture, correctness in agentic systems goes beyond response quality. It depends on selecting the right tools, executing the correct workflow, adhering to business constraints, and grounding outputs in data.

By combining built-in evaluators with custom evaluators, teams can systematically validate both general response quality and domain-specific decision accuracy. Incorporating explainability-focused evaluators further make sure that agents clearly articulate their reasoning, reference supporting data, and explain tradeoffs in a way that business users can trust and act upon. This evaluation-driven approach enables continuous improvement using real execution data, establishes quality gates before production rollout, and provides a scalable framework for delivering consistent, transparent, and business-aligned multi-agent systems.

To get started, explore Amazon Bedrock AgentCore Evaluations and apply these patterns to your own multi-agent applications. Find the complete source code in the Amazon Bedrock AgentCore samples repository on GitHub. Begin by enabling observability, defining key evaluation dimensions for your use case, and incrementally introducing built-in and custom evaluators to measure what matters most.

To learn more, visit the Amazon Bedrock AgentCore service page or get started directly in the Amazon Bedrock console.

Related posts:

---

About the author