HPE Zerto 如何借助 Amazon Bedrock 构建 Agentic 故障排查系统

本文由 AWS 与 HPE Zerto 团队联合撰写。

如果你管理混合云和多云基础设施，可能已经在借助 AI 系统评估运行状况、调查问题，并更快地对问题采取行动。HPE Zerto 通过构建一个由 Amazon Bedrock 驱动的 Agentic 故障排查系统，应对了这一挑战。

HPE Zerto Software 可帮助你增强网络韧性、灾难恢复和持续数据保护能力。你可以利用这些能力大幅减少数据丢失和停机时间，同时支持从勒索软件事件、故障和其他中断中快速恢复。

本文介绍了我们在开发由 Amazon Bedrock 驱动的 Agentic 故障排查系统时所选择的系统架构，以及面临的挑战。我们将其部署在本地环境中，并对运行信号、环境上下文和产品知识实施受控且安全的访问。

该系统提供自然语言交互体验，帮助你快速从问题转向可执行的洞察，从而支持更快速、更明智的恢复决策。

业务挑战

数据保护和灾难恢复团队所在环境的规模、复杂性和业务关键性都在不断提升。他们必须持续监控保护状态、验证服务等级协议（SLA）合规性、解读告警、调查故障，并为恢复事件做准备。这些工作通常跨越多个站点和大量受保护工作负载。

• 回答运维问题所需的信息通常分散在产品告警、事件和文档中，团队必须先手动收集上下文才能采取行动。
• 运维人员需要在仪表板、报告和知识来源之间频繁切换，才能了解正在发生什么以及下一步该做什么，这会造成时间损耗。
• 经验较少的管理员可能难以解读症状，或确定最安全的修复路径。
• 管理者可能无法快速获得有关风险、SLA 暴露情况或恢复就绪状态的清晰摘要。
• 在宕机或网络事件期间，这些延迟会变得更加严重，因为不确定性的代价很高，团队需要快速获得清晰、可信的指导。

因此，我们的客户需要一种方式，能够在不同环境中部署时，将复杂的韧性数据转化为可执行的答案，而不增加额外运维负担。

解决方案概述

从根本上说，我们将该系统设计为：

1. 充当支持助手，帮助你快速解决配置问题和排查故障，而无需创建支持工单。
2. 就系统运行状况问题向你发出告警，并协助缓解问题。
3. 代表你执行任务，以加快配置时间和功能采用速度。

我们将该系统作为一个 Pod 部署在你本地环境中的 Zerto 产品内，可直接从你已经用于日常运维的同一 UI 访问。你可以使用自然语言交互；该系统旨在进行推理、采取行动，并提供以你环境实时状态为基础、具有上下文且可执行的回答。

我们选择 Amazon Bedrock，是因为它能够支持安全的企业部署、模型灵活性和运维控制。

关键考量包括：

• 安全与治理 – Amazon Bedrock 支持对基础模型的受控访问，帮助我们使 AI 使用方式符合企业安全、合规和数据保护要求。
• 模型灵活性 – 该服务使我们能够访问并评估多个基础模型，并根据质量、延迟和成本选择最合适的模型。
• 运维集成 – Amazon Bedrock 可与用于护栏、可观测性和知识检索的 AWS 服务集成，从而支持更易于管理的生产架构。

我们使用 Amazon Bedrock Guardrails 实施内容安全控制，确保你的提示词和系统输出符合公司政策、合规及安全要求。

下图展示了该架构的关键组件：

• UI 层 – 你通过嵌入现有 Zerto UI 的聊天界面与系统交互。使用 Server-Sent Events（SSE）进行实时流式传输，可在调查进行过程中显示进度，从而提供更好的用户体验。
• Agentic 层 – 使用 Strands Agents framework 构建的 Agent 作为 Pod 运行在你的本地环境中，并可访问以下资源：
  • 会话历史 – 本地存储的历史聊天对话和上下文。
  • 内部工具 – 本地 Model Context Protocol（MCP）服务器，公开 Agent 可调用的 ZVM（Zerto Manager）API，以结构化方式访问实时环境数据。
  • 外部工具 – 通过 Amazon Bedrock Knowledge Bases 智能获取公开文档、相关运行手册和运维知识；该服务是一项集中式、完全托管的检索增强生成（RAG）能力，具备用于上下文检索的内部语义搜索功能。
• 智能层 – 由三个主要组件构成：
  • 推理 – 请求被发送至 Amazon Bedrock，后者将其路由至选定的基础模型。通过每个租户所承担的 AWS Identity and Access Management（IAM）角色标签，完成按租户的请求标记。
  • 安全性 – 在推理前后应用 Amazon Bedrock Guardrails，以强制执行内容边界，并帮助验证回答仍限定在运维主题范围内。
  • 限制与配额 – 结合使用 Amazon CloudWatch、Amazon DynamoDB 和 AWS Lambda，以实施按租户的配额和限制，并维持具有成本效益的系统。
• 可观测性层 – 从系统发送至 Amazon CloudWatch 的遥测数据，可帮助了解 Agent 性能、错误率和使用模式。

图 1：解决方案架构

下图展示了嵌入 Zerto 产品中的 Agentic 系统。

图 2：嵌入 HPE Zerto 管理控制台的 Agentic 系统

Agentic 流程

该系统由多个子 Agent 构成，并由一个 Orchestrator Agent 控制；后者可根据当前任务，将每个子 Agent 作为工具使用。每个子 Agent 又拥有自己的工具集和独立上下文，用于处理其子任务。图 3 展示了 Agent 之间的流程和层级关系。

Orchestrator 决定何时委派工作。常规问题由其直接回答，而需要深入分析大量日志的调查任务，则会被路由至限定在相关组件范围内的专业子 Agent。每个子 Agent 都在独立的新鲜上下文中运行，拥有专用系统提示词和自身工具，并使用更强的模型处理需要大量推理的工作。其最终回答是一份紧凑、提炼后的报告，不会让原始日志和调查过程草稿污染或膨胀父 Agent 的上下文。

• Orchestrator Agent – 用于两类目的。你可以使用它检查保护组状态、查看近期告警，或咨询配置最佳实践。当收到系统中当前存在的问题时，该 Agent 会在各子 Agent 之间编排故障排查流程。它会从回答中移除机密信息或原始日志，然后将最终答案转达给你。
• ZVM Agent – 负责排查与 ZVM 相关的问题，例如服务崩溃、网络问题和升级失败。它使用提供领域特定知识及特定日志模式的专用技能，分析日志，并通过 MCP 添加环境数据。这可对当前问题及其可能原因提供全面分析。
• VRA Agent – 负责排查与 VRA（Zerto Replication Engine）相关的问题，例如复制问题和延迟、站点与安装之间的网络问题。它同样具备领域特定知识技能，为 Agent 提供结构化日志模式上下文，以便与日志以及通过 MCP 获取的环境数据进行匹配。

图 3：Agentic 流程

挑战

构建 HPE Zerto Agentic 故障排查系统需要应对若干关键工程决策，包括模型选择、评估方法，以及本地部署的约束条件。

模型选择

作为开发流程的一部分，我们对多个用于推理和推断的模型进行了比较和组合，同时考虑质量、性能和成本之间的权衡。

在最初的概念验证（POC）阶段，我们首先评估了规模更大、能力更强的模型，以验证可行性；随后，我们实施了 Agent 评估机制（如下节所述），并比较了其他模型，其中一些模型速度更快、规模更小且成本更低。最终，我们选择了此前所述的多模型架构。随着新模型的推出，我们仍会持续对其进行评估。

有关按 AWS 区域划分的模型可用性，请参阅 Amazon Bedrock 文档中的 Supported models by AWS Region。

Agent 评估

我们使用 PyTest 和 Strands Agents Evals SDK 对 Agent 进行系统化和持续性评估，同时不断升级系统。我们还创建了用于运行完整评估套件的作业，以观察 Agent 行为中的回归和改进。

我们创建了三类评估：

• 基础查询评估 – 单轮查询—响应评估：向 Agent 提供一个查询作为测试，并对其响应进行评估。
• 多轮评估 – 完整对话评估：向 Agent 提供一个初始查询作为测试，Agent 作出响应后，由充当真实用户的用户模拟器大语言模型（LLM）进行互动。该技术使用 Strands Agents ActorSimulators。在完整对话结束时进行评估。
• 动态测试评估 – 为评估尚未覆盖且动态变化的用例，我们使用 Strands Agents Experiment Generator 创建了一套完全动态的测试套件，该工具会根据特定标准生成独特的测试套件。每项测试包含四个评估器：
• 响应评估器 – 根据特定预定义标准评估 Agent 响应，并使用可解析的结构化输出提取相关数据。
• 轨迹评估器 – 根据特定预定义标准评估 Agent 的工具选择。该评估器同样使用结构化输出。
• 延迟评估器 – 评估 Agent 延迟，并检查其是否处于定义的阈值范围内。
• Token 评估器 – 评估 Agent 的 Token 使用量，并检查其是否保持在边界范围内。

“Strands Agents 消除了构建 AI Agent 的复杂工作——无需僵化的工作流定义，无需样板化的工具链式调用代码，只需要模型、提示词以及所需的工具。借助内置评估和可观测性功能，我的团队构建、测试和迭代 Agent 的速度，比我们尝试过的任何框架都更快。它覆盖了完整的开发生命周期，因此我们可以专注于解决实际问题，而不是编写胶水代码。”

— Shahak Gabay，HPE Zerto 高级软件工程师

本地部署模型

我们 Agentic 故障排查系统的部署约束之一，是它必须在你自己的环境中运行。灾难恢复（DR）基础设施通常采用气隙隔离、对延迟敏感，或受到严格的数据驻留要求约束。我们使用 Strands Agents SDK 实现了 Agent 运行时，它完全在 ZVM 设备内的本地环境中运行。只有模型推理请求和知识库查询会通过 HTTPS 发送至 AWS。

业务成果

借助 HPE Zerto Agentic 系统，你可以在增强恢复就绪能力的同时提高运维效率。通过将响应建立在实时环境数据和可信产品文档之上，该系统可帮助你回答运维问题、调查故障、识别可能的根本原因，并确定需要采取行动的风险优先级。

自 2026 年第二季度发布以来，已有超过 20% 的 HPE Zerto 客户采用并积极使用该 Agentic 系统。这表明市场对 AI 辅助灾难恢复运维存在强劲需求。在生产使用中，我们观察到，由该 Agentic 系统支持的工作流，其支持案例数量减少了 10%。你现在可以调查并

This post was co-written by AWS and the HPE Zerto team.

If you manage hybrid and multi-cloud infrastructures, you may already be turning to AI systems to assess health, investigate issues, and act on problems faster. HPE Zerto addressed this challenge by building an agentic troubleshooting system powered by Amazon Bedrock.

HPE Zerto Software helps you strengthen cyber resilience, disaster recovery, and continuous data protection. You can use these capabilities to substantially reduce data loss and downtime, while supporting rapid recovery from ransomware events, outages, and other disruptions.

This post describes the system architecture we selected and the challenges we faced when developing an agentic troubleshooting system powered by Amazon Bedrock. We deployed it in an on-premises environment, with controlled and secure access to operational signals, environmental context, and product knowledge.

The system provides a natural language experience that helps you move quickly from questions to actionable insights to support faster, more informed recovery decisions.

Business challenge

Data protection and disaster recovery teams operate in environments that are growing in scale, complexity, and business criticality. They must continuously monitor protection health, validate service level agreement (SLA) compliance, interpret alerts, investigate failures, and prepare for recovery events. This work often spans multiple sites and large numbers of protected workloads.

• Information needed to answer operational questions is often fragmented across product alerts, events, and documentation, forcing teams to manually gather context before they can act.
• Operators lose time moving between dashboards, reports, and knowledge sources to understand what is happening and what to do next.
• Less experienced administrators may struggle to interpret symptoms or determine the safest remediation path.
• Managers may lack fast access to clear summaries of risk, SLA exposure, or recovery readiness.
• During outages or cyber events, these delays become even more significant because the cost of uncertainty is high and teams need clear, trusted guidance quickly.

Our customers therefore need a way to turn complex resilience data into actionable answers without adding more operational burden while deployed in different environments.

Solution overview

Fundamentally, we designed the system to:

1. Serve as a support assistant helping you solve configuration issues and troubleshoot problems in a short-circuit without needing to create a support ticket.
2. Alert you on system health problems and assist with mitigation.
3. Perform tasks on your behalf to accelerate setup time and feature adoption.

We deploy the system as a pod in the Zerto product inside your on-premises environment, accessible directly from the same UI you already use for day-to-day operations. You can interact in natural language, and the system is designed to reason, act, and respond with contextual, actionable answers grounded in the live state of your environment.

We selected Amazon Bedrock because of its ability to support secure enterprise deployment, model flexibility, and operational control.

The key considerations included:

• Security and governance – Amazon Bedrock supports controlled access to foundation models, helping us align AI usage with enterprise security, compliance, and data protection requirements.
• Model flexibility – The service made it possible to access and evaluate multiple foundation models and select the best fit based on quality, latency, and cost.
• Operational integration – Amazon Bedrock integrates with AWS services used for guardrails, observability, and knowledge retrieval, supporting a more manageable production architecture.

We enforce content security using Amazon Bedrock Guardrails to make sure your prompts and system output are aligned with company policy, compliance, and security requirements.

The following figure shows the key components of the architecture:

• UI layer – You interact with the system through the chat interface embedded in the existing Zerto UI. Real-time streaming using Server-Sent Events (SSE) shows investigation progress as it happens, providing a better user experience.
• Agentic layer – Agents built using the Strands Agents framework run as a pod inside your environment on-premises, with access to the following:
  • Session history – Locally stored historical chat conversations and context.
  • Internal tools – Local Model Context Protocol (MCP) server exposing ZVM (Zerto Manager) APIs the agent can call for structured access to live environment data.
  • External tools – Smart fetching of public documentation, relevant runbooks and operational knowledge using Amazon Bedrock Knowledge Bases, a centralized, fully managed Retrieval Augmented Generation (RAG) capability with internal semantic search for context retrieval.
• Intelligence layer – Comprised of three main components:
  • Inference – Requests are sent to Amazon Bedrock which routes them to the selected foundation model. Per-tenant request tagging is done using AWS Identity and Access Management (IAM) role tags each tenant assumes.
  • Security – Amazon Bedrock Guardrails are applied before and after inference to enforce content boundaries and help verify responses remain scoped to operational topics.
  • Limits and quotas – Amazon CloudWatch, Amazon DynamoDB, and AWS Lambda are combined to enforce per-tenant quotas and limits and maintain a cost-efficient system.
• Observability layer – Telemetry data sent from the system to Amazon CloudWatch provides visibility into agent performance, error rates, and usage patterns.

Figure 1: Solution architecture

The following figure shows the agentic system embedded in the Zerto product.

Figure 2: The embedded agentic system in the HPE Zerto management console

Agentic flow

The system consists of multiple sub-agents, controlled by an Orchestrator agent which can use each of those sub-agents as tools based on the task in hand. In turn, each sub-agent gets its own set of tools and separate context to handle its sub-task. Figure 3 shows the flow and hierarchy between the agents.

The Orchestrator decides when to delegate work. Routine questions are answered directly, while deep, log-heavy investigations are routed to a specialized sub-agent, scoped to the relevant component. Each sub-agent runs with its own fresh context, dedicated system prompt, its own tools, and uses a stronger model for the reasoning-heavy work. Its final response is a compact, distilled report, without raw logs and investigation scratch-work polluting or inflating the parent’s context.

• Orchestrator agent – Serves two purposes. You can use it for checking protection group status, reviewing recent alerts, or asking about configuration best practices. When given an issue that currently exists in the system, the agent orchestrates the troubleshooting flow between the sub-agents. It relays the final answer to you after stripping secrets or raw logs from the response.
• ZVM agent – Responsible for troubleshooting ZVM related issues such as service crashes, networking issues, and upgrade failures. Using dedicated skills that provide domain-specific knowledge and specific log patterns, it analyzes logs and adds environmental data through MCP. This provides a thorough analysis of the issue at hand and its possible causes.
• VRA agent – Responsible for troubleshooting VRA (Zerto Replication Engine) related issues such as replication problems and delays, networking between sites and installations. It also has domain-specific knowledge skills that provide structured log patterns context for the agent to match against the logs and the environmental data available through MCP.

Figure 3: Agentic flow

Challenges

Building the HPE Zerto agentic troubleshooting system required navigating several key engineering decisions, including model selection, evaluation methodology, and the constraints of on-premises deployment.

Model selection

As part of our development process, we compared and then combined several models for reasoning and inference, considering the tradeoff between quality, performance, and cost.

We evaluated bigger, stronger models first for the initial proof of concept (POC), to prove feasibility, and later implemented an agent evaluations mechanism (described in the following section) and compared other models, some of them faster, smaller, and cheaper. Eventually, a multi-model architecture was selected as described earlier. We continue to evaluate newer models as they become available.

For model availability by AWS Region, refer to Supported models by AWS Region in Amazon Bedrock.

Agent evaluations

We used PyTest and Strands Agents Evals SDK for evaluating agents systematically and on an ongoing basis as we keep upgrading the system. We have also created jobs for running the full evaluations suite to see regressions and improvements in the agent behavior.

Three types of evaluations were created:

• Basic query evaluation – Single-turn query-response evaluation where the agent is given a query as a test and the response is evaluated.
• Multi-turn evaluation – Full conversation evaluation where the agent is given an initial query as a test, the agent responds and a user simulator large language model (LLM) acts as an actual user. This technique uses Strands Agents ActorSimulators. The full conversation is evaluated at the end.
• Dynamic tests evaluation – To evaluate uncovered and dynamic use cases, we created a fully dynamic test suite using the Strands Agents Experiment Generator, which creates a unique test suite based on specific criteria.For each test there are four evaluators:
• Response evaluator – Evaluates the agent response based on specific predefined criteria and using structured output that can be parsed to extract relevant data.
• Trajectory evaluator – Evaluates the agent tools selection based on specific pre-defined criteria. This evaluator also uses structured output.
• Latency evaluator – Evaluates the agent latency and checks it stands within the threshold defined.
• Token evaluator – Evaluates the agent token usage and checks it remains within boundaries.

“Strands Agents eliminates the complex work of building AI agents — no rigid workflow definitions, no boilerplate tool-chaining code, just a model, a prompt, and the tools you need. With built-in evaluations and observability, my team builds, tests, and iterates on agents faster than with any framework we’ve tried. It covers the full development lifecycle so we can focus on solving real problems, not writing glue code.”

— Shahak Gabay, Senior Software Engineer, HPE Zerto

On-premises deployment model

One of the constraints around deployment of our agentic troubleshooting system is that it must run inside your own environment. Disaster recovery (DR) infrastructure is frequently air-gapped, latency-sensitive, or subject to strict data residency requirements. We implemented the agent runtime using the Strands Agents SDK, which runs entirely on-premises within the ZVM appliance. Only model inference requests and knowledge base queries go out to AWS over HTTPS.

Business outcomes

With the HPE Zerto agentic system, you can improve operational efficiency while strengthening recovery readiness. By grounding responses in live environment data and trusted product documentation, the system helps you answer operational questions, investigate issues, identify likely root causes, and prioritize risks that require action.

Since its release in Q2 of 2026, more than 20 percent of HPE Zerto customers have already adopted and are actively using the agentic system. This demonstrates strong demand for AI-assisted disaster recovery operations. Across production usage, we have observed a 10 percent reduction in support cases for workflows supported by the agentic system. You can now investigate and resolve more common operational issues directly within the product through guided self-service.

The impact extends beyond support-case reduction. By reducing repetitive data gathering and manual investigation, the system helps accelerate your troubleshooting and root-cause identification while improving administrator productivity. You can generate operational summaries faster, identify SLA and protection risks earlier, and make more informed decisions during both routine operations and incident response.

As adoption continues to grow, these capabilities can help you reduce operational friction, improve recovery readiness, and scale disaster recovery operations more effectively without requiring a proportional increase in specialized expertise.

Internally, our engineers have started using the agentic system’s troubleshooting capabilities to investigate and resolve common operational issues. Moreover, our QA automation team has integrated the agentic system into their workflows, now performing faster and more accurate investigation and resolution of issues found during testing.

Lessons learned

Building and deploying this solution surfaced several insights that shaped our approach to agentic architectures. These insights center on on-premises constraints, agent decomposition, and real-time user feedback.

• On-premises deployment changes the architecture fundamentally. The constraint that no customer data could leave your on-premises network shaped every architectural decision, from our choice of the Strands Agents SDK for local orchestration to the MCP server design for local data access.
• MCP servers are an effective pattern for grounding agents in live operational data. Wrapping an existing product API in an MCP server gives agents structured, typed access to live environment data and significantly accelerated our agent development.
• Multi-agent decomposition improves reliability for complex tasks. Breaking the workflow into multiple specialized agents, each with a bounded responsibility, made each component independently testable and the overall system easier to debug.We wired the sub-agents in a hub-and-spoke topology. Every delegation flows through the parent, and a sub-agent that discovers the issue belongs elsewhere reports back to the parent rather than calling a peer directly. This design keeps a single point of control for safety rules and the final answer, and it keeps the flow linear and straightforward to trace in telemetry. It also helps prevent runaway agent-to-agent recursion that could exhaust the token budget.The result is better answers on complicated problems without sacrificing the speed and predictability of everyday chat.
• Streaming matters for user trust. Showing investigation progress in real time through SSE significantly improved the user experience. Users see each step as it happens rather than waiting without visibility.

Conclusion

HPE Zerto Software brings together trusted resilience expertise, operational data, and AI-driven assistance, with Amazon Bedrock generative AI service, to help you protect, recover, and operate with greater confidence. By embedding intelligent guidance into everyday protection and recovery workflows, we help you turn complex resilience signals into clear, actionable insights, supporting faster decisions and stronger SLA readiness.

With HPE Zerto, you can move beyond point-in-time recovery readiness and build a more intelligent, resilient approach to continuous business protection.

To read more and go deeper, explore these related resources:

---

About the authors

Snir Cohen

Solutions Architect, working with Global Enterprise organizations, accelerating adoption and modernization of existing workloads in the cloud. Highly focused on generative AI projects, leading enablement, architecture design and use of best practices to support customers get from ideation to production.

Oz Altagar

Prototype Architect at AWS, working hands-on with customers to build generative AI workloads. Oz works directly alongside customer teams, driving innovation across generative AI and specifically Agentic AI through hands-on development, architecture design, and best practices.

Idan Aharoni

Software Engineer and Squad Lead, working on advanced AWS and AI solutions for HPE Zerto. Highly focused on Agentic AI initiatives, leading architecture, development, and adoption of intelligent systems while driving best practices and innovation across cloud and AI technologies.

Shahak Gabay

Senior software engineer and domain lead of AI Agents and AWS domains. Leading the technical side of Zerto’s product AI initiatives and responsible for the AWS DR product.

Marianna Zemelman

Leads product strategy and development across Cloud and AI at HPE Zerto, working at the intersection of technology, customer needs, and business priorities. Partners closely with engineering teams to shape capabilities that modernize disaster recovery, improve recovery readiness, and strengthen operational resilience.