使用 Amazon Bedrock AgentCore Browser Tool 自动化遗留 Web 应用

医疗、制造、零售和金融服务行业的企业，普遍难以自动化那些需要类人交互、且传统机器人流程自动化（RPA）难以大规模处理的遗留 Web 应用。Amazon Bedrock AgentCore Browser Tool 与 Strands Agents 相结合，通过完全托管的浏览器服务填补了这一空白，使 AI Agent 能够借助安全、隔离的会话驱动这些遗留界面。大多数企业仍在遗留技术上运行关键工作负载，而这些系统大多只通过服务端中间件渲染 HTML，而非暴露现代 API；因此，大规模人工数据录入依然是重要的成本中心，也是数字化转型的障碍。

设想一个各行业企业都很熟悉的场景：一家大型保险公司每年要在遗留保单管理系统中处理数万项运营变更，包括计划修改、理赔调整、保障更新和批单。专职员工需要操作由遗留服务端中间件生成的复杂 Web 界面，企业仅因人工错误每年就会遭受显著损失。每项变更都需要穿越多步骤工作流、在多个页面录入数据，并依据业务规则进行验证。这些任务会消耗数千小时。在任何遗留 Web 应用仍是关键业务运营支柱的行业中，这一模式都会反复出现。

Amazon Bedrock AgentCore Browser Tool 通过提供完全托管的云端浏览器服务解决了这些限制，AI Agent 可通过安全、隔离的浏览器会话与遗留 Web 界面交互。Browser Tool 通过基于 WebSocket 的 Chrome DevTools Protocol（CDP）连接集成 Playwright，因此无论底层技术栈为何，AI Agent 都能与遗留 Web 应用交互。Browser Tool 在云端运行托管的 Chromium 实例。无论目标遗留应用最初是为哪种浏览器设计的，只要能够通过 HTTP 或 HTTPS 访问即可。

结合用于模型驱动编排的 Strands Agents，组织可以实现复杂的自动化工作流，从单步骤自动化扩展至复杂的多 Agent 工作流。该方案通过 Amazon Bedrock AgentCore runtime 与 Amazon Bedrock Foundation Model（FM）集成，提供采用 AWS Identity and Access Management（IAM）控制的会话隔离安全能力以及完整审计追踪。这可帮助企业在保留人工监督、满足监管合规要求的同时，现代化其关键工作流。

本文将介绍一个使用 Amazon Bedrock AgentCore Browser Tool 和 Strands Agents 构建的 AI 驱动数字员工参考实现。我们将介绍其架构、关键设计决策，以及包含 Terraform 部署的企业级蓝图。完整源代码可在 GitHub 获取。

理解业务与技术挑战

企业 RPA 实施通常面临三项重大技术挑战，而传统自动化方法难以解决：遗留 Web 应用集成复杂性、受监管行业的合规要求，以及阻碍成功投入生产的可扩展性限制。

遗留 Web 应用复杂性

回到保险公司场景，该保单管理系统与各行业的许多遗留应用一样，数十年前构建于服务端中间件之上，由其为浏览器生成 HTML、CSS 和 JavaScript。这些系统缺少现代 REST API，要求自动化方案通过原本为人类用户设计的 Web 界面进行交互。应用包含复杂的多步骤工作流、动态表单验证和依赖会话的状态管理，传统 RPA Bot 无法可靠地处理。历经数十年修改后留下的文档缺口进一步增加了复杂性：关键业务逻辑往往仅存在于机构知识中，而非可访问的系统规范内。

身份验证机制进一步加剧了问题。一些遗留应用要求多因素认证（MFA），一些使用专有单点登录（SSO），许多则依赖于过期规则难以预测的会话令牌。基于规则的 RPA Bot 难以应对这些情况，因此生产部署最终往往会留下大量需要人工干预的长尾场景。在我们的保险案例中，操作人员通过复杂的 Web 界面进行计划修改。他们更新保障金额、调整受益人并处理批单；每一项操作都需要经过多个页面和验证步骤。当 UI 的渲染方式与预期不同时，脆弱的、基于规则的 Bot 无法可靠处理。

合规与审计要求

受监管行业必须实施全面的审计追踪，记录用户身份、时间戳、数据修改和系统交互，以满足 GDPR、HIPAA 及金融监管要求。技术实现需要不可篡改的日志系统、加密数据传输和基于角色的访问控制，而传统 RPA 系统若不进行大量定制开发，无法提供这些能力。金融法规要求以防篡改方式保存记录六年，并确保最近 90 天的记录可以立即访问。这些要求需要复杂的日志架构，而大多数 RPA 系统并不具备这种架构。

在该场景中，每一项保单修改都必须可追溯。这意味着需要记录由谁发起变更、变更了什么、何时发生，以及是否已获批准。AI 驱动的数字员工必须提供与人工操作人员同等甚至更高水平的可审计性。

可扩展性与脆弱性

传统 RPA 解决方案面临根本性的架构约束，阻碍其在企业规模上取得成功。基于 UI 的自动化会对屏幕布局、元素位置和应用时序产生脆弱依赖，轻微的系统更新就可能导致其失效。传统 RPA 基于规则的特性，无法处理复杂业务流程所需的异常、动态内容或认知决策。性能限制进一步加剧了这些问题。传统 RPA Bot 需要专用虚拟机或物理系统，带来资源开销和扩展限制。处理多个并发流程时，会话管理变得复杂；错误恢复机制也局限于预定义场景，无法进行自适应问题解决。

对于这家保险公司而言，组织每年处理数万项计划变更。事实证明，要让传统 RPA 在保持可靠性的同时处理投保期、监管变化和年终处理等高峰工作量并不现实。该组织需要一种能够弹性扩展，并在页面以意外状态渲染时进行智能恢复的方法。

解决方案架构

应对这些挑战需要一种自动化方案：能够处理 JavaScript 密集型 Web 界面、隔离不同流程之间的会话、通过 API 暴露程序化控制、不可篡改地记录每次交互、在置信度较低时交由人工处理，并能够弹性扩展。参考实现通过一组职责范围严格限定的组件满足这些要求。这些组件包括：面向操作人员的 React 单页应用、用于解决特定浏览器限制的 TLS 终止代理、在 Amazon Bedrock AgentCore runtime 上运行 Strands Agents 的 Python Worker，以及 Amazon Bedrock AgentCore Browser Tool 提供的托管浏览器环境。Amazon Cognito 为操作人员完成登录并签发贯穿整个技术栈的 JWT。该架构旨在帮助提高自动化投资回报，同时保持安全性、合规性和人工监督。

图 1：解决方案架构

图中的编号步骤展示了端到端流程：

1. 加载 UI。 用户浏览器从 Amazon CloudFront 加载 React 聊天界面，该界面由托管静态 UI 资产的私有 Amazon Simple Storage Service（Amazon S3） 存储桶提供支持。
2. OIDC 登录。 用户通过 Amazon Cognito 完成身份验证；Amazon Cognito 配置了 SPA 和机器对机器应用客户端，并采用标准 OpenID Connect（OIDC）登录流程。
3. WSS。 浏览器通过 AWS Application Load Balancer（ALB）建立 WebSocket 连接，ALB 使用 AWS Certificate Manager（ACM）证书终止 TLS。
4. 明文 HTTP。 ALB 将 WebSocket 连接转发至私有子网中以 AWS Fargate 任务运行的 NGINX 反向代理。该代理从查询字符串中提取 JWT，并为下一跳做好准备。
5. WSS Auth Bearer。 NGINX 代理将 JWT 放入 Authorization 标头，再将连接转发至 Amazon Bedrock AgentCore runtime。AgentCore runtime 通过其 JWT 授权器验证令牌，并将请求路由至运行 Strands Agents、VisualBrowserTool 和人工介入处理程序的 browser-agent 容器。
6. InvokeModel。 Agent 调用 Amazon Bedrock 上具备视觉能力的 Foundation Model，对截图进行分析并确定下一项浏览器操作。
7. CDP / WSS（Playwright）。 Agent 通过 Chrome DevTools Protocol 驱动 Amazon Bedrock AgentCore Browser Tool 中隔离的 Chrome 实例。浏览器会访问客户网络上的目标遗留 Web 应用。
8. PutObject。 Agent 将会话记录和截图存储到 Amazon S3 存储桶，为每张截图生成预签名 URL，并通过现有 WebSocket 连接将该 URL 流式返回至用户浏览器。
9. 预签名 URL Get。 用户浏览器使用预签名 URL 直接从 S3 获取截图，并将其与 Agent 的推理轨迹一起呈现在聊天 UI 中。图像字节不会经过 WebSocket；仅会传输有效期较短的 URL。

支持性基础设施包括 Amazon Elastic Container Registry（Amazon ECR）（托管 NGINX 网关和 browser-agent 容器镜像）以及 Amazon CloudWatch（审计日志和可观测性）。

Amazon Bedrock AgentCore Browser Tool 提供完全托管的云端浏览器服务。AI Agent 可通过安全、隔离的浏览器会话与遗留 Web 界面交互。每个会话都在独立、隔离的环境中运行，拥有各自的 CPU、内存和文件系统。会话结束后，该环境会被终止，状态也会被清理。

Agent 通过基于 WebSocket 的 CDP 上的 Playwright 进行连接，从而以编程方式控制 Web 应用，包括 JavaScript 密集型界面、动态表单和多步骤工作流，而无需考虑底层技术栈。

以下三项能力直接应对了前文所述的挑战：

• 浏览器配置文件可跨会话持久保存身份验证状态。仅需认证一次，或由人工操作人员通过实时视图完成初始登录，后续会话即可恢复已登录状态，从而缓解重复的 SSO 和 MFA 流程。
• 代理配置可通过基于域名的路由，以及 AWS Secrets Manager 中保存的凭证，将流量经由企业代理基础设施转发，使企业网络内部的遗留应用或受 IP 白名单限制的门户能够访问。
• 会话录制会捕获浏览器交互，包括点击、表单输入和页面导航，并将其存储在 Amazon S3 中。结合 AWS CloudTrail 日志记录，该能力可支持金融法规要求的审计追踪。

回到保险案例，Browser Tool 会像人工操作人员一样导航保单管理系统。它使用由配置文件持久保存的会话状态登录，在多个页面中录入修改，并提交变更；所有操作都在一个安全、隔离的会话内完成，并生成完整、可回放的审计记录。

Amazon Bedrock Foundation Model 与 Strands Agents 编排

Amazon Bedrock 提供驱动 Agent 决策循环的多模态 Foundation Model。参考实现通过跨区域推理，使用 Amazon Bedrock 上具备视觉能力的 Foundation Model 分析浏览器当前状态的截图，并决定下一步应采取的操作。这是该方案区别于传统 RPA 的核心能力。模型能够看到页面、推理屏幕上显示的内容，并确定下一项浏览器操作，无论是点击、填充、滚动，还是暂停以等待人工确认。

Strands Agents 是将模型决策转化为具体工具调用的编排框架。模型并非遵循固定脚本，而是通过 ReAct（Reason + Act，推理 + 行动）循环驱动自己的行为。模型获取截图，以视觉方式分析页面，决定调用哪个工具，执行调用，观察结果，然后重复上述过程。该框架支持多种编排模式，包括用于分层委派的 Agents-as-Tools、用于结构化工作流的 Graphs，以及用于复杂多步骤流程的协作模式。

当操作人员提供诸如“打开保单 #12345，将保障金额更新为 $500,000，并提交审批”这样的自然语言指令时，模型会将其分解为一系列顺序执行的浏览器操作。每次操作后，模型都会截取屏幕截图，以视觉方式解释新的页面状态，a

Enterprises across healthcare, manufacturing, retail, and financial services struggle to automate legacy web applications that demand human-like interaction beyond what standard Robotic Process Automation (RPA) can provide at scale. Amazon Bedrock AgentCore Browser Tool, combined with Strands Agents, addresses this gap with a fully managed browser service that lets AI agents drive these legacy interfaces through secure, isolated sessions. With the majority of enterprises still running critical workloads on legacy technology, and with most of those systems exposing only HTML rendered by server-side middleware rather than modern APIs, high-volume manual data entry remains a significant cost center and a barrier to digital transformation.

Consider a scenario familiar to enterprises across industries: a large insurance company processes tens of thousands of operational changes annually across legacy policy administration systems, including plan modifications, claims adjustments, coverage updates, and endorsements. With dedicated staff navigating complex web interfaces generated by legacy server-side middleware, the organization faces significant annual losses from manual errors alone. Each change requires navigating multi-step workflows, entering data across multiple screens, and validating against business rules. These tasks consume thousands of hours. This pattern repeats across industries wherever legacy web applications remain the backbone of critical business operations.

The Amazon Bedrock AgentCore Browser Tool addresses these limitations by providing a fully managed, cloud-based browser service that AI agents use to interact with legacy web interfaces through secure, isolated browser sessions. The Browser Tool uses Playwright integration through WebSocket-based Chrome DevTools Protocol (CDP) connections, so AI agents can interact with legacy web applications regardless of their underlying technology stack. The Browser Tool runs a managed Chromium instance in the cloud. The target legacy application needs to be accessible over HTTP or HTTPS, regardless of which browser it was originally designed for.

Combined with Strands Agents for model-driven orchestration, organizations can implement sophisticated automation workflows that scale from single-step automations to complex multi-agent workflows. The solution integrates with Amazon Bedrock foundation models (FMs) through Amazon Bedrock AgentCore runtime, providing session-isolated security with AWS Identity and Access Management (IAM) controls and full audit trails. This can help companies modernize critical workflows while supporting their regulatory compliance requirements and preserving human oversight.

In this post, we walk through a reference implementation of an AI-powered digital worker built with Amazon Bedrock AgentCore Browser Tool and Strands Agents. We cover the architecture, the key design decisions, and an enterprise blueprint with terraform deployment. The complete source code is available on GitHub.

Understanding the business and technical challenge

Enterprise RPA implementations typically face three significant technical challenges that traditional automation approaches struggle to address. These are legacy web application integration complexity, regulatory compliance requirements in regulated industries, and scalability limitations that prevent successful production deployment.

Legacy web application complexity

Returning to our insurance company scenario, the policy administration system, like many legacy applications across industries, was built decades ago on server-side middleware that generates HTML, CSS, and JavaScript for the browser. These systems lack modern REST APIs, requiring automation solutions to interact through web interfaces originally designed for human users. The applications feature complex multi-step workflows, dynamic form validation, and session-dependent state management that traditional RPA bots can’t reliably navigate. Documentation gaps after decades of modifications create additional complexity, with critical business logic existing only in institutional knowledge rather than accessible system specifications.

Authentication mechanisms compound the problem. Some legacy applications require multi-factor authentication (MFA), some use proprietary single sign-on (SSO), and many rely on session tokens with unpredictable expiration rules. Rules-based RPA bots struggle here, which is why production deployments end up with a long tail of manual interventions. In our insurance scenario, operators navigate a complex web interface to make plan modifications. They update coverage amounts, adjust beneficiaries, and process endorsements, each requiring multiple screens and validation steps that a brittle, rules-based bot cannot reliably handle when the UI renders differently than expected.

Compliance and audit requirements

Regulated industries must implement comprehensive audit trails that capture user identity, timestamps, data modifications, and system interactions for GDPR, HIPAA, and financial regulations. Technical implementation requires immutable logging systems, encrypted data transmission, and role-based access controls that traditional RPA systems cannot provide without extensive custom development. Financial regulations mandate tamper-proof record retention for six years with immediate accessibility for the most recent 90 days. These requirements necessitate sophisticated logging architectures that most RPA systems lack.

Every policy modification in this scenario must be traceable. That means capturing who initiated the change, what was changed, when, and whether it was approved. An AI-powered digital worker must provide the same level of auditability as a human operator, if not more.

Scalability and brittleness

Traditional RPA solutions face fundamental architectural constraints that prevent enterprise-scale success. UI-based automation creates brittle dependencies on screen layouts, element positioning, and application timing that break with minor system updates. The rules-based nature of traditional RPA can’t handle exceptions, dynamic content, or cognitive decision-making required for complex business processes. Performance limitations compound these issues. Traditional RPA bots require dedicated virtual machines or physical systems, creating resource overhead and scaling constraints. Session management becomes complex when handling multiple concurrent processes, and error recovery mechanisms are limited to predefined scenarios rather than adaptive problem-solving.

For our insurance company, the organization processes tens of thousands of plan changes annually. Scaling traditional RPA to handle peak volumes such as enrollment periods, regulatory changes, and year-end processing while maintaining reliability proved impractical. The organization needed an approach that could scale elastically and recover intelligently when a page renders in an unexpected state.

Solution architecture

Addressing these challenges requires automation that can handle JavaScript-heavy web interfaces, isolate sessions between processes, expose programmatic control through APIs, log every interaction immutably, defer to a human when confidence is low, and scale elastically. The reference implementation meets these requirements through a small set of components with tightly scoped responsibilities. These include a React single-page app for the operator, a TLS-terminating proxy that solves a specific browser limitation, a Python worker running Strands Agents on Amazon Bedrock AgentCore runtime, and the managed browser environment provided by Amazon Bedrock AgentCore Browser Tool. Amazon Cognito signs the operator in and issues the JWT that flows through the stack. The architecture is designed to help increase automation return on investment while maintaining security, compliance, and human oversight.

Figure 1: Solution architecture

The numbered steps in the diagram trace the end-to-end flow:

1. Load UI. The user’s browser loads the React chat interface from Amazon CloudFront, backed by a private Amazon Simple Storage Service (Amazon S3) bucket hosting the static UI assets.
2. OIDC Login. The user authenticates through Amazon Cognito (configured with SPA and machine-to-machine app clients) through the standard OpenID Connect (OIDC) sign-in flow.
3. WSS. The browser opens a WebSocket connection through an AWS Application Load Balancer (ALB) that terminates TLS using an AWS Certificate Manager (ACM) certificate.
4. Plain HTTP. The ALB forwards the WebSocket connection to an NGINX reverse proxy running as an AWS Fargate task in a private subnet. The proxy extracts the JWT from the query string and prepares it for the next hop.
5. WSS Auth Bearer. The NGINX proxy forwards the connection to Amazon Bedrock AgentCore runtime with the JWT in the Authorization header. AgentCore runtime validates the token through its JWT authorizer and routes to the browser-agent container running Strands Agents, VisualBrowserTool, and the human-in-the-loop handler.
6. InvokeModel. The agent calls a vision-capable foundation model on Amazon Bedrock to analyze screenshots and determine the next browser action.
7. CDP / WSS (Playwright). The agent drives an isolated Chrome instance in Amazon Bedrock AgentCore Browser Tool over the Chrome DevTools Protocol. The browser navigates the target legacy web applications on the customer’s network.
8. PutObject. The agent stores session transcripts and screenshots to an Amazon S3 bucket, generates a pre-signed URL for each screenshot, and streams the URL back to the user’s browser over the existing WebSocket connection.
9. Pre-Signed URL Get. The user’s browser fetches the screenshot image directly from S3 using the pre-signed URL, rendering it in the chat UI alongside the agent’s reasoning trace. Image bytes don’t travel through the WebSocket. Only the short-lived URL does.

Supporting infrastructure includes Amazon Elastic Container Registry (Amazon ECR) (hosting the NGINX gateway and browser-agent container images) and Amazon CloudWatch (audit logging and observability).

Amazon Bedrock AgentCore Browser Tool provides a fully managed, cloud-based browser service. AI agents interact with legacy web interfaces through secure, isolated browser sessions. Each session runs in a dedicated, isolated environment with its own CPU, memory, and filesystem. When a session completes, the environment is terminated and the state is sanitized.

Agents connect through Playwright over WebSocket-based CDP for programmatic control of web applications, including JavaScript-heavy interfaces, dynamic forms, and multi-step workflows, regardless of the underlying technology stack.

Three capabilities directly address the challenges described earlier:

• Browser profiles persist authentication state across sessions. Authenticate once, or have a human operator perform the initial login through live-view, and subsequent sessions resume already signed in, alleviating repeated SSO and MFA flows.
• Proxy configuration routes traffic through corporate proxy infrastructure with domain-based routing and credentials on AWS Secrets Manager, making internal legacy applications behind corporate networks or IP-allowlisted portals reachable.
• Session recording captures browser interactions, including clicks, form inputs, and page navigation, and stores them in Amazon S3. Combined with AWS CloudTrail logging, this supports the audit trail requirements of financial regulations.

Returning to our insurance example, the Browser Tool navigates the policy administration system exactly as a human operator would. It signs in with profile-persisted session state, enters modifications across multiple screens, and submits changes, all within a secure, isolated session that produces a complete, replayable audit record.

Amazon Bedrock foundation models and Strands Agents orchestration

Amazon Bedrock provides the multi-modal foundation model that drives the agent’s decision loop. The reference implementation uses a vision-capable foundation model on Amazon Bedrock through cross-region inference to analyze screenshots of the browser’s current state and decide what action to take next. This is the core capability that differentiates the solution from traditional RPA. The model sees the page, reasons about what’s on screen, and determines the next browser action, whether that is a click, a fill, a scroll, or a pause for human confirmation.

Strands Agents is the orchestration framework that turns the model’s decisions into concrete tool calls. Rather than following a fixed script, the model drives its own behavior through a ReAct (Reason + Act) loop. The model captures a screenshot, analyzes the page visually, decides which tool to call, executes it, observes the result, and repeats. The framework supports multiple orchestration patterns, including Agents-as-Tools for hierarchical delegation, Graphs for structured workflows, and collaborative patterns for complex multi-step processes.

When an operator provides natural language instructions such as “open policy #12345, update the coverage amount to $500,000, and submit for approval,” the model decomposes these into sequential browser actions. After each action, the model takes a screenshot, interprets the new page state visually, and decides the next step. If a confirmation dialog appears unexpectedly or a field validates differently than anticipated, the model reasons through the new state rather than failing. This alleviates the brittleness that makes traditional RPA impractical at scale.

Applied to our insurance scenario, a single natural language instruction can drive a complete plan modification workflow. The model visually navigates to the right policy, identifies form fields from the screenshot, enters changes across multiple screens, handles unexpected validation prompts, and submits for approval. No human needs to script each step or maintain brittle selectors as the UI evolves.

Human-in-the-loop integration

The operator sees the automation as it happens. The agent streams reasoning traces and screenshots to the chat interface in real time. When the model determines it needs human input before, for example confirming that the correct policy was opened before submitting a change, it calls the handoff_to_user tool, which pauses the automation and presents the operator with a question and the current screenshot.

The browser session stays alive while the agent waits for a response (up to 300 seconds by default). The operator can approve, reject, provide additional instructions, or ask the agent to try a different approach. Once the response arrives, the model resumes the ReAct loop with the operator’s input. It takes a fresh screenshot to assess the current state and continues from there. If the operator doesn’t respond within the timeout window, the model decides whether to retry, attempt an alternative approach, or abort gracefully.

In the context of our insurance workflow, this means the digital worker pauses before critical actions such as submitting a plan modification, confirming a beneficiary change, or processing an endorsement that exceeds a threshold. This gives the operator the opportunity to verify accuracy on screen before the change becomes permanent. The operator remains in control without needing to drive every step manually.

Security and compliance

The architecture provides session isolation, IAM controls for access management, and ephemeral sessions where the state is cleared after each use. The reference implementation uses Amazon Cognito as the identity provider, with Microsoft Entra ID and Okta supported by swapping the OIDC discovery URL. Standards-aligned OIDC providers work with the same configuration surface.

For our insurance company, this means every policy modification performed by the digital worker is fully traceable, with session recordings stored in Amazon S3 and actions logged through Amazon CloudWatch. This supports the retention requirements of financial regulations.

Implementation walkthrough

The reference implementation ships as a Python worker built on Strands Agents, deployed on Amazon Bedrock AgentCore runtime. The agent uses the AgentCore Browser Tool SDK to create isolated browser sessions through the Strands AgentCoreBrowser base class. The complete source code is available in the GitHub repository.

How the agent drives the browser

The following diagram traces a single operator instruction through the full automation loop. The operator issues a natural language command, the agent starts a browser session, and the foundation model enters a ReAct cycle. It captures screenshots, analyzes page state visually, decides the next action, executes it through Playwright, and repeats until the task is complete.

Figure 2: Chat flow, from operator instruction through the ReAct loop

When the model determines it needs human confirmation before a critical step, such as submitting a form or confirming a record selection, it pauses the automation and presents the operator with the current screenshot and a question. The operator responds, and the model resumes.

Figure 3: Human-in-the-loop, where the agent pauses for an operator decision

Key design decisions

Two implementation choices are worth calling out because they directly affect production reliability:

Event loop isolation. The browser agent runs alongside a WebSocket server that streams updates to the operator. The upstream browser tool’s threading model interferes with the server’s event loop, causing connections to drop after about 15 seconds. The reference implementation overrides this so browser operations run on their own isolated loop, keeping the operator’s connection stable throughout the session. See visual_browser_tool.py for the full implementation.

Semantic locators over CSS selectors. Traditional RPA bots break when UI elements move or get renamed. The semantic_action tool takes a different approach. The model describes what it sees (“click the button labeled Submit”) and the tool maps that to Playwright’s semantic Locator API using case-insensitive fuzzy matching. This makes the automation resilient to the minor UI variations common in legacy applications. This is exactly the brittleness problem the solution set out to solve. See semantic_action_tool.py for the full implementation.

Deployment and testing

The entire solution deploys with a single Terraform stack. The following steps take you from git clone to a working chat URL.

Prerequisites

You must have the following:

• An AWS account with Amazon Bedrock model access enabled for a vision-capable foundation model.
• AWS Command Line Interface (AWS CLI) v2 configured with credentials.
• Docker 24+.
• Terraform 1.5+.
• Node 20+ and Yarn.
• Python 3.12.

Deploy

git clone <repo-url> && cd <repo>
cp deployment/terraform/stacks/all/terraform.tfvars.sample \
deployment/terraform/stacks/all/terraform.tfvars
# Edit terraform.tfvars - set aws_region, project_name, cognito_domain_prefix

terraform -chdir=deployment/terraform/stacks/all init
terraform -chdir=deployment/terraform/stacks/all apply

The stack provisions the Cognito User Pool, CloudFront distribution, ECR repositories, ECS proxy service, AgentCore runtime with JWT authorizer, and S3 session storage. Terraform provisioners build and push both container images, compile the React UI, sync it to S3, and invalidate CloudFront, all in one pass. First apply takes approximately 20–30 minutes, most of which is CloudFront propagation and container image builds.

Test the solution

After deployment completes, Terraform outputs the CloudFront URL. Open it in a browser, sign in with a Cognito user, and try a prompt such as:

“Open https://httpbin.org/forms/post, fill the customer name with John Smith, pick medium pizza, check bacon topping, and submit the form.”

You should see the reasoning trace stream into the chat, screenshots appear alongside each step, and a human-in-the-loop prompt before the final submit. For additional test scenarios, example prompts, and screenshots of the UI in action, see the repository README.

Clean up

To avoid ongoing charges, destroy the infrastructure when you’re done:

terraform -chdir=deployment/terraform/stacks/all destroy

This removes the provisioned resources including the Cognito User Pool, CloudFront distribution, ECR repositories, ECS services, AgentCore runtime, and S3 buckets.

Conclusion and next steps

The AI-powered digital worker described in this post makes that case achievable. Where traditional RPA is brittle because it couples to exact UI layouts, the combination of the vision-capable foundation models in Amazon Bedrock, the model-driven orchestration of Strands Agents, and the isolated browser sessions of AgentCore Browser Tool produces automation that reasons over page state, recovers from unexpected branches, and defers to a human when confidence is low. It does this without sacrificing the audit trail that regulated industries require.

The reference implementation leaves room to grow. If your workflow needs cross-session context, Amazon Bedrock AgentCore memory provides managed short-term and long-term recall. If you need to integrate existing REST APIs as tools the agent can call alongside its browser actions, Amazon Bedrock AgentCore Gateway handles that without writing custom Strands tools. For portals that require custom browser behavior, Chrome extensions can be loaded into AgentCore Browser Tool sessions at creation time. The complete source code is available on GitHub.

---

About the authors

Salman Moghal

Salman is a Principal Consultant at AWS Professional Services Canada, specializing in agentic AI and cloud-native architecture. He architects and builds the AI systems that modernize operations across banking, insurance, and financial services.

Brian Tol

Brian leads the North America Application Modernization practice at AWS Professional Services, where his teams help enterprise customers accelerate their cloud transformations using the most advanced AI technologies.

Nasar Khan

Nasar is a Delivery Consultant with AWS Professional Services, helping customers design and implement Agentic AI solutions that turn ambitious ideas into working production systems. He specializes in building autonomous, multi-agent architectures on AWS, connecting large language models with tools, data, and enterprise workflows to solve real business problems at scale.