Octave 如何在 Amazon Bedrock 上构建安全的多租户 AI 基础设施底座

在受监管行业的质量管理中，解决问题的“组织记忆”几乎总是深埋在客户的历史数据之中。去年调查过的类似缺陷、解决根本原因的纠正和预防措施（CAPA）、标出相同缺陷的审计发现。这些知识分散在数十亿份文档中，没有任何人工能够在不同语言和格式之间依据语义相似度进行全面检索，因此每次调查都只能从头开始。Octave Reliance 帮助每个组织收集跨多个系统的文档，以便他们更轻松地搜索并追踪所拥有的一切。Octave Reliance 团队将释放这些历史记录视为帮助客户提高工作效率并保持合规的最大机遇。但阻碍同样显而易见：投诉记录、审计发现和纠正措施遵循平台上最严格的数据隔离要求，而这些恰恰是 AI 需要读取的记录。在一个为医疗器械、制药和食品制造等领域超过 14,000 家客户提供服务的平台中，一次从一个客户向另一个客户的意外泄露所带来的监管后果，将远远超出任何 AI 带来的收益。

Octave 的业务遍及 180 个国家/地区，服务超过 60% 的全球财富 500 强企业。其云原生质量管理系统 Reliance 为 650 多家企业客户提供 40 多个应用程序支持，涵盖从投诉管理、CAPA 跟踪到审计管理与供应商质量等各个环节。每个人都在生成可供 AI 进行推理的合规监管数据。每个客户都是一个独立的租户，任何数据都绝不能跨越租户边界。

面对这一现实，Reliance 团队做出了一个不同寻常的决定：他们没有匆忙推出单一的 AI 功能，而是将第一个周期用于构建所有未来功能都将依赖的通用底座。自那之后，该底座将每个用例的部署时间缩短了 50%。如果您正将 AI 引入一个数据价值极高但数据隔离绝不可妥协的软件即服务（SaaS）平台，那么其背后的思考逻辑将远超质量管理本身的范畴。在本文中，我们将深入探讨塑造这一设计的四个矛盾与张力，以及 Octave 基于 Amazon Bedrock 上的 Cohere Embed v4 构建的三层底座架构。随后，我们将介绍嵌入 20 亿份文档的数据流水线，以及在生产环境中验证该架构的客诉分拣助手（Complaint Triage Assistant）。

为什么做底座，而不是单一功能

与 AWS 展开的构想研讨会梳理出了四个高影响力的用例：客诉总结与分拣、审计就绪状态监控、事件叙述生成，以及预测性质量风险评分。研讨会同时也揭示了一个深刻影响后续决策的洞见：数十个应用程序，各自拥有完全不同的文档类型和工作流。如果孤立地为其中某一个构建 AI，就意味着下一个功能还要重新构建底层基础设施。

因此，团队逆转了计划：先构建横向的平台底座，然后再向其中切入纵向的功能切片。客诉总结与分拣最先上线，用于在其他功能跟进之前验证这套架构。

四大矛盾与挑战

在选择任何具体模型之前，团队花费大量时间应对四个相互冲突的问题。其中任何一个都不可妥协。

在不冒意外泄露风险的前提下使数据可用。 投诉记录、CAPA、标准作业程序（SOP）和审计发现均归单一租户所有，必须对其他租户完全不可达。哪怕是一次错误的 AI 查询也绝不能发生跨租户访问。向量数据库、检索逻辑和生成上下文都必须严格限定在单一租户范围内。

在隔离环境中执行 Agent。 控制 Agent 的工具、数据存储和模型端点的权限必须限定在发起请求的用户本人，而不是共享的服务角色（service role）；这些权限只能在执行期间有效，并且必须尊重该用户的实际访问权限。采用带有静态凭据的共享 Agent 池在这三点上均无法满足要求。

读取不可信的内容。 此处的 Agent 需要摄入来自组织外部的投诉、电子邮件附件、扫描版 PDF 和图像。OWASP 大语言模型（LLM）应用程序十大安全威胁（2025）直接指出了适用的风险：通过摄入内容进行的提示词注入（LLM01）、敏感信息泄露（LLM02）、不当的输出处理（LLM05）以及过度代理权限（LLM06）。在如此庞大的数据量下，附件 PDF 中暗藏恶意隐藏指令绝非危言耸听。

在无需耗费整整一个季度等待的前提下嵌入 20 亿份文档。 Reliance 托管了超过 20 亿份文档，单个租户就存储着数百万份 Word 文件、PDF、扫描表单、电子表格、图像和电子邮件。团队承诺即使是规模最大的租户，也必须在一个工作日内完成上线导入。因为这些文档直接用于指导监管决策，所以绝不能牺牲质量来换取吞吐量。哪怕遗漏一条投诉，都会动摇用户对整个系统的信任。

最终确立的架构

同时兼顾这四大张力极大地收窄了技术选型范围。Amazon Bedrock 将它们作为一个统一平台整体解决：通过统一 API 实现多模型访问、依托现有 AWS 账户已严格执行的 AWS Region 数据驻留与 AWS Identity and Access Management (IAM) 治理，以及提供开箱即用、无需自行构建的 Guardrails 生产级内容安全管控能力。

该底座架构划分为三层，如图 1 所示。数据准备层通过 Amazon Bedrock 上的 Cohere Embed v4 对质量文档进行嵌入编码，并将其写入 Amazon OpenSearch Service 上的独立单租户向量库。安全执行层将用户请求通过 AgentCore Gateway（Amazon Bedrock AgentCore 的一项功能）进行传递，在将请求交给调用 Amazon Bedrock 上 Anthropic Claude 模型的隔离微虚拟机（microVM）会话之前，完成认证鉴权与策略校验。作为两层的外层防护，内容保护机制双重应用了 Amazon Bedrock Guardrails：在输入进入时执行输入过滤，在任何内容抵达处理人员前执行输出过滤。检索、执行与审查保持在各自独立的阶段，从而使每一层都能独立执行并提供合规审计证据。

图 1：Reliance AI 底座的高层架构图

选择如何表征数据

在文档实现精准检索之前，其他一切都是空谈。这意味着检索必须准确无误、跨越多种语言与格式，并且绝不跨越租户边界。团队基于自有的实际语料库而非公开排行榜来评估嵌入模型。决定性的考量完全源自这类特定数据：一张外壳破裂的照片与描述该缺陷的段落之间有何关联，德语投诉与英语 CAPA 之间又有何关联。

Amazon Bedrock 上的 Cohere Embed v4 凭借五个维度的优势胜出，而这些优势彼此紧密关联。首先决定的，是它减少了多少不必要的流水线构建工作。质量记录除了文本外，还包含缺陷产品照片、扫描的检验表单和带有注释的图表。Embed v4 将这两种模态置于统一的向量空间中。破裂外壳的照片与描述该缺陷的段落可以通过单一查询一并检索出来，无需为每种模态维护独立的流水线。原生多语言支持在另一个维度上实现了同样的效果：文档以多种语言呈现，往往存在于同一个租户的语料库内部，并且无需经过翻译预处理。这意味着团队省去了构建两条复杂流水线的工作，同时也减少了文档在检索链路中丢失的两个潜在环节。

其余三个维度决定了向量的驻留位置及其成本。客户数据绝不能离开其所在的 AWS Region（即使是推理也不行），因此在 Amazon Bedrock 上获得区域内的模型可用性是先决条件，而不仅是一种便利。Embed v4 支持在由美国东部（弗吉尼亚北部）区域（us-east-1）、欧洲（爱尔兰）区域（eu-west-1）和亚太地区（东京）区域（ap-northeast-1）提供的区域内推理，满足了 Octave 的三大区域合规要求。模型和功能的可用性因区域而异且随时间演进。有关当前支持范围，请参阅 Amazon Bedrock 用户指南中的各模型区域可用性。在每个区域内，Amazon OpenSearch Service 的成本随着向量维度的增加呈线性增长。这使得测试 Embed v4 支持的 256–1,536 可配置输出维度成为一件极具价值的事情，而不是直接沿用默认值。在针对典型生产查询的测试中，512 维在保持目标精度的同时，相比 1,536 默认维度将存储成本削减了约 66%。而其 128K token 的上下文窗口则在另一个完全不同的地方展现了价值：约 4K token 的文本块可以密集打包在单次请求中，这也是实现极高租户接入吞吐量的关键所在。

在运行时保持租户隔离

更棘手的问题是在何处实施租户隔离。在应用层代码中实现固然可行，但这意味着未来的每一个功能开发团队在每条代码路径上都必须万无一失，任何一处漏检都可能导致意外的跨租户泄露风险。因此，团队选择将隔离机制下沉到运行时环境中。Amazon Bedrock AgentCore 的能力之一 —— AgentCore 运行时，在应用程序层之下强制执行隔离，并且由于它与具体的框架和模型解耦，客诉分拣 Assistant 所选用的 Agent 框架和模型不会受到隔离决策的束缚。

每个用户会话都运行在拥有独立 CPU、内存和文件系统的专属微虚拟机（microVM）中，并在会话结束时立即停止并清理净化内存。这种硬件级别的隔离旨在确保无论 Agent 作出何种行为，隔离机制始终稳固。AgentCore 会将已认证的用户身份贯穿至整个执行链路中：每用户独立的 OAuth 令牌保险库确保 Agent 代表该具体用户行动，而非使用共享的服务身份，且凭据按 Agent 划分作用域，仅在会话期间存活。AgentCore Gateway 在工具、操作和参数级别执行精细授权。自动化推理技术依据既定规则提供形式化验证。这与 IAM 及 Amazon Simple Storage Service (Amazon S3) 策略验证背后的工程规范如出一辙。图 2 追踪了跨两个租户的这种隔离架构。

图 2：租户之间的 AgentCore 会话隔离。每个 microVM 在使用后均会停止并净化内存。

因此，由租户 A 的质量处理人员提交的客诉只会访问租户 A 的令牌保险库、工具以及位于 AgentCore Code Interpreter（Amazon Bedrock AgentCore 的一项功能）沙箱中的运行环境。租户 B 的会话绝无途径触及它们之中的任何一项。

将摄入的内容视为不可信输入

这种冲突没有简单的解法。客诉附件属于不可信输入，但它同时又是 Agent 必须读取的核心内容，因此绝不能直接拒绝可疑内容。团队最终确定了深度防御策略。

Octave 将 Amazon Bedrock Guardrails 部署在 Agent 逻辑的外层而非内部，嵌入在客诉工作流的三个关键节点上。ApplyGuardrail API 在文本向量嵌入之前评估从附件中提取的内容、在检索到的文档和工具输出进入 Agent 上下文之前进行评估，并在综合报告呈交给处理人员之前完成最终把关。提取自附件的内容在传入时会被打上输入标记（input tagging），明确标识为“数据”而非“指令”。正是这种机制使得隐藏在文档中间的注入攻击变得可被检测。同时，个人身份信息（PII）在进入系统时即被脱敏遮蔽。

仅靠检测依然不够，因此底层还配置了受约束的代理权限（bounded agency）。无论要求 Agent 执行什么任务，AgentCore Gateway 始终控制着其能够调用的工具，且文档提取在 AgentCore Code Interpreter 沙箱中运行，无外部网络出向访问权限，在会话结束后不保留任何持久状态。隐藏在 PDF 中的恶意载荷必须攻破每一层防线，而无法仅靠绕过单点防护。

成效： 检测和脱敏遮蔽率按租户分别追踪，因此对于拦截了哪些潜在风险，Octave 可以拿出确凿证据，而不仅是做出空泛断言。

驱动 20 亿份文档流经处理流水线

在向量嵌入模型之前，面临着一个更艰巨的挑战：从 Word 文档、电子表格、数字 PDF、扫描页面、照片和电子邮件线程中提取可用内容。多种语言交织出现，有时甚至混合在同一条记录中。而且流水线必须运行得足够快，使得租户接入能够在一天之内完成，而不是耗费整个季度。图 3 完整展示了最终成型的端到端处理流水线。

图 3：摄取流水线：从应用程序平台经由 Step Functions 编排到各租户专属 OpenSearch 索引的事件驱动型文档处理。

文档事件通过 Amazon Simple Notification Service（Amazon SNS）和 Amazon Simple Queue Service（Amazon SQS）分发到 AWS Step Functions 状态机中。该状态机在专用 Amazon Virtual Private Cloud（Amazon VPC）内部协调并发执行的 AWS Lambda，完成解析、分块和向量嵌入工作。由于并发并行能力来自于跨执行实例分发批次，而非单纯扩展单一资源，因此吞吐量可以在无需重构架构的情况下实现扩展。

解析阶段负责处理这种异构性：语义……

In quality management for regulated industries, the institutional memory that resolves a problem is almost always buried somewhere in a customer’s history. A similar defect investigated last year, a Corrective and Preventive Action (CAPA) that fixed the root cause, an audit finding that flagged the same gap. That knowledge sits in billions of documents no human can search by semantic similarity across languages and formats, so every investigation starts from scratch. Octave Reliance has helped each organization collect documents across multiple systems to help them search more easily and keep track of everything they have. The Octave Reliance team saw unlocking this history as its single largest opportunity to help customers work faster and stay compliant. The obstacle was equally clear: complaint records, audit findings, and corrective actions carry the platform’s strictest isolation requirements, and those were exactly the records the AI would need to read. In a platform serving over 14,000 customers across medical devices, pharmaceuticals, and food manufacturing, a single inadvertent disclosure from one customer to another would carry regulatory consequences far outweighing any AI benefit.

Octave operates across 180 countries, serving more than 60% of the Global Fortune 500. Its cloud-native quality management system, Reliance, supports over 650 enterprise customers with over 40 applications, from complaint management and CAPA tracking to audit management and supplier quality. Everyone generates regulated data that AI could reason over. Each customer is a separate tenant and no data can cross a tenant boundary.

Faced with that, the Reliance team made an unusual call: rather than shipping one AI feature, it spent its first cycle building a foundation every future feature would sit on. That foundation has since cut deployment time per use case by 50 percent. If you are bringing AI to a software as a service (SaaS) platform where the data is valuable, but isolation is non-negotiable, the reasoning travels beyond quality management. In this post, we walk through the four tensions that shaped the design and the three-layer foundation Octave built with Cohere Embed v4 on Amazon Bedrock. We then follow the pipeline that embedded 2 billion documents and the Complaint Triage Assistant that proved it in production.

Why a foundation, not a feature

Envisioning workshops with AWS surfaced four high-impact use cases: complaint summarization and triage, audit readiness monitoring, incident narrative generation, and predictive quality risk scoring. They also surfaced the insight that shaped everything after dozens of applications, each with different document types and workflows. Building AI for one in a silo meant rebuilding the infrastructure for the next.

So, the team inverted the plan: build the horizontal platform first, then cut vertical feature slices into it. Complaint summarization and triage went first, to prove the architecture before the rest followed.

The four tensions

Before selecting a single model, the team spent its time on four problems that pulled in different directions. None could be traded away.

Making data available without risking inadvertent disclosure. Complaint records, CAPAs, standard operating procedures (SOPs), and audit findings belong to one tenant and must be unreachable by another. Not even an errant AI query could cross. Vector store, retrieval logic, and generation context all had to be scoped to a single tenant.

Executing agents in isolation. The permissions governing an agent’s tools, data stores, and model endpoints had to be scoped to the requesting user rather than a shared service role, available only during execution, and respectful of that user’s access rights. A shared agent pool with static credentials failed all three.

Reading content that cannot be trusted. Agents here ingest complaints, email attachments, scanned PDFs, and images from outside the organization. The OWASP Top 10 for large language model (LLM) Applications (2025) names the threats that apply directly: prompt injection through ingested content (LLM01), sensitive information disclosure (LLM02), improper output handling (LLM05), and excessive agency (LLM06). Hidden instructions in an attached PDF are not hypothetical at this volume.

Embedding 2 billion documents without a quarter-long wait. Reliance hosts more than 2 billion documents, with individual tenants storing millions of Word files, PDFs, scanned forms, spreadsheets, images, and email. The team committed to onboarding even the largest tenant within a single working day. Because these documents inform regulatory decisions, quality could not be traded for throughput. A missed complaint erodes trust in the whole system.

Where the architecture landed

Holding four tensions at once narrowed the options considerably. Amazon Bedrock addressed them as one platform: multi-model access through a single API, the AWS Region residency and AWS Identity and Access Management (IAM) governance the AWS account already enforces, and Guardrails as production content controls rather than something to build.

The foundation resolves into three layers, shown in Figure 1. Data preparation embeds quality documents with Cohere Embed v4 on Amazon Bedrock and writes them to a per-tenant vector store on Amazon OpenSearch Service. Secure execution passes a user request through AgentCore Gateway, a capability of Amazon Bedrock AgentCore, which applies authentication and policy before handing it to an isolated microVM session that calls an Anthropic Claude model on Amazon Bedrock. Wrapping both, content protection applies Amazon Bedrock Guardrails twice: input filtering on the way in, output filtering before anything reaches the handler. Retrieval, execution, and screening stay in separate stages, so each can be enforced and evidenced on its own.

Figure 1: High-level architecture of the Reliance AI foundation

Choosing how to represent the data

Nothing else worked until the documents were retrievable. That meant accurately, across languages and formats, and not across a tenant boundary. The team evaluated embedding models against its own corpus rather than public leaderboards. The deciding questions were specific to this data: what a photo of a cracked housing has to do with the paragraph describing it, and a German complaint with an English CAPA.

Cohere Embed v4 on Amazon Bedrock won on five counts and they weren’t independent of each other. What was decided first was how much of the pipeline was made unnecessary. Quality records carry photos of defective products, scanned inspection forms, and annotated diagrams alongside text. Embed v4 places both modalities in one vector space. A photo of a cracked housing and the paragraph describing that defect are retrievable from a single query, with no separate pipeline per modality. Native multilingual support does the same work along the other axis: documents arrive in multiple languages, often inside one tenant’s corpus, and need no translation preprocessing. Two pipelines the team never had to build, and two fewer places for a document to fall out of retrieval.

The remaining three counts settled where the vectors could live and what they would cost. Customer data can’t leave its AWS Region, even for inference, so in-Region model availability on Amazon Bedrock was a precondition rather than a convenience. Embed v4 supports in-Region inference in the US East (N. Virginia) Region (us-east-1), Europe (Ireland) Region (eu-west-1), and Asia Pacific (Tokyo) Region (ap-northeast-1), covering three of Octave’s regional compliance requirements. Model and feature availability varies by Region and changes over time. For current coverage, see Regional availability by models in the Amazon Bedrock User Guide. Within each Region, Amazon OpenSearch Service costs scale linearly with dimension count. That made Embed v4’s configurable output dimensions of 256–1,536 worth testing rather than accepting by default. Testing against representative production queries showed 512 dimensions holding target accuracy while cutting storage roughly 66 percent against the 1,536 default. The 128K token context window paid off elsewhere entirely: chunks of roughly 4K tokens pack many to a request, which is where the onboarding throughput came from.

Keeping tenants apart at runtime

The harder question was where to enforce tenant separation. Application code would have worked, but it would have meant every future feature team getting it right on every code path, where one missed check risks inadvertent cross-tenant disclosure. The team pushed isolation into the runtime instead. AgentCore runtime, a capability of Amazon Bedrock AgentCore, enforces separation below the application layer, and because it stays framework- and model-independent, the agent framework and models chosen for the Complaint Triage Assistant were not constrained by the isolation decision.

Each user session runs in a dedicated microVM with its own CPU, memory, and filesystem, stopped and sanitized when the session ends. That is hardware-level separation designed to hold regardless of how the agent behaves. AgentCore carries the authenticated user’s identity through the whole execution chain: per-user OAuth token vaults verify the agent acts for that specific user rather than a shared service identity, and credentials are scoped per-agent and live only during the session. AgentCore Gateway authorizes at the tool, operation, and parameter level. Automated reasoning provides formal verification against the defined rules. It’s the same discipline behind IAM and Amazon Simple Storage Service (Amazon S3) policy validation. Figure 2 traces that separation across two tenants.

Figure 2: AgentCore session separation between tenants. Each microVM is stopped after use with memory sanitized.

A complaint submitted by a quality handler at Tenant A therefore reaches only Tenant A’s token vault, tools, and sandbox in AgentCore Code Interpreter, a capability of Amazon Bedrock AgentCore. Tenant B’s session has no path to any of them.

Treating ingested content as untrusted

This tension had no clean answer. A complaint attachment is untrusted input, but it’s also the thing the agent exists to read, so suspicious content couldn’t be rejected. The team settled on defense in depth.

Octave applied Amazon Bedrock Guardrails over the agent logic rather than inside it, at three points in the complaint workflow. The ApplyGuardrail API evaluates text extracted from attachments before embedding, retrieved documents and tool outputs before they enter the agent’s context, and the synthesized brief before the handler sees it. Attachment-derived content is passed with input tagging, marking it as data rather than instruction. That is what makes an injection buried mid-document detectable. Personally Identifiable Information (PII) is masked on the way in.

Detection alone wasn’t sufficient, so bounded agency sits underneath. Whatever an agent is asked to do, AgentCore Gateway still governs which tools it can invoke, and document extraction runs in the AgentCore Code Interpreter sandbox with no network egress and no persistence beyond the session. A payload hidden in a PDF has to defeat every layer, not one.

The effect: detection and masking rates are tracked per tenant, so what the screening caught is something Octave can evidence rather than assert.

Getting 2 billion documents through the pipeline

Ahead of the embedding model sat the harder problem: extracting usable content from Word files, spreadsheets, digital PDFs, scanned pages, photographs, and email threads. Several languages appear, sometimes inside one record. And it had to run fast enough that onboarding took a day rather than a quarter. Figure 3 traces the resulting pipeline end to end.

Figure 3: Ingestion pipeline: event-driven document processing from the application platform through Step Functions orchestration to the per-tenant OpenSearch index.

Document events fan out through Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) into an AWS Step Functions state machine. That state machine orchestrates parsing, chunking, and embedding across concurrent AWS Lambda executions inside a dedicated Amazon Virtual Private Cloud (Amazon VPC). Because parallelism comes from fanning batches across executions rather than scaling one resource, throughput grows without rearchitecting.

Parsing handles the heterogeneity: semantic hierarchy preserved from Word files and spreadsheets, optical character recognition (OCR) applied to scanned PDFs lacking a text layer, embedded images treated as first-class content. Chunks carry metadata: tenant, document type, source application, language, and a content hash. That metadata drives both retrieval filtering and per-tenant namespace enforcement.

The throughput came from packing. Interleaving many 4K chunks into single requests against Embed v4’s 128K context window cut the API calls for a large corpus by about an order of magnitude. In Octave’s internal testing on Amazon Bedrock, Embed v4 delivered 28 million tokens per minute. After tuning batch sizes with Cohere’s solutions architects until memory bottlenecks cleared, the pipeline sustained 1.4 million documents per 24 hours per tenant workload. That is what made the single-working-day commitment real.

The Complaint Triage Assistant

A hospital reports defective syringes to a medical device manufacturer, and the record lands with email, photos of damaged units, and supporting documentation attached. A quality handler has to piece the context together across those formats, work out whether anything like it has been reported before, and find what resolved it. Only then can a priority be assigned, often inside a reporting window measured in days.

The assistant is built with the Strands Agents SDK, open-sourced by AWS, which composes agentic workflows as graphs, with nodes as tool invocations or model calls and edges as the data dependencies between them. AgentCore runtime runs that graph and isolates it per session, and five stages carry a complaint from raw attachments to a finished brief.

Document extraction comes first. PDFs, images, email attachments, and videos all enter AgentCore Code Interpreter, which parses each artifact into a structured intermediate representation. A Claude model on Amazon Bedrock then summarizes the record, and that summary, rather than the raw multi-thousand-token source, becomes the retrieval query. Retrieval runs it against the tenant’s OpenSearch index of Embed v4 embeddings, surfacing similar past complaints and their CAPAs, with AgentCore Gateway enforcing that only authorized namespaces are reachable.

What comes back is not trusted on similarity alone. A separate model call scores results by confidence and discards matches that share vocabulary but differ in root cause. What survives goes to synthesis, where Anthropic Claude Sonnet on Amazon Bedrock assembles the output: key facts, sentiment, urgency indicators, the closest historical records with their outcomes, and a recommended triage priority.

Completed briefs are cached in Amazon DynamoDB with a per-environment time to live (TTL), so repeated views cost nothing extra. The handler no longer starts from zero, but from institutional memory.

Figure 4 places that engine behind a private API boundary. A portal request reaches AgentCore runtime through a private Amazon API Gateway and a Lambda function, and retrieval reads the document index through role-based access with pre-signed URLs while attachments stay in Amazon S3. Cached summaries and OpenTelemetry traces exit to the summarization table and the evaluation dashboard.

Figure 4: Summarizing engine: the AgentCore-powered workflow handling document extraction, retrieval, reasoning, and synthesis within a private API boundary, with response caching and full observability.

Figure 5 reduces the same path to a single line, with Guardrails filtering at both ends.

Figure 5: Complaint Triage Assistant pipeline from ingestion to handler output

Proving what the agent did

Observability here is not only how the engineering team debugs but also how they integrate with system auditing capabilities to meet compliance requirements. An auditor can ask why a particular complaint was triaged the way it was, and “the model decided” will not survive that conversation.

AgentCore captures OpenTelemetry-compatible traces across every node of the graph: each model call, retrieval, and AgentCore Code Interpreter run. Months later, Octave can show which retrieved documents and prompts produced a given brief. The model invocation logging of Amazon Bedrock adds a per-tenant record of every request and response, scoped through the IAM identity AgentCore propagates, with costs attributing to the initiating tenant in Amazon CloudWatch.

Offline, AgentCore Evals reruns the workflow against expert-annotated ground truth after any change to prompts, models, or retrieval parameters, reporting summarization faithfulness, retrieval precision at K, and output completeness. A prompt that reads better but retrieves worse gets caught before users see it.

Results

The Complaint Triage Assistant has been running in production for three months, and the following numbers reflect that period.

The handler who used to open an investigation with nothing now opens it with the relevant history already assembled. The 4–6 hours Octave measures per investigation is the work the pipeline absorbed: extraction across formats, the hunt for precedent, and the reading it takes to tell a genuinely similar defect from a coincidence of wording.

The foundation also changed how quickly Octave ships. By its own accounting, development cycles for new AI capabilities went from months to weeks. Shared data preparation, secure execution, and retrieval are the three layers of Figure 1. They cut deployment time per use case by 50 percent, because each new capability inherits them rather than rebuilding them. That landed semantic search in Q2, two quarters ahead of the original Q4 target.

Looking back

The decision to build a foundation first was the one the team second-guessed most. Months of infrastructure work is hard to defend when stakeholders want an AI feature they can see. The next round of use cases has borne it out. Two choices carried it: isolation pushed into microVM sessions and identity propagation rather than application code, and embedding models evaluated against Octave’s own corpus, which surfaced the 512-dimension trade-off no benchmark would have shown.

The team also drew a line it chose not to cross. The assistant surfaces context and precedent. The quality handler still makes the call. In an industry where that call carries regulatory weight, that boundary shaped what got automated and what did not.

What’s next

Octave is now extending the foundation across the Reliance suite. Audit readiness monitoring, incident narrative generation, and predictive quality risk scoring are next, each building on the same infrastructure.

If your platform faces a similar tension: valuable data behind hard isolation requirements. Octave’s experience shows the trade-offs are workable. To learn more about the services behind this story, see Amazon Bedrock, the Amazon Bedrock AgentCore documentation, and Cohere on Amazon Bedrock. For related architectural guidance, see AWS vector solutions: Build agentic AI where your data lives and Scaling agentic AI: Enterprise patterns without vendor lock-in.

---

About the authors

Urvish Saraiya

Urvish is an engineering leader at Octave, where he is building an industrial AI platform on AWS. With over 18 years of experience in distributed software systems, data platforms, and machine learning, he brings deep technical expertise across public safety, payments, and digital marketing. Urvish combines strong technical skills with a strategic, product minded approach, focused on solving business needs with commercial impact.

Manu Mishra

Manu is a Senior Solutions Architect at AWS with over 18 years of experience in the software industry, specializing in artificial intelligence, data and analytics, and security. His expertise spans strategic oversight and hands-on technical leadership, where he reviews and guides the work of both internal and external customers. You can connect with or follow Manu on LinkedIn.

Madhunika Reddy Mikkili

Madhunika is a Data and Machine Learning Engineer at AWS. She helps enterprise customers solve complex business problems with Generative AI and Machine Learning.

Payal Singh

Payal is a Solutions Architect at Cohere with over 15 years of cross-domain expertise in DevOps, Cloud, Security, SDN, Data Center Architecture, and Virtualization. She drives partnerships at Cohere and helps customers with complex GenAI solution integrations.

Nate Stecz

Nate is a Senior Partner Development Manager who helps leading generative AI model providers grow their revenue through AWS partnership. Nate has been with Amazon Web Services for nearly 10 years. Nate is located in Seattle, WA, and enjoys outdoor activities and learning about emerging technologies. You can connect with or follow Nate on LinkedIn.