利用查询感知压缩降低 Amazon Bedrock 上的 RAG 成本

每次调用发送给基础模型（FM）的输入 token，往往是大规模运行检索增强生成（RAG）时成本的重要组成部分。查询感知压缩提供了一种减少到达模型的输入 token 数量的方法。Amazon Bedrock提供了构建 RAG 应用所需的基础模型和功能。RAG 检索通常针对高召回率进行调优，会返回一组范围较广、可能相关的文本块，以便主模型拥有充足的源材料可供处理。这种设计有助于构建者确信，在推理时能够获得正确的信息。随着工作负载扩展，构建者通常会设法优化成本与性能之间的权衡：在保持答案质量的同时，减少主模型需要处理的输入 token 数量。Amazon Bedrock 的开放式可组合架构支持自定义检索后处理步骤，从而优化最终发送给主模型的内容。

本文介绍一种检索后自定义模式，可在保持答案质量的同时显著减少输入 token，进而节省成本。它兼容 Amazon Bedrock 上的 RAG 检索器，包括 Amazon Bedrock Knowledge Bases。作为额外收益，移除无关上下文还会缩小模型产生幻觉的空间。在检索完成、最终答案调用开始之前，Amazon Bedrock 上一个规模更小、成本更低的模型会根据用户查询筛选检索到的文本块。随后，主模型接收筛选后的上下文并生成答案。

我们将从高层介绍该模式的架构，展示在 AWS Lambda 函数中的核心 Amazon Bedrock 实现，讲解成本模型与延迟权衡，并说明我们如何评估答案质量。我们还将探讨这一模式如何叠加现有 Amazon Bedrock 功能，例如提示词缓存、Amazon Bedrock Intelligent Prompt Routing 以及 Rerank API，以获得叠加式成本节省。

前提条件

要实现该方案，请完成以下前提步骤：

1. 拥有一个有效的 AWS 账户。
2. 为 Lambda 函数创建一个 AWS Identity and Access Management（IAM）角色，使其能够访问 Amazon Bedrock。有关说明，请参阅创建角色以将权限委托给 AWS 服务。
3. 向 IAM 角色添加策略权限。
4. 在您的 AWS 区域中获取级联所用两个模型的 Amazon Bedrock 模型访问权限：较小的压缩模型（本文使用 Anthropic Claude Haiku）和主模型（Anthropic Claude Sonnet）。在 Amazon Bedrock 控制台的“模型访问”页面申请访问权限。

查询感知压缩如何降低 Amazon Bedrock 上的 RAG 成本

使用传统 RAG 基础设施或框架的 RAG 流程如下：

1. 应用程序嵌入用户查询。
2. 向量索引返回 top-k 个文本块（通常为 5–20 个）。
3. 可选地，重排序器会按相关性重新排列这些文本块。
4. 将所有检索到的文本块拼接到提示词中。
5. 主模型生成答案。

检索上下文的规模随 top-k 和文本块大小而变化：在典型文本块大小下，检索 5–20 个文本块会使许多技术文档和法律领域 RAG 工作负载的每次查询输入达到数千 token。减少每次查询的 token 数量可以带来显著的成本节省。

架构概述

较小的模型会结合查询读取检索到的文本块，并仅输出与问题相关的原文片段。本文使用 Claude Haiku，但该模式也适用于 Amazon Bedrock 中同一模型系列内的其他小模型／主模型组合。压缩调用和主模型的答案调用都在单个 AWS Lambda 函数中运行。在上游，检索器对查询进行嵌入并返回 top-k 个文本块。由 Amazon OpenSearch Serverless 支持、完全托管的 RAG 功能 Amazon Bedrock knowledge base，就是这样一种检索器。Lambda 函数接收这些文本块作为输入，并返回最终答案。与标准 RAG 流程相比，唯一新增的步骤就是压缩调用。

由于较小模型的每 token 成本低于主模型，因此，在成本更高的答案调用之前裁剪上下文正是成本节省的来源。节省幅度取决于两个因素。

下图展示了解决方案架构。

图 1：Amazon Bedrock 上的查询感知上下文压缩架构

流程按以下步骤进行：

1. 用户向应用程序提交查询。
2. 应用程序将查询发送给检索器，检索器搜索向量化的知识源。
3. 检索器将 top-k 个文本块（完整检索上下文）返回给应用程序。
4. 应用程序将查询和所有检索到的文本块传递给 AWS Lambda 函数；该函数通过 Amazon Bedrock Converse API 将它们发送给较小模型（Claude Haiku）。这就是压缩调用。
5. 较小模型仅输出与查询相关的原文片段（压缩后的上下文）。
6. Lambda 函数通过 Converse API 将查询和压缩后的上下文发送给主模型（Claude Sonnet）。这就是答案调用。
7. 主模型生成最终答案，应用程序再将答案返回给用户。

这一模式为何具备经济性

其经济性取决于两个因素：Amazon Bedrock 上小模型与主模型之间的价格比，以及小模型实现的压缩比。

对于单次 RAG 查询，设检索到的输入 token 数为 R、压缩比为 c（其中 c > 1）、最终答案输出为 A 个 token，且每 token 价格分别为 P_small_in / P_small_out（小模型输入和输出价格）以及 P_large_in / P_large_out（主模型输入和输出价格）：

阶段	基线	使用压缩后	发生的情况
小模型读取文本块	—	R × P_small_in	小模型接收查询及全部检索文本块作为输入
小模型生成压缩输出	—	(R/c) × P_small_out	小模型仅输出与查询相关的原文片段
主模型读取上下文	R × P_large_in	(R/c) × P_large_in	主模型接收上下文作为输入以生成答案
主模型生成答案	A × P_large_out	A × P_large_out	主模型为用户生成最终答案
总计	R·P_large_in + A·P_large_out	R·P_small_in + (R/c)·P_small_out + (R/c)·P_large_in + A·P_large_out

压缩调用增加了小模型的输入和输出成本。成本节省来自向主模型发送 R/c 而不是 R 个 token。当满足以下条件时，压缩在经济性上更具优势：

• 检索到的上下文较大（在较大模型上节省的成本占主导）。
• 大模型与小模型之间的价格比很高（例如 Sonnet/Opus 与 Haiku 配对）。
• 对于特定查询，在不影响答案质量的情况下，可以裁剪掉相当一部分检索内容。

在 Amazon Bedrock 上实现

该模式位于检索与最终答案调用之间。我们将其实现为单个 AWS Lambda 函数，通过 Converse API 编排两次 Amazon Bedrock 模型调用。该函数将用户查询和检索到的文本块作为输入事件。

压缩提示词

压缩提示词是实现中最重要的部分。它必须指示较小模型提取片段而非进行摘要，禁止其改写和重写，并保留足够的周边上下文，以确保引用仍然准确。

COMPRESSION_SYSTEM_PROMPT = """You extract evidence from retrieved documents.
You will receive a user QUESTION and a list of CHUNKS.

Your job:
1. For each chunk, identify the spans (verbatim sentences or short paragraphs)
that contain evidence directly relevant to answering the QUESTION.
2. Output ONLY those spans, copied verbatim from the source. Do not paraphrase,
summarize, or rewrite.
3. Preserve chunk identifiers so the downstream system can cite sources.
4. If a chunk contains no relevant evidence, output the chunk identifier
followed by NO_RELEVANT_EVIDENCE.

Output format (strict):
[CHUNK_ID: <id>]
<verbatim span 1>
<verbatim span 2>
[CHUNK_ID: <id>]
NO_RELEVANT_EVIDENCE

Do not add commentary, headings, conclusions, or your own words. Only verbatim
spans from the source chunks, grouped by chunk identifier."""

Lambda 函数

该函数接收用户查询和检索到的文本块，从环境变量中读取两个模型 ID，并初始化一个配置了自适应重试的 Amazon Bedrock Runtime 客户端。随后，它通过 Bedrock Converse API 发起两次调用：第一次调用较小模型压缩文本块，第二次调用主模型基于压缩后的证据生成答案。压缩调用的 temperature 设置为 0.0，以保持提取结果的确定性，使较小模型按源文本中的原样复制片段：

第 1 步：压缩。较小模型筛选检索到的文本块。

compressed = bedrock.converse(
    modelId=COMPRESSION_MODEL_ID,
    system=[{"text": COMPRESSION_SYSTEM_PROMPT}],
    messages=[{"role": "user",
               "content": [{"text": f"QUESTION:\n{query}\n\nCHUNKS:\n{chunks}"}]}],
    inferenceConfig={"temperature": 0.0},
)

第 2 步：回答。主模型基于筛选后的证据进行推理。

answer = bedrock.converse(
    modelId=ANSWER_MODEL_ID,
    system=[{"text": "Answer using only the evidence provided. Cite sources."}],
    messages=[{"role": "user",
               "content": [{"text": f"QUESTION:\n{query}\n\nEVIDENCE:\n{compressed}"}]}],
)

**注意：**这些提示词仅为示例，应根据您的文档和问题类型进行调整。

方法论

在推荐该模式之前，我们对其进行了实证评估。基准测试涵盖：

• 一个包含超过 50 万份文档的语料库，覆盖 9 类企业数据源，包括聊天消息、电子邮件、问题跟踪器工单、共享驱动器文档、CRM 记录、会议记录、代码仓库和 Wiki 页面。
• 一组包含 10 个类别的 500 个问题，范围从狭义事实查询到更广泛的多部分问题，同时包含措辞精确和表达更口语化的查询。
• 多种压缩条件：不压缩（基线）、压缩，以及重排序加压缩。
• 每个查询均通过基线和优化后的流水线运行，并由大语言模型（LLM）裁判对答案评分。
• 相对于参考答案，在四个维度上评估质量：正确性、完整性、引用准确性和简洁性。另行跟踪忠实度，即根据主模型接收到的证据核查每个答案中的主张。

这些数据反映的是一个语料库、一个领域和一种查询分布。您自己的文档和查询上的结果会有所不同。

结果

下表汇总了各查询的核心结果，对比基线、压缩和重排序 + 压缩三种流水线。

指标	基线	压缩	重排序 + 压缩
成本	100%（参考）	67%	64%
发送给模型的 token	100%（参考）	12%	10%
延迟	0%（参考）	+19%（更慢）	+12%（更慢）
质量（综合，4 个维度）	100%（参考）	97.5%	97.6%
幻觉率	51%	−7 个百分点	−13 个百分点

下列图表来自该基准测试。您自己的语料库、查询和模型选择的结果将有所不同。下图展示了查询成本节省的平均值（左轴，相对于基线的百分比）以及发送给主模型的上下文减少幅度（右轴，以 token 数减少倍数表示）。压缩实现了 33% 的成本节省，或 token 数减少 8.6 倍。重排序 + 压缩实现了 36% 的成本节省和 token 数减少 10.1 倍。

图 2：成本节省以及发送给主模型的上下文 token 减少情况

下图展示了 LL

Input tokens sent to the foundation model (FM) on every call are often a meaningful part of the cost of running Retrieval Augmented Generation (RAG) at scale. Query-aware compression offers one way to reduce how many of them reach the model. Amazon Bedrock provides the foundation models and features to build RAG applications. RAG retrieval usually tunes for high recall, returning a broad set of potentially relevant chunks so the primary model has thorough source material to work with. This design helps builders feel confident that the right information is available at inference time. As workloads scale, builders often look for ways to optimize the cost-performance tradeoff by reducing the number of input tokens the primary model processes while maintaining answer quality. The open, composable architecture of Amazon Bedrock supports custom post-retrieval processing steps that refine what reaches the primary model.

In this post, we describe a post-retrieval customization pattern that achieves significant input-token reduction, and therefore cost savings, while preserving answer quality. It’s compatible with RAG retrievers on Amazon Bedrock, including Amazon Bedrock Knowledge Bases. As a secondary benefit, removing irrelevant context reduces the surface area for hallucination. After retrieval but before the final answer call, a smaller, lower-cost model on Amazon Bedrock filters retrieved chunks against the user’s query. The primary model then receives the filtered context and generates the answer.

We cover the pattern’s architecture at a high level, show the core Amazon Bedrock implementation in a AWS Lambda function, walk through the cost model and the latency tradeoff, and describe how we evaluated answer quality. We also look at how this pattern can layer on top of existing Amazon Bedrock capabilities like prompt caching, Amazon Bedrock Intelligent Prompt Routing, and the Rerank API for compounding cost savings.

Prerequisites

To implement the solution, complete the following prerequisite steps:

1. Have an active AWS account.
2. Create an AWS Identity and Access Management (IAM) role for the Lambda function to access Amazon Bedrock. For instructions, refer to Create a role to delegate permissions to an AWS service.
3. Add policy permissions to the IAM role.
4. Amazon Bedrock model access in your AWS Region for the two models in the cascade: a smaller compression model (Anthropic Claude Haiku in this post) and a primary model (Anthropic Claude Sonnet). Request access on the Amazon Bedrock console under Model access.

How query-aware compression reduces RAG costs on Amazon Bedrock

A RAG flow using traditional RAG infrastructure or frameworks looks like:

1. The application embeds the user query.
2. The vector index returns the top-k chunks (commonly 5–20).
3. Optionally, a reranker reorders those chunks by relevance.
4. All retrieved chunks are concatenated into the prompt.
5. The primary model generates the answer.

Retrieved context scales with top-k and chunk size: retrieving 5–20 chunks at typical chunk sizes puts many technical-documentation and legal RAG workloads in the range of several thousand input tokens per query. Reducing the per-query token count can yield meaningful cost savings.

Architecture overview

A smaller model reads the retrieved chunks alongside the query and outputs only the verbatim spans relevant to the question. We use Claude Haiku in this post, but the pattern works with other small/primary model pairs within a model family on Amazon Bedrock. Both the compression call and the primary model’s answer call run inside a single AWS Lambda function. Upstream, a retriever embeds the query and returns the top-k chunks. An Amazon Bedrock knowledge base, the fully managed RAG capability backed by Amazon OpenSearch Serverless, is one such retriever. The Lambda function receives those chunks as input and returns the final answer. The compression call is the only step added to a standard RAG flow.

Because the smaller model costs less per token than the primary model, trimming the context before the expensive answer call is where the savings come from. How large those savings are comes down to two things.

The following diagram shows the solution architecture.

Figure 1: Query-aware context compression architecture on Amazon Bedrock

The flow proceeds through the following steps:

1. The user submits a query to the application.
2. The application sends the query to the retriever, which searches the vectorized knowledge sources.
3. The retriever returns the top-k chunks (full retrieved context) to the application.
4. The application passes the query and all retrieved chunks to an AWS Lambda function, which sends them to a smaller model (Claude Haiku) through the Amazon Bedrock Converse API. This is the compression call.
5. The smaller model outputs only the verbatim spans relevant to the query (the compressed context).
6. The Lambda function sends the query and the compressed context to the primary model (Claude Sonnet) through the Converse API. This is the answer call.
7. The primary model generates the final answer, which the application returns to the user.

Why this works economically

The economics depend on two factors: the price ratio between the small and primary models on Amazon Bedrock, and the compression ratio the smaller model achieves.

For a single RAG query with R retrieved input tokens, a compression ratio of c (where c > 1), a final answer output of A tokens, and per-token prices P_small_in / P_small_out (smaller model input and output price) and P_large_in / P_large_out (primary model input and output price):

Stage	Baseline	With compression	What happens
Smaller model reads chunks	—	R × P_small_in	The smaller model receives the query and all retrieved chunks as input
Smaller model writes compressed output	—	(R/c) × P_small_out	The smaller model outputs only the verbatim spans relevant to the query
Primary model reads context	R × P_large_in	(R/c) × P_large_in	The primary model receives the context as input to generate the answer
Primary model writes answer	A × P_large_out	A × P_large_out	The primary model generates the final answer for the user
Total	R·P_large_in + A·P_large_out	R·P_small_in + (R/c)·P_small_out + (R/c)·P_large_in + A·P_large_out

The compression call adds the input and output cost of the smaller model. Savings come from sending R/c instead of R tokens to the primary model. The economics favor compression when:

• Retrieved context is large (the savings on the larger model dominate).
• The price ratio between large and small is high (Sonnet/Opus paired with Haiku).
• A meaningful portion of the retrieved content can be trimmed for a given query without affecting answer quality.

Implementation on Amazon Bedrock

The pattern fits between retrieval and the final answer call. We implement it as a single AWS Lambda function that orchestrates the two Amazon Bedrock model invocations using the Converse API. The function receives the user query and the retrieved chunks as its input event.

The compression prompt

The compression prompt is the most important part of the implementation. It must instruct the smaller model to extract spans rather than summarize, forbid paraphrasing and rewriting, and preserve enough surrounding context for citations to remain accurate.

COMPRESSION_SYSTEM_PROMPT = """You extract evidence from retrieved documents.
You will receive a user QUESTION and a list of CHUNKS.

Your job:
1. For each chunk, identify the spans (verbatim sentences or short paragraphs)
that contain evidence directly relevant to answering the QUESTION.
2. Output ONLY those spans, copied verbatim from the source. Do not paraphrase,
summarize, or rewrite.
3. Preserve chunk identifiers so the downstream system can cite sources.
4. If a chunk contains no relevant evidence, output the chunk identifier
followed by NO_RELEVANT_EVIDENCE.

Output format (strict):
[CHUNK_ID: <id>]
<verbatim span 1>
<verbatim span 2>
[CHUNK_ID: <id>]
NO_RELEVANT_EVIDENCE

Do not add commentary, headings, conclusions, or your own words. Only verbatim
spans from the source chunks, grouped by chunk identifier."""

The Lambda function

The function takes the user query and the retrieved chunks, reads the two model IDs from environment variables, and initializes an Amazon Bedrock Runtime client configured with adaptive retries. It then makes two calls through the Bedrock Converse API: the first to the smaller model to compress the chunks, and the second to the primary model to generate the answer from the compressed evidence. The compression call runs at temperature 0.0, which keeps the extraction deterministic so the smaller model copies spans as they appear in the source:

Step 1: Compress. The smaller model filters the retrieved chunks.

compressed = bedrock.converse(
    modelId=COMPRESSION_MODEL_ID,
    system=[{"text": COMPRESSION_SYSTEM_PROMPT}],
    messages=[{"role": "user",
               "content": [{"text": f"QUESTION:\n{query}\n\nCHUNKS:\n{chunks}"}]}],
    inferenceConfig={"temperature": 0.0},
)

Step 2: Answer. The primary model reasons over the filtered evidence.

answer = bedrock.converse(
    modelId=ANSWER_MODEL_ID,
    system=[{"text": "Answer using only the evidence provided. Cite sources."}],
    messages=[{"role": "user",
               "content": [{"text": f"QUESTION:\n{query}\n\nEVIDENCE:\n{compressed}"}]}],
)

Note: The prompts are an example and should be adapted to your documents and question types.

Methodology

Before recommending this pattern, we evaluated it empirically. The benchmark covered:

• A corpus of more than 500,000 documents spanning 9 enterprise source types, including chat messages, email, issue-tracker tickets, shared-drive documents, CRM records, meeting transcripts, code repositories, and wiki pages.
• A set of 500 questions across 10 categories, ranging from narrow factual lookups to broader multi-part questions, including both precisely worded and more informally phrased queries.
• Multiple compression conditions: no compression (baseline), compression, and rerank-plus-compression.
• Each query run through baseline and optimized pipelines, with answers scored by a large language model (LLM) judge.
• Quality scoring across four dimensions (correctness, completeness, citation accuracy, and conciseness) against a reference answer. Faithfulness was tracked separately, checking each answer’s claims against the evidence the primary model received.

These figures describe one corpus, one domain, and one query distribution. Results on your own documents and queries will differ.

Results

The following table summarizes the headline results across the queries, comparing the baseline, compression, and rerank + compression pipelines.

Metric	Baseline	Compression	Rerank + Compression
Cost	100% (ref)	67%	64%
Tokens to model	100% (ref)	12%	10%
Latency	0% (ref)	+19% (slower)	+12% (slower)
Quality (composite, 4 dims)	100% (ref)	97.5%	97.6%
Hallucination rate	51%	−7 pts	−13 pts

The following figures come from the benchmark. Results on your own corpus, queries, and model choices will differ. The following figure shows the average of query cost saving (left axis, percentage versus baseline) and the reduction in context sent to the primary model (right axis, times fewer tokens). Compression achieved a 33 percent cost saving or 8.6× fewer tokens. Rerank + compression reached 36 percent cost saving and 10.1× fewer tokens.

Figure 2: Cost savings and reduction in context tokens sent to the primary model

The following figure shows the LLM-judge scores (1–5) across the four answer-quality dimensions for each pipeline. Correctness stays within 0.07 of baseline across conditions. Completeness and citation accuracy are slightly lower under compression, while conciseness is slightly higher.

Figure 3: LLM-judge scores across the 4 answer-quality dimensions by pipeline

The following figure shows the hallucination rate for each pipeline, measured as the share of answers containing at least one claim not supported by the reference. The baseline is 51 percent, compression 44 percent, and rerank + compression 38 percent.

Figure 4: Hallucination rate by pipeline

The following figure shows the cost saving versus baseline for the typical-query set and the hard-query set. Compression moves from 37 percent to 26 percent, and rerank + compression from 40 percent to 30 percent between the two sets.

Figure 5: Cost savings for the typical-query and hard-query sets

Considerations for production use

Three things must be weighed before this pattern is adopted: the latency of the added compression call, the impact on answer quality, and whether the workload is a fit. Each is covered in the following sections.

Latency

Adding a smaller-model call introduces one extra step in the path. Claude Haiku is optimized for speed, and because the primary model then processes a smaller, focused context, part of that added time is recovered on the answer call. The total end-to-end latency is the compression call plus the answer call on the focused context. Net impact depends on how compute-bound the primary model is on the original context size.

For latency-critical surfaces (sub-second chat), measure with your own context sizes before deploying.

Quality

Compression involves a few factors worth deliberate engineering.

Dimension	Design consideration	How the pattern handles it
Factual consistency	The primary model should receive all the evidence it needs	The prompt instructs faithful, verbatim extraction of all query-relevant evidence
Citation preservation	Spans stay tied to their source chunk	[CHUNK_ID: <id>] markers carried through the prompt
Faithful extraction	Output stays verbatim from the source	temperature=0.0, “verbatim only” instruction, optional post-hoc span validation
Multi-step reasoning	Evidence chains stay intact for analytical questions	The compression model preserves more spans when the question draws on multiple sources
Numerical reasoning	Specific numbers are retained	The “verbatim only” instruction preserves numbers and calculations as they appear in the source

Choosing the smaller model

A few considerations can help guide the selection of the smaller model:

• Relative pricing. The savings of this pattern depend on the price ratio between the smaller model and the primary model. A lower relative price for the smaller model leaves more room for net savings once the cost of the compression call is included. Amazon Bedrock model families span a range of price points, with current rates available on the Amazon Bedrock pricing page.
• Inference speed. The compression call is added to the request path, so a smaller model optimized for fast inference limits the additional latency.
• Model-family alignment. Selecting the smaller and primary models from the same family keeps formatting, chunk markers, and instruction handling consistent across both calls. Claude Haiku with Claude Sonnet, and Amazon Nova Micro with Amazon Nova Pro, are examples of such pairings on Amazon Bedrock. For model availability by Region, refer to Supported models by AWS Region in Amazon Bedrock.

Recommended evaluation approach before deploying

• Assemble an evaluation set of representative queries from your application logs (de-identified), covering both narrow lookups and broader multi-fact questions in the proportion they occur in production.
• Run the baseline and compressed pipelines on the same retrieval results, so the comparison reflects only the effect of compression.
• Score both sets of answers on the dimensions that matter for the workload, such as correctness, completeness, and citation accuracy against a reference answer, along with faithfulness against the evidence the primary model received, which surfaces claims not supported by that evidence.
• Set acceptance thresholds that fit the workload, and promote the configuration after it meets them.

For production deployments, Amazon Bedrock Guardrails can provide additional content filtering and grounding validation as complementary controls alongside the compression pattern.

Where this pattern fits well

The pattern provides value when three conditions line up. Retrieved context is large, the primary model is the expensive part of the call, and the question is narrow relative to the breadth of what was retrieved. The following scenarios share these characteristics.

• Regulated-industry compliance and policy assistants. Utilities, financial services, insurance, and healthcare keep long bulletins, tariffs, and policy documents where a single section answers most questions. Retrieval pulls whole bulletins (8,000–15,000 tokens) but the analyst needs one clause. Verbatim extraction preserves the exact regulatory wording and the source ID, which matters when the answer has to be defensible in an audit.
• Customer-support copilots over a large product knowledge base. Top-k is set high so the assistant is less likely to miss an edge case, meaning most tickets retrieve far more than they need. A “how do I reset X” or “what’s the fee for Y” ticket usually resolves from one or two paragraphs buried in a long article. Compression cuts the per-ticket inference cost without touching the retrieval strategy support engineers already trust. At support volumes (hundreds of thousands of tickets a month), small per-query savings accumulate quickly.
• Internal engineering and operations assistants over runbooks and wikis. Confluence, SharePoint, and runbook RAG retrieves large chunks because internal docs are verbose and loosely structured. Questions are usually specific (“which IAM role does service Z assume”, “what’s the rollback step for migration N”). The answer is a few lines. The surrounding policy and background is not needed for the response.
• Financial research and earnings-call analysis. Retrieved chunks are dense (10-K sections, transcript paragraphs) and questions are pointed (“what did the CFO say about FX exposure in Q3”). Compression preserves the analyst’s ability to quote the source verbatim while reducing the per-query cost on the larger model.

This pattern is likely worth a prototype if most of these are true for your workload:

• Average retrieved context per query exceeds roughly 5,000 tokens.
• The primary model’s input tokens are expensive enough that a 3–10× reduction in context is material.
• Most questions are narrow relative to the volume retrieved, rather than broad “summarize everything” requests.
• Your latency budget can absorb one extra model call (a few hundred milliseconds to approximately 1 second).
• You have, or can build, an evaluation set to compare quality before and after.

If your workload is sub-second conversational chat with small retrieved context, this pattern may not be the best fit. Consider Amazon Bedrock features like prompt caching and Intelligent Prompt Routing, which provide more value with less added latency.

Conclusion

The pattern fits into RAG pipelines built on Amazon Bedrock: A single Lambda function between retrieval and the final model call compresses context and fine-tunes cost and quality for your specific workload.

The compression prompt handles both simple lookups and complex multi-source questions without requiring separate routing logic. Combined with existing features like prompt caching, Intelligent Prompt Routing, and the Rerank API, this approach delivers compounding optimizations across the full RAG pipeline.

You can use this pattern behind a feature flag, measure it on your real query distribution, and tune the compression prompts to match your domain.

To get started, refer to the Amazon Bedrock documentation and the Amazon Bedrock pricing page for current model costs.

---

About the authors

Aakanksha Veesam

Aakanksha is a Delivery Consultant and AI/ML Engineer at Amazon Web Services (AWS) Professional Services, based in Dallas, Texas. She specializes in designing and building production machine learning and generative AI solutions for enterprise customers, helping organizations operationalize scalable, fit-for-purpose AI systems on AWS.

Amit Maindola

Amit is a Senior Data Architect with AWS ProServe team focused on data engineering, analytics, and AI/ML at Amazon Web Services. He helps customers in their digital transformation journey and enables them to build highly scalable, robust, and secure cloud-based analytical solutions on AWS to gain timely insights and make critical business decisions.