将 RAG 上下文精简至答案真正所需之处
英文标题：🔶 Pruning RAG context down to what the answer actually needs
Kapa 在检索器与生成器之间插入一个廉价的小型 LLM 作为“剪枝器”，一次性读取问题与所有检索到的文本块，判断哪些对回答必不可少，丢弃约 68% 的上下文，仅损失约 4% 的召回，单次查询成本因此降低约三分之一。该方法解决了传统点式重排序无法识别互补相关性的痛点，并通过清单式评分让固定阈值可靠工作。剪枝功能已默认集成于其 Product Agent SDK，并可选用于检索 API 与 MCP 服务器，对构建基于大量文档的智能体的开发者而言，能有效压缩上下文、降低开销。
82.5 KB
19.4 KB
19.4 KB
23.1 KB
23.1 KB
25.8 KB
25.8 KB
41.9 KB
中文
Kapa 构建的是能够在庞大的产品知识库上回答复杂问题的 AI 助手。这些知识库包括技术文档、API 参考、PDF、论坛、支持讨论串等。开发者调用我们的检索 API 就能为他们的 Agent 提供产品上下文，而同一检索层也支撑着我们的端到端助手。
尽管 2026 年围绕“Agent 是否仍需要 RAG”争论不休，但在我们的领域，当知识库变得庞大而复杂时，还没有任何方案能与之接近。我们的检索以多种形式出现，有的带 Agent 行为，有的是单次传递，但它们都共享同一种结构：一个检索器，负责找到与问题相关的文档片段，以及一个生成器——即写出答案的大语言模型。
本文的简短版本：我们在两者之间加入了第三步。一个小巧、便宜的 LLM 会同时阅读问题与所有检索到的片段，并在昂贵的模型看到之前，把答案并不需要的片段丢弃。这能丢弃约 68% 的上下文，保留约 96% 的召回，并将每次查询的成本削减约三分之一（已扣除其自身开销）。本文解释了我们是如何走到这一步的。
被忽略的片段依然要花钱
检索器就像一个漏斗。嵌入与关键词检索将数十万条片段的知识库缩减为数百个候选对象，重排序器对其进行排序，前 15 条左右进入生成器——链中最大、最昂贵的模型。即使如此，生成器读取的大部分内容对问题而言也并非必需。这是有意为之：检索器追求最大召回，并信任生成器会忽略噪声。
但生成器会为它忽略的每一个片段付费。在我们的助手中，检索到的片段约占一次查询成本的三分之二，比答案、对话历史与系统提示加起来还多。每少一个片段，查询成本就能下降约 4%。而在 Agent 中，每次工具调用都会将其输出倒入同一上下文，因此上下文增长很快；更精炼的检索结果为 Agent 必须持有的其他所有信息腾出空间，也减少了上下文的腐烂。
关键在于召回。一旦丢弃了答案需要的某个片段，就等于用错误答案换回几分钱。剪枝器的好坏完全取决于这一权衡：每损失一个百分点的召回，能换来多少压缩。
显而易见的办法行不通
我们已经在返回前 K 个片段前进行了重排序，因此有时会被要求直接暴露重排序分数，让调用方自行按分数截断：把分数高于 0.7 的都留下，其余丢掉。这个方法有两个失败原因，而第二个塑造了我们之后构建的一切。
首先，重排序分数是一种排序，而非度量。它表示在当前查询下，片段 A 优于片段 B，仅此而已。这些分数无法在不同查询之间校准，Cohere 也这么说，所以没有固定的截断阈值能起作用。排序支持的唯一截断方式是位置式的 top-N，而这种做法会丢弃最后一个片段，无论它是噪声还是答案。
其次，即使校准完美地解决了第一个问题，这一点依然存在：相关性并非单个片段的属性。大多数流水线中的重排序器是点式交叉编码器。它们单独对每个“查询-片段”对进行评分，从不考虑该片段与其他被检索出的片段一同出现的情况。这里有一个匿名的生产环境示例：
第二个片段完全没有提及审计日志，因此被评定为噪声，然而它却是答案的一半，没有任何点式评分能看出这一点，因为只有与第一个片段放在一起，该片段才具备相关性。不同片段还会分割多步问题，单个片段都毫无用处。真正的问题从来不是“一个片段自身是否相关”，而是“它是否属于一个能够共同回答问题的集合”。
一个巧妙的方案，却因相同原因失败
在放弃重排序器之前，我们尝试了锚文档（Sinhababu 等）：通过将已知相关性的合成片段植入排序中，使重排序器的标度变得绝对化。每个相关层级（从“必不可少”到“无关”）各写入一条，然后将排在这些锚点之下的所有真实片段丢弃。这是在已经运行的重排序之上额外增加一次 LLM 调用，而且设计确实精巧。
它没能奏效，原因与前面相同。锚点解决了校准问题，但无法修正评分本身，重排序器仍会把部分相关和间接相关的片段排在明显无关的片段之下。为了保留它们，锚点必须放在极低的位置，导致几乎没有任何内容被剪除。
这一失败带来了有用的结论：任何剪枝操作都必须同时看见问题与所有片段，因为被评判的对象是“集合”。
于是我们让 LLM 给片段打分
我们交付的方案是在重排序器与生成器之间加入一次清单式 LLM 调用。它将问题与所有片段一并输入，并根据写入提示词的五级量表对每个片段打分：
| LLM 分数 | 级别 | 含义 |
|---|---|---|
| 5 | 必不可少 | 没有该片段就无法写出答案，无论它直接回答，还是作为其他片段所依赖的定义或前提。 |
| 4 | 起贡献作用 | 自身无法单独回答，但与其他片段结合后能提供完整答案所需的信息。 |
| 3 | 起支持作用 | 切题且可能有用，但没有它答案很可能依然完整。 |
| 2 | 边缘相关 | 同一领域或共享术语，但没有具体贡献。 |
| 1 | 无关 | 不存在有意义的联系。 |
达到或高于某一阈值的片段得以保留。这一设计回应了之前的两个失败。由于每一级别都由文字定义，“4”在每个查询中含义一致，固定的截断阈值终于能够生效。同时，由于模型同时看到问题与所有片段，它可以评判集合，部分相关和间接相关也终于有了容身之处。
有三个旋钮至关重要：
- 模型： 剪枝器的成本要从其节省的开销中支出，因此按设计旗舰模型就已被排除；各小型快速模型的表现近似，因此我们选择了推理行为最少、最快且最便宜的那个。
- 阈值： 平衡压缩与召回的主要控制杆。
- keep-top-k： 重排序后的前几个片段无论得分如何都直接通过，以保护最强片段免受评分失误的影响。
我们还运行了两种更简单的设计来让自己保持诚实。预算选择：保留前几个片段，并让 LLM 最多再添加 N 个；这使上下文规模可预测，但预算一旦用尽，后续无论多相关都会被丢弃。还有最简单的一种剪枝器：直接问 LLM 该保留哪些片段，不设等级。如果一个方案连直接提问都打不过，那它就不值得构建。
结果
我们在一个已标注的真实问题集上测量召回，其中我们确切知道每个答案需要哪些片段，然后通过回放一个月内随机的生产对话中每条查询实际发送给生成器的确切片段，验证了每种配置的压缩率、成本与延迟。
每个点代表一种配置，用两个关键指标绘制。x 轴为压缩率，即剪枝器丢弃的检索片段比例。y 轴为保留的召回，即剪枝后答案所需的所有片段依然完整的提问比例：100% 表示没有提问丢失所需片段，90% 表示十分之一丢失了。右上角为更优。各线条连接了每种策略的最佳配置，灰色虚线是任何剪枝器都需超越的基线：单纯的 top-N 截断，即从重排序器只返回更少的片段。
所有策略都远超基线。把召回保持在 98%：截断能丢弃一个片段，约 7% 压缩。而每种 LLM 策略都能达到 30% 或更高，相关性评分更可丢弃近半片段。在每一种压缩水平下，评分线条都压过其他两种策略，因此唯一需要决定的，就是在它的曲线上选择哪个点。
我们选择了靠近积极一端的点：约 96% 的召回被保留，约 68% 的片段被丢弃。这意味着每 25 个提问中有一个会丢失所需片段；作为交换，三分之二的上下文消失，去除剪枝器自身开销后，每次查询的费用下降约 34%。
它在延迟上要付出什么
剪枝器运行在检索与生成之间，处于关键路径上，因此每次查询都会增加一次模型调用，其速度决定了要付出多大代价。在整个生产数据集中，我们选定的配置每次查询运行约 0.7 秒。更重的配置会快速攀升，因此采用推理行为少的小型模型，才能把增量控制在一秒以内。
生成阶段的提速几乎可以忽略：更少的片段意味着生成器的输入 token 变少，因此它开始响应的时间稍早一点，但只有零点几秒，远不足以抵消剪枝器自身的调用开销。
因此，剪枝是以少量、固定的延迟换取压缩，在我们上线的配置下远不到一秒。在延迟敏感的单次路径上，这是需要权衡的真实成本。而在 Agent 内部，每回合本就包含多次模型调用，增加一次精简的调用影响不大。
我们在哪里启用了它
我们首先在“检索只是众多工具之一”的场景中推出：在基于我们的检索构建 Agent 的客户中。一个 Agent 携带数十个工具，每次调用都会将输出倒入同一上下文，此时文档搜索返回的上下文减少三分之二，就能为其他一切腾出空间。丢失的召回在这种场景下也并不那么危险：Agent 如果察觉到缺失，可以再次搜索。
剪枝功能默认启用在我们Product Agent SDK的知识库搜索中，并在检索 API 及 MCP 服务器中作为可选项提供。
English / 原文
Kapa builds AI assistants that answer complex questions over large product knowledge bases. Think technical documentation, API references, PDFs, forums, support threads. Developers use our retrieval API to give their agents context about their product, and the same retrieval layer powers our end-to-end assistants.
For all the debate in 2026 about whether agents still need RAG, in our domain nothing comes close when knowledge bases get large and complex. Our retrieval comes in several forms, some agentic, some single-pass, but they all share the same shape: a retriever, which finds the chunks of documentation relevant to a question, and a generator, the LLM that writes the answer from them.
The short version of this post: we added a third step between the two. A small, cheap LLM reads the question and all the retrieved chunks together, and throws out the chunks the answer will not need before the expensive model ever sees them. It drops about 68% of the context, keeps about 96% of recall, and cuts the cost of a query by a third, net of its own cost. This post explains how we got there.
Ignored chunks still cost money
A retriever is a funnel. Embedding and keyword search cut a knowledge base of hundreds of thousands of chunks down to a few hundred candidates, a reranker orders them, and the top 15 or so reach the generator, the largest and most expensive model in the chain. Even then, most of what the generator reads is not needed for the question. That is deliberate: retrievers aim for maximum recall and trust the generator to ignore the noise.
But the generator is billed for every chunk it ignores. In our assistants, retrieved chunks are about two-thirds of the cost of a query, more than the answer, the conversation history, and the system prompt combined. Every chunk fewer cuts the query cost by about 4%. And in an agent, every tool call pours its output into the same context, so the context grows quickly; a tighter retrieval result buys room for everything else the agent has to hold and leaves less context to rot.
The catch is recall. Drop a chunk the answer needed and you traded a few cents for a wrong answer. A pruner is exactly as good as that tradeoff: compression gained per point of recall lost.
The obvious fix does not work
We already rerank before returning the top K, so we are sometimes asked to just expose the rerank scores and let callers cut on them: keep everything above 0.7, drop the rest. It fails for two reasons, and the second shaped everything we built.
First, a rerank score is an ordering, not a measurement. It says chunk A beats chunk B on this query, nothing more. The scores are not calibrated across queries, Cohere too says as much, so no fixed cutoff works. The only cutoff a ranking supports is positional, top-N, and that drops the last chunk whether it is noise or the answer.
Second, and this survives even perfect calibration: relevance is not a property of a single chunk. The rerankers in most pipelines are pointwise cross-encoders. They score each query-chunk pair alone, never alongside the other chunks it was retrieved with. Here is an anonymized production example:
The second chunk never mentions audit logs, so it scores as noise, yet it is half the answer, and no pointwise score can see that, because the chunk is only relevant next to the first one. Chunks also split multi-part questions between them, each useless alone. The real question is never whether a chunk is relevant by itself, but whether it belongs to a set that together answers the question.
A clever fix that fails the same way
Before giving up on the reranker we tried anchor documents (Sinhababu et al.): make the reranker's scale absolute by planting synthetic chunks of known relevance into the ranking, one written per level from Essential to Unrelated, then drop every real chunk that ranks below the anchor of the lowest level you want to keep. One extra LLM call on top of a rerank you already run, and genuinely elegant.
It did not work, for the same underlying reason. Anchors fix calibration, but they cannot fix the scores, and the reranker kept placing partially and indirectly relevant chunks below plainly irrelevant ones. To keep them, the anchor has to sit so low that hardly anything gets pruned.
That failure was the useful result: whatever prunes has to see the question and all the chunks at once, because the thing being judged is the set.
So we let an LLM grade the chunks
What we shipped is one listwise LLM call between the reranker and the generator. It gets the question and all the chunks, and grades every chunk against a five-level scale written into its prompt:
| LLM Score | Level | Meaning |
|---|---|---|
| 5 | ESSENTIAL | The answer cannot be produced without this chunk, whether it answers directly or is a definition or prerequisite another chunk depends on. |
| 4 | CONTRIBUTING | Does not answer on its own, but supplies something a complete answer needs in combination with other chunks. |
| 3 | SUPPORTING | On topic and plausibly useful, but the answer is likely complete without it. |
| 2 | TANGENTIAL | Same domain or shared terminology, no concrete contribution. |
| 1 | UNRELATED | No meaningful connection. |
Chunks at or above a threshold survive. The design answers both failures from earlier. Because each level is defined in words, a 4 means the same thing on every query, so a fixed cutoff finally works. And because the model sees the question and all the chunks together, it can judge the set, so partial and indirect relevance finally have somewhere to land.
Three knobs matter:
-
The model: the pruner is paid for out of what it saves, so flagship models are ruled out by construction; the small fast tiers all judged similarly, so we picked the fastest and cheapest at low reasoning effort.
-
The threshold : the main dial between compression and recall.
-
keep-top-k : the top few reranked chunks pass regardless of grade, protecting the strongest chunks from a grading mistake.
We also ran two simpler designs to keep ourselves honest. Budget-select: keep the top few, let the LLM add at most N more; predictable size, but once the budget is spent every further chunk is dropped no matter how relevant. And the simplest possible pruner: just ask the LLM which chunks to keep, no scale. If a scheme cannot beat asking directly, it is not worth building.
The results
We measured recall on a labeled set of real questions where we know exactly which chunks the answer needs, then verified compression, cost, and latency by replaying every configuration over a random month of production conversations, on the exact chunks each query actually sent to the generator.
Every point is one configuration, plotted by the two things that matter. Compression, on the x-axis, is the share of retrieved chunks the pruner throws away. Recall preserved, on the y-axis, is the share of questions that still have every chunk their answer needs after pruning: at 100% no question lost a chunk it needed, at 90% one in ten did. Up and to the right is better. The lines connect each strategy's best configurations, and the dashed grey line is the baseline any pruner has to beat: naive top-N truncation, just returning fewer chunks from the reranker.
Everything beats it, by a lot. Hold recall at 98%: truncation can drop one chunk, about 7% compression. Every LLM strategy reaches 30% or more, and relevance scoring drops close to half the chunks. The scoring line also dominates the other two at every compression level, so the only decision left was where on it to sit.
We picked a point near the aggressive end: about 96% recall preserved, about 68% of chunks dropped. One question in twenty-five loses a chunk it needed; in exchange, two-thirds of the context is gone, and the per-query bill falls by about 34%, net of the pruner's own cost.
What it costs in latency
The pruner runs between retrieval and generation, in the critical path, so its model call is added to every query, and its speed decides what that costs. Across the production set, the configuration we picked ran in about 0.7 seconds per query. Heavier settings climb fast, so a small model at low reasoning effort is what keeps the addition under a second.
The generation barely speeds up in return: fewer chunks mean fewer input tokens for the generator, so it starts responding a little sooner, but only a fraction of a second, nowhere near enough to cancel out the pruner's own call.
So pruning buys its compression at the cost of a small, fixed amount of latency, well under a second on the configuration we ship. On a latency-sensitive single-shot path that is a real cost to weigh. Inside an agent, which already makes several model calls per turn, one more lean call is marginal.
Where we turned it on
We rolled it out first where retrieval is one tool among many: customers building agents on top of our retrieval. An agent carries dozens of tools, every call pours output into the same context, and a documentation search that returns two-thirds less buys room for everything else. The lost recall is also less dangerous there: an agent that notices something missing can search again.
Pruning is on by default in our Product Agent SDK's knowledge base search, and optional in the retrieval API and MCP servers.