在杂乱的真实企业知识中评测 Agent 检索能力

各个领域的团队都在快速改变检索方式。Cursor 最近停止了通过 embedding 搜索代码，转而仅依赖 grep 和索引搜索。也有团队正将传统 reranker 换成 Jev 等新模型。

Agent 所依赖的最重要知识类型之一，是企业知识：文档、工单、聊天消息、内部 wiki 和代码。然而，大多数团队无法判断哪种检索方式在这些知识上效果最佳，因为他们没有办法衡量检索系统在真实数据上的表现。

Kapa 是一个用于索引企业知识、并让 Agent 搜索这些知识以获得上下文的平台。你连接数据源后，Kapa 会将它们整合为一个可搜索的知识库，任何 Agent 都可以查询其中所需的信息。检索是我们构建产品的核心，我们也一直在不断调整检索方法。

Company Knowledge Bench 是我们为改进自身系统而构建的基准：其中包含从真实生产数据标注而来的 1,000 个评测案例。本文将说明它如何运作，并用它为几种常见检索实现评分。

图表：Company Knowledge Bench 结果——检索分数与单次查询耗时。对 1,000 个评测案例测试 7 个检索器，包括固定检索、Agentic grep 和 Kapa；分数越高且位置越靠左越好。

检索器类别	检索器	检索分数
固定检索（传统 RAG）	Hybrid search	0.41
固定检索（传统 RAG）	Hybrid search + rerank	0.50
固定检索（传统 RAG）	Query decomposition	0.56
Kapa	Kapa Default	0.61
Kapa	Kapa Deep	0.65
Agentic retriever with grep	Agent + grep (Luna)	0.54
Agentic retriever with grep	Agent + grep (Sol)	0.61

图表横轴为单次查询耗时（秒），标注刻度为 0 秒、5 秒、10 秒和 15 秒；纵轴为检索分数，标注刻度为 0.4、0.5、0.6 和 0.7。

结论是：一个除 grep 外什么都不用的前沿模型，能以 0.61 的成绩匹敌经过调优的现代检索管线，但耗时是其五倍。经过优化的 Agentic retriever（Kapa Deep）表现更好：约 5 秒即可达到 0.65 分。

关于偏差的说明：检索是我们产品的核心，这 7 个检索器均由我们构建，并且使用通过我们的管线摄取的文档。

为什么公开基准不适合我们

公开基准不适合我们，因为没有任何一个覆盖了我们遇到的全部使用场景和查询类型。

使用场景

团队在 Kapa 中索引其数据源，并将 Agent 连接到检索系统；该 Agent 可以服务于截然不同的场景，每个场景都有各自的文档，以及提出问题的不同人群：

• 开发者通过产品文档、API 规范、代码和 GitHub issue 询问产品问题，通常会经由 Claude Code 提问。

• 销售及其他员工通过 Slack、Confluence、Notion 和 Google Drive 询问产品、流程和客户相关问题。

• 支持团队基于旧工单、帮助中心文章和内部手册，为新工单起草回复。

查询

正如这些使用场景所示，许多不同类型的 Agent 会向 Kapa 的检索系统发送查询；查询的形式取决于由谁写出：

• 人类会把所有内容塞进一条消息：多个问题、一段粘贴的错误信息，以及对先前说过内容的引用。

• Agent，例如 Claude Code，会自行撰写查询，将复杂问题拆分为简短、精确的搜索，例如 webhook retry backoff config。

公开检索基准大多过于狭窄：要么围绕法律或医疗等单一领域构建，要么基于合成文档和问题，显得过于人工化。没有任何一个覆盖这一范围，因此我们构建了自己的基准。

我们如何为 Agent 检索评分

在衡量优秀检索之前，我们必须先定义它是什么。

先介绍几个术语。**查询（query）**是发送给检索器的内容。**语料库（corpus）**是一个团队在 Kapa 中索引的全部内容。**分块（chunk）**是其中的一小段内容，例如页面中的一个章节。**检索器（retriever）**接收查询，并从语料库中返回与之最相关的分块。

我们的定义是：

检索器的目标是利用可用的最高质量来源，收集一组最小化的分块集合，以完整回答查询。

Company Knowledge Bench 建立在三个属性之上：

• 完整性。 接收这些分块的 Agent 无需其他信息，就能回答查询。如果这组分块不完整，Agent 就会给出不完整或不正确的回答。

• 最小性。 移除任何一个分块都会使这组内容变得不完整。查询不需要的额外分块会增加成本，也会让模型更难进行推理。

• 来源优先级。 语料库往往包含多组能够回答同一查询的分块，而它们的质量几乎从不相同。我们的基准只接受优先来源。判断哪个来源应被优先采用并不容易，其中有很多模糊地带；但总体上取决于两个因素：

  • 来源权威性。 专门的参考页面优先于教程、issue 讨论串，或重复相同事实的博客文章。

  • 时效性。 当前有效的来源优先于过时来源。上周的 Slack 讨论串优于三年前最后编辑的 Confluence 页面。

除这些原则外，该基准还为特定情形制定了规则。例如：

• 查询存在歧义。 检索器必须针对每一种合理理解返回相应分块。选择其中一种理解是 Agent 的职责。

• 一个问题存在多个有效解决方案。 检索器必须返回每一种已记录解决方案对应的分块，以便 Agent 解释不同选项及其权衡。

• 语料库中没有能回答查询的内容。 检索器必须返回最有力的现有证据：表明功能不受支持的声明、该功能未包含在内的完整列表，或已有文档记录的替代方案。如果这些都不存在，正确结果就是完全不返回任何内容。

所有这些规则都被精确记录在我们的基准标注手册中。

一个评测案例是什么样子

我们的基准由一组评测案例构成。每个案例包括一个真实生产环境中的查询（query）、该查询提出时的语料库（corpus）快照，以及一个检索标准（retrieval criterion）：用于指定从语料库中检索哪些分块对该查询有效的布尔表达式。

下面是一个简化的评测案例，并根据某个检索器返回的分块进行评分：

查询

"Which plans include SSO, and how do I turn it on?"

检索标准

{ "operator": "AND", "operands": ["chunk_1", "chunk_2"] }

检索器返回的分块

chunk_1 Pricing page "SSO is included in the Team and Enterprise plans."

chunk_2 SSO setup guide "To enable SSO, go to Settings > Security, choose your identity provider, and paste in its metadata URL."

chunk_3 Community forum post "You get SSO on the Team and Enterprise plans."

chunk_4 Changelog "Dark mode is now available in the dashboard."

得分

• 通过。 标准要求 chunk_1 和 chunk_2，两者均被返回。

• 精度：4 个中有 2 个，即 50%。 chunk_3 和 chunk_4 不在标准之中。

论坛帖子不在标准中，是因为来源优先级。它回答了与定价页相同的查询部分——哪些套餐包含 SSO——但定价页具有更高的来源权威性。更新日志则完全不相关。这两者都会降低精度，但原因不同：一个是不相关，另一个虽相关但并非优先来源。

总计，我们为 Company Knowledge Bench 创建了 1,000 个评测案例，本文后续结果均基于这些案例。

我们如何创建这个基准

手工标注一千个评测案例并不可行。对它们进行恰当采样更难：查询来自我们摄取的每一种来源，既包括内部语料库，也包括公开语料库，并且覆盖我们服务的多个行业。因此，要实现这一规模，标注必须由 Agent 完成。

当然，只有 Agent 能正确完成标注时，让它们创建基准才有意义。错误的标准意味着我们针对该标准测试的每一个检索器都会得到错误分数。因此，在 Agent 构建我们的基准之前，我们需要第二个基准来衡量它们构建基准的能力：一个用于评测基准创建的基准。

这个第二基准由 170 个评测案例组成，全部由人类依据同一份基准标注手册手工完成标注。每个案例都包含一个真实查询、一个语料库，以及由人工通读语料库后写出的检索标准；因此，它们共同编码了标注手册中的规则。

因此，流程是：构建能完成与人类相同任务的 Agent，在这一数据集的查询上运行它们，并将它们写出的标准与人工标准进行比较。我们持续改进 Agent，直到它们与人工标注者达到足够高的一致率。只有这样，我们才让它们根据生产查询生成基准中的 1,000 个评测案例。

标注工作被拆分给两类 Agent：

• **候选 Agent（candidate agents）**扫描完整语料库，寻找可能应纳入检索标准的分块。它们以召回率为目标，返回一大组可能相关的分块。

• **标准 Agent（criteria agents）**接收这些候选项，并遵循基准标注手册中的规则，将其转化为实际的检索标准。

要让基准给出良好信号，Agent 不必逐字复现人工标准，但必须足够接近。为达到这一目标，我们对 Agent 及手册本身都进行了多次迭代。在开始生成基准前，我们确保 Agent 与人工标注者达到了足够的一致性，并对其进行了广泛验证。

结果是，这些标签接近人类会写出的结果；与手工标签不同，它们的可扩展性远超本基准生成的 1,000 个评测案例。

各检索器的对比

我们在 Company Knowledge Bench 上运行了 7 个检索器。它们都搜索相同的语料库，并以相同方式完成摄取和分块，因此唯一的差异是检索策略。

固定管线

这些是过去几年中的传统 RAG 方法。它们对每个查询都运行相同步骤，没有模型决定下一步该做什么。

• 混合搜索（Hybrid search）。 大多数 RAG 管线的起点基线。查询同时与关键词索引和 embedding 索引进行匹配，embedding 使用 gemini-embedding-001，并返回排名前 15 的分块。

• 带 reranker 的混合搜索。 使用同样的搜索，但先检索排名前 100 的分块，再将其交给 Voyage AI 的 rerank-2。该 reranker 会一并读取查询和每个分块，并对分块回答查询的程度评分。最终返回 rerank 后排名前 15 的分块。

• 查询分解、混合搜索和 reranker。 搜索前，模型 gpt-6.1-luna 会将查询拆分为最多 3 个子查询。每个子查询通过混合搜索检索排名前 50 的分块，再将汇总结果一并 rerank，筛选至前 15 个。

Agentic retriever

这是检索正在转向的方向。模型驱动搜索，决定接下来要寻找什么，以及何时已经获得足够信息。过去，这种方式对每一次查询而言都过慢且过于昂贵；但模型正变得更强、更便宜、更快。

• Agentic grep，大模型。 gpt-6.1-sol 配备一个工具：针对分块进行全文和正则表达式搜索。完全不使用 embedding，就像编程 Agent 探索代码仓库的方式。

• Agentic grep，小模型。 使用更小、更快的 gpt-6.1-luna 采用相同设置，以观察模型能力的强弱会带来多大影响。

Kapa 的检索

• 默认模式（Default mode）。 Kapa 的更快模式：带有小型规划步骤和严格预算的固定管线，返回固定数量、排名最佳的分块。

• 深度模式（Deep mode）。 Kapa 的检索 Agent。它会搜索、继续阅读已找到文档中的后续内容，并剔除后来发现不相关的内容；最终按查询所需返回相应数量的分块。

结果

为比较这些检索器，我们从四个维度考察：检索分数、延迟、精度，以及它们返回的 token 数量。一个检索器即使得分很高，也仍然可能太慢，无法直接部署在用户面前。

How teams do retrieval is changing fast, in every domain. Cursor recently stopped searching your code via embeddings in favour of relying only on grep and indexed search. Others are swapping their traditional rerankers for new models like Jev.

One of the most important kinds of knowledge that agents rely on is company knowledge: documentation, tickets, chat messages, internal wikis, and code. Yet most teams cannot tell which retrieval works best on it, because they have no way to measure how their retrieval performs on real data.

Kapa is a platform for indexing your company knowledge and letting your agents search them for context. You connect your sources, Kapa turns them into one searchable knowledge base, and any agent can query it for the information it needs. Retrieval is the core of what we build, and we change how we do it constantly.

The Company Knowledge Bench is what we built for ourselves to improve our own system: 1,000 eval cases annotated from real production data. In this post we explain how it works, and score a few common retrieval implementations on it.

Company Knowledge Bench results: score against time per query

Fixed retrieval, agentic grep and Kapa: 7 retrievers on 1,000 eval cases. Higher and further left is better.

Fixed retrieval (traditional RAG)Agentic retriever with grepKapaBest score for the time

Retrieval score0.40.50.60.70 s5 s10 s15 sTime per query, secondsHybrid search 0.41Hybrid search + rerank 0.50Query decomposition 0.56Kapa Default 0.61Kapa Deep 0.65Agent + grep (Luna) 0.54Agent + grep (Sol) 0.61

The takeaway: a frontier model with nothing but grep matches a tuned modern retrieval pipeline at 0.61, but takes five times as long. An optimized agentic retriever (Kapa Deep) does better still: 0.65 in about five seconds.

A note on bias: retrieval is core to our product, and all seven retrievers were built by us and use documents ingested with our pipeline.

Why public benchmarks do not work for us

Public benchmarks do not work for us because none of them cover all the use cases and types of queries that we see.

Use cases

Teams index their sources in Kapa and connect an agent to the retrieval, and that agent can serve quite different use cases, each with its own documents and its own people asking questions:

• Developers ask about a product over its docs, API specs, code and GitHub issues, often through Claude Code.

• Sales and other employees ask about products, processes and customers over Slack, Confluence, Notion and Google Drive.

• Support teams draft replies to new tickets from old tickets, help center articles and internal handbooks.

Queries

As these use cases show, many different kinds of agents send queries to Kapa’s retrieval, and the queries look different depending on who wrote them:

• People put everything into one message: several questions, a pasted error, a reference to something said earlier.

• Agents like Claude Code write their own, breaking a complex question into short, precise searches, like webhook retry backoff config

Public retrieval benchmarks are mostly too narrow, built around one domain like law or medicine, or too artificial, built from synthetic documents and questions. None of them cover this range, so we built our own.

How we score retrieval for agents

Before we can measure good retrieval, we have to define what it is.

A few terms first. A query is what is sent to the retriever. The corpus is everything a team has indexed in Kapa. A chunk is a short piece of it, such as a section of a page. A retriever takes a query and returns the most relevant chunks from the corpus for it.

Our definition:

The retriever’s goal is to collect a minimal set of chunks that completely answers the query, using the highest-quality sources available.

The Company Knowledge Bench rests on three properties:

• Completeness. The agent you hand the chunks to needs nothing else to answer the query. If the set is incomplete, the agent gives an incomplete or incorrect answer.

• Minimality. Removing any chunk would make the set incomplete. Extra chunks the query does not need cost money and make it harder for the model to reason.

• Source preference. A corpus often holds several sets of chunks that could answer the same query, and they are almost never of equal quality. Our benchmark only accepts the preferred ones. Deciding which source is preferred is hard and has a lot of grey zones, but in general it comes down to two factors:

  • Source authority. A dedicated reference page outranks a tutorial, an issue thread, or a blog post that repeats the same fact.

  • Currency. A current source outranks an outdated one. A Slack thread from last week beats a Confluence page last edited three years ago.

Beyond these, the benchmark enforces rules for specific situations. For example:

• The query is ambiguous. The retriever has to return chunks for every reasonable reading of it. Picking one reading is the agent’s job.

• A problem has several valid solutions. The retriever has to return chunks for each documented solution, so the agent can explain the options and their trade-offs.

• The corpus holds nothing that answers the query. The retriever has to return the strongest evidence there is: a statement that the feature is not supported, a complete list the feature is missing from, or a documented workaround. If none of those exists, the right result is nothing at all.

All of it is written down precisely in our benchmark labelling handbook.

What an eval case looks like

Our benchmark is a set of eval cases. Each one consists of a real production query , a snapshot of the corpus as it was when the query was asked, and a retrieval criterion : a boolean expression that specifies which chunks from the corpus are valid to retrieve for the query.

Here is a simplified eval case, scored against the chunks one retriever returned:

Query

 "Which plans include SSO, and how do I turn it on?"


"Which plans include SSO, and how do I turn it on?"


"Which plans include SSO, and how do I turn it on?"


"Which plans include SSO, and how do I turn it on?"

Retrieval criterion

{
  "operator": "AND",
  "operands": ["chunk_1", "chunk_2"]
}


{
  "operator": "AND",
  "operands": ["chunk_1", "chunk_2"]
}


{
  "operator": "AND",
  "operands": ["chunk_1", "chunk_2"]
}


{
  "operator": "AND",
  "operands": ["chunk_1", "chunk_2"]
}

Chunks the retriever returned

chunk_1  Pricing page
         "SSO is included in the Team and Enterprise plans."

chunk_2  SSO setup guide
         "To enable SSO, go to Settings > Security, choose your identity
         provider, and paste in its metadata URL."

chunk_3  Community forum post
         "You get SSO on the Team and Enterprise plans."

chunk_4  Changelog
         "Dark mode is now available in the dashboard."


chunk_1  Pricing page
         "SSO is included in the Team and Enterprise plans."

chunk_2  SSO setup guide
         "To enable SSO, go to Settings > Security, choose your identity
         provider, and paste in its metadata URL."

chunk_3  Community forum post
         "You get SSO on the Team and Enterprise plans."

chunk_4  Changelog
         "Dark mode is now available in the dashboard."


chunk_1  Pricing page
         "SSO is included in the Team and Enterprise plans."

chunk_2  SSO setup guide
         "To enable SSO, go to Settings > Security, choose your identity
         provider, and paste in its metadata URL."

chunk_3  Community forum post
         "You get SSO on the Team and Enterprise plans."

chunk_4  Changelog
         "Dark mode is now available in the dashboard."


chunk_1  Pricing page
         "SSO is included in the Team and Enterprise plans."

chunk_2  SSO setup guide
         "To enable SSO, go to Settings > Security, choose your identity
         provider, and paste in its metadata URL."

chunk_3  Community forum post
         "You get SSO on the Team and Enterprise plans."

chunk_4  Changelog
         "Dark mode is now available in the dashboard."

Score

• Pass. The criterion requires chunk_1 and chunk_2, and both were returned.

• Precision: 2 of 4, or 50%. chunk_3 and chunk_4 are not in the criterion.

The forum post is not in the criterion because of source preference. It answers the same part of the query as the pricing page, which plans include SSO, and the pricing page has more source authority. The changelog is simply not relevant. Both count against precision, but for different reasons: one is irrelevant, the other is relevant but not preferred.

In total, we created 1,000 eval cases for the Company Knowledge Bench, and they are what the results later in this post are based on.

How we created the benchmark

Labelling a thousand eval cases by hand is not feasible. Sampling them properly makes it even harder: the queries come from every type of source we ingest, internal as well as public corpora, and the industries we serve. So to achieve this scale, the labelling has to be done by agents.

Of course, letting agents create the benchmark only works if they do it correctly. A wrong criterion means a wrong score for every retriever we test against it. So before the agents can build our benchmark, we need a second benchmark that measures how well they build it: a benchmark for benchmark creation.

That second benchmark is a set of 170 eval cases that humans labelled completely by hand, following the same benchmark labelling handbook. Each has a real query, a corpus, and a retrieval criterion a person wrote after working through the corpus themselves, so together they encode the handbook’s rules.

So the process was: build agents that can do the same task the humans did for this set, run them on its queries, and compare the criteria they write with the human ones. We kept improving the agents until they reached a high enough agreement rate with the human labellers. Only then did we let them generate the 1,000 eval cases of the benchmark from production queries.

The labelling is split between two kinds of agents:

• Candidate agents scan the full corpus for chunks that might belong in the retrieval criterion. They are built for recall and return a large set of possibly relevant chunks.

• Criteria agents take those candidates and turn them into the actual retrieval criterion, following the rules in the benchmark labelling handbook.

The agents do not have to reproduce the human criteria exactly for the benchmark to give a good signal, but they do have to come close. Getting there took many iterations, on the agents and on the handbook itself. We made sure the agents reached sufficient agreement with the human labellers, and validated them extensively, before moving on to generate the benchmark.

The result is labels that are close to what a human would write, and unlike hand labels they scale well beyond the 1,000 eval cases we generated for this benchmark.

How the retrievers compare

We ran seven retrievers on the Company Knowledge Bench. All of them search the same corpora, ingested and chunked the same way, so the only thing that differs is the retrieval strategy.

Fixed pipelines

These are the traditional RAG approaches of the last few years. They run the same steps for every query, with no model deciding what to do next.

• Hybrid search. The baseline most RAG pipelines start from. The query is matched against a keyword index and an embedding index at once, using gemini-embedding-001 for the embeddings, and the top 15 chunks are returned.

• Hybrid search with a reranker. The same search, but it retrieves the top 100 chunks and passes them through a reranker, Voyage AI’s rerank-2, which reads the query and each chunk together and scores how well the chunk answers it. The top 15 after reranking are returned.

• Query decomposition, hybrid search, and a reranker. Before searching, a model, gpt-6.1-luna, splits the query into up to three sub-queries. Each sub-query retrieves its top 50 chunks through hybrid search, and the pooled results are reranked together down to the top 15.

Agentic retrievers

This is where retrieval is moving. A model drives the search, deciding what to look for next and when it has enough. That used to be too slow and expensive for every query, but models are getting better, cheaper, and faster.

• Agentic grep, large model. gpt-6.1-sol with a single tool: full-text and regex search over the chunks. No embeddings at all, the way a coding agent explores a repository.

• Agentic grep, small model. The same setup with gpt-6.1-luna, a smaller and faster model, to see how much the strength of the model matters.

Kapa’s retrieval

• Default mode. Kapa’s faster mode: a fixed pipeline with a small planning step and a strict budget, returning a fixed number of the best-ranked chunks.

• Deep mode. Kapa’s retrieval agent. It searches, reads on in documents it has found, and prunes what turns out not to be relevant, returning as many chunks as the query needs.

Results

To compare the retrievers, we look at them across four dimensions: retrieval score, latency, precision, and how many tokens they return. A retriever that scores well can still be too slow to put in front of a user, or return so much that the agent reading it pays for it on every call.

Company Knowledge Bench results by retriever

Score, time, precision and tokens returned per query. 7 retrievers on the same ingested data, 1,000 eval cases.

Scorehigher is betterTime per querylower is betterPrecisionshare of chunks neededTokens returnedfewer is betterKapa Deep0.655.1 s26%5.1kKapa Default0.613.3 s12%9.9kAgent + grep (Sol)0.6117 s4%40.8kQuery decomposition0.561.8 s10%10.2kAgent + grep (Luna)0.5413 s4%38.8kHybrid search + rerank0.500.7 s9%10.6kHybrid search0.410.4 s8%11.1k

Fixed pipelines

Reranking is the cheapest big win. Plain hybrid search scores 0.41. Retrieving 100 candidates and reranking them down to 15 lifts that to 0.50, for a third of a second more, no more tokens, and a reranker call that is cheap compared with a language model. If you change one thing in a basic RAG pipeline, add a reranker.

Searching with several queries helps, at a price. Splitting the query into sub-queries first lifts the score to 0.56. But it comes with a latency penalty, more than doubling the time to 1.8 seconds, and it adds the cost of a model call to every query.

An optimized fixed pipeline gets surprisingly far. You can view Kapa Default as a fully optimized fixed pipeline. It scores 0.61 in 3.3 seconds, the same score as the strongest grep agent in a fifth of the time, with the best precision of any fixed pipeline.

Agentic retrievers

A strong model with simple tools goes a long way. Throwing a reasoning model at retrieval works well, even when its only tool is grep. gpt-6.1-sol reaches 0.61, the same score as Kapa Default, and the smaller gpt-6.1-luna reaches 0.54, close to query decomposition.

For agentic retrievers, intelligence matters. Swapping gpt-6.1-sol for the smaller gpt-6.1-luna costs 0.07 in score. Query decomposition, a fixed pipeline, beats the small-model agent on score in a seventh of the time. An agentic loop is not better by default.

What the answering model pays to read what each retriever returns

Input cost per 1,000 queries, with GPT-6.1 Sol as the answering model at $2.00 per million tokens

Kapa Deep$10.16Kapa Default$19.85Query decomposition$20.46Hybrid search + rerank$21.10Hybrid search$22.27Agent + grep (Luna)$77.58Agent + grep (Sol)$81.67

The biggest caveat with agentic retrievers is cost. It shows up in latency, with the grep agents taking 13 to 17 seconds per query, and in money, in two places. The first is the retriever itself: the gpt-6.1-sol grep agent spends about $0.17 in model calls per query, which at production volumes can be prohibitively expensive. The second is what it returns. Every retrieved chunk becomes input tokens for the agent you hand the result to, and the grep agents return about 40,000 tokens per query, roughly four times as much as any fixed pipeline, with only one chunk in twenty-five actually needed. The chart above prices that second cost. It assumes the agent receiving the retrieval result also runs on gpt-6.1-sol, at $2.00 per million input tokens with tokens counted by the o200k_base tokenizer, and shows what you pay in inference per 1,000 queries just to read the retrieved chunks once.

An optimized agentic retriever gets the best of both. You can think of Kapa Deep as an optimized agentic retriever, built to return only what the query needs. It has the highest score, at 0.65 in about five seconds, and the highest precision, at 0.26 against 0.12 or less for every other retriever. It returns about 5,000 tokens per query, half of any fixed pipeline and an eighth of the grep agents, which also makes it the cheapest for the agent to read, at about $10 per 1,000 queries.

What’s next

We’ve built retrieval benchmarks before, but this is the first that looks like how agents actually use company knowledge, and it shows a lot of headroom left. As enterprises put more agents to work, latency and cost matter as much as accuracy, so we’ll keep growing the benchmark and climbing it on all three. We’ll publish more findings as we go, so follow the blog if you’re curious.

The benchmark is private since it’s built from real production data. If you have questions though, please reach out.