🔶 ATLAS：评估搜索密集型任务中的 Agent
英文标题：🔶 ATLAS: Evaluating Agents on Search-Intensive Tasks
一系列消融实验与对比实验表明，ATLAS 旨在修正当前主流 Agent 搜索评测中的多项缺陷，更真实地衡量搜索能力，并按搜索质量和投入区分不同系统。固定 Agent harness 后，不同搜索 API 会显著影响成绩；当屏蔽搜索结果前列内容时，ATLAS 分数约下降一半，而 BrowseComp、WideSearch 和 DeepSearchQA 仍可保留原始分数的 81%—89%。ATLAS 的数据记忆污染较低，新增搜索能力相对于无搜索基线能带来更明显的性能提升。该基准要求同时进行广泛的实体发现和深入的属性补全，开发者需要具备多跳、多域名检索与信息整合能力，而非仅依赖单一网页或模型记忆。
4.2 KB
141.3 KB
80.6 KB
124.4 KB
165.9 KB
173.6 KB
190.6 KB
186.5 KB
243.7 KB
中文
基于一套消融实验和对比实验，我们认为，ATLAS 修正了当前流行 Agent 搜索评测中存在的诸多缺陷，提供了一个能够真正衡量搜索能力、并按搜索质量与投入区分不同系统的基准。
ATLAS 能够强力区分不同搜索服务提供商：††每项指标均为 F1，即 精确率（返回单元中正确单元所占比例）与 召回率（被返回的标准答案单元所占比例）的调和平均数。它们仅在统计单元上有所不同：**Discovery F1：**实体。
Row F1（标题指标）：完整行，只有每个单元格都正确时才算正确。
**Item F1：**单个单元格，覆盖实体发现和信息补全。
Discovery F1
Row F1
Item F1
**图 3：**在 harness 固定时，仅搜索 API 就会显著影响性能。每个搜索 API 都在同一个中立的 Scout harness 中运行，模型使用 GPT-6 Luna 或 Claude Haiku 5.5。
#ATLAS 奖励搜索质量
ATLAS 对搜索质量的依赖远高于现有基准：当我们通过隐藏每 10 条搜索结果中的前 7 条，严重降低搜索质量时，同一个搜索系统的 ATLAS 分数会损失约一半。相比之下，在 BrowseComp、WideSearch 和 DeepSearchQA 上，该搜索系统仍能保留原始分数的 81%—89%，这表明这些基准实际上并未真正衡量搜索质量。
**图 4：**搜索质量退化对 ATLAS 的影响远大于其他基准。我们以合成方式降低搜索质量：对于每次 Exa 搜索，屏蔽 10 条结果中的前 k 条，并返回第 k+1 至第 10 条结果。结果基于 GPT-5.6 Luna 配合 Exa 搜索、运行在中立的 Scout harness 中得出。
#ATLAS 的记忆污染较低
我们通过检查 GPT-6 Astra、GPT-5.6 Sol 或 Claude Opus 5 中是否至少有一个模型能凭记忆召回任务的完整答案集，来衡量评测的记忆程度。BrowseComp、WideSearch 和 DeepSearchQA 中接近 50% 或更多的任务已被模型记忆（表 1）。
相比之下，ATLAS 较早的表格行比 WideSearch 的更少被记忆，并且包含超出当前前沿模型知识截止日期的新鲜信息；在这些信息上，前沿模型若不使用搜索，几乎无法正确回答。因此，与当前流行评测相比，为前沿模型加入搜索能力后，相对于无搜索基线可获得更大的性能提升（图 5）。
**图 5：**ATLAS 在很大程度上未被模型记忆。 (a) WideSearch 的记忆程度远高于 ATLAS，以至于搜索带来的性能提升微乎其微。 (b) 按每一行信息何时可被公众获知划分，三个闭卷模型（Claude Opus 5、GPT-5.6 Sol、GPT-6 Astra）中至少有一个能够找到的标准答案行比例。较早的 ATLAS 表格行通常属于小众信息，因此较少被记忆；而近期表格行几乎从未被召回。不同于按完整任务计数的表 1，(b) 按单独的表格行计数。
#ATLAS 要求广泛且深入的搜索
ATLAS 中有 74% 的任务要求找出 10 个或更多实体，且每个实体需要补全 2—10 项属性。此外，这些广泛且深入的表格任务确实需要多跳搜索，而非从单个 URL 或域名中直接检索即可完成。根据数据集构建日志，我们估计，发现实体平均每项任务需要访问 4 个不同域名；完成完整的实体发现与信息补全，平均每项任务需要访问 18 个不同域名。相比之下，许多流行的现有评测名义上要求广度或深度，但由于模型记忆、数据污染或基准设计缺陷，实际上只需极少的检索步骤即可解决。例如，我们在实验中发现，在 WideSearch 上，一个强 Agent 在 48% 的任务中仅引用一个网站；而在 ATLAS 上，这一比例仅为 0.2%。
ATLAS 的广度与深度都会增加难度。我们发现，同时提升二者会使所有测试中的搜索系统都更难完成任务。
**图 6：**更广泛、更深入的任务更难完成。 (a) 按任务中需要发现的标准答案行数量划分的 Row F1。 (b) 按需要补全的列数量划分，已找到的表格行中完全正确的比例。
English / 原文
Based on a suite of ablations and experiments, we believe that ATLAS rectifies the multitude of flaws present in current popular agentic search evals, providing a benchmark that truly measures search and distinguishes various systems by search quality and effort.
ATLAS strongly differentiates search providers:††Each metric is an F1: the harmonic mean of precision (share of returned units that are correct) and recall (share of gold units returned). They differ only in the unit counted: Discovery F1: entities.
Row F1 (headline): whole rows, correct only if every cell is.
Item F1: individual cells, covering discovery and enrichment.
Discovery F1Row F1Item F1
Figure 3: With the harness fixed, the search API alone strongly impacts performance. Every search API runs in the same neutral Scout harness, with GPT-6 Luna or Claude Haiku 5.5 as the model.
#ATLAS rewards search quality
ATLAS depends on search quality far more than existing benchmarks: when we severely degrade the quality of search by hiding the top 7 of every 10 search results, the same searcher loses roughly half of its ATLAS score. In contrast, on BrowseComp, WideSearch and DeepSearchQA, the searcher retains 81–89% of its original score, indicating that these benchmarks do not actually measure search quality.
Figure 4: Degraded search hurts ATLAS far more than other benchmarks. We synthetically degrade search quality: on every Exa search the top k of 10 results are censored and results k+1 to 10 returned. Results are shown for GPT-5.6 Luna with Exa search in the neutral Scout harness.
#ATLAS is unmemorized
We measure memorization of evals by checking the percent of tasks whose full answer set is recalled by at least one of GPT-6 Astra, GPT-5.6 Sol, or Claude Opus 5. Nearly 50% or more of BrowseComp, WideSearch, and DeepSearchQA are memorized (Table 1).
In contrast, ATLAS’s older rows are less memorized than WideSearch’s, and it includes fresh information past current frontier model knowledge cutoffs, where frontier models almost never answer correctly without search. As a result, adding search to frontier models gives a much bigger lift in performance over a no-search baseline than current popular evals (Figure 5).
Figure 5: ATLAS is largely unmemorized. (a) WideSearch is far more memorized than ATLAS such that there is barely any lift from search. (b) Share of gold rows that at least one of three closed-book models (Claude Opus 5, GPT-5.6 Sol, GPT-6 Astra) finds, by when the row became publicly knowable. Older ATLAS rows are often niche, and therefore less memorized, and recent ones are almost never recalled. Unlike Table 1, which counts whole tasks, (b) counts individual rows.
#ATLAS requires wide and deep search
74% of ATLAS’s tasks ask for 10 or more entities, and each entity needs 2–10 attributes. Moreover, these wide and deep tables actually require multi-hop searches as opposed to retrieval from a single URL or domain. We estimate, based on our dataset construction logs, that discovery of entities requires a median of 4 different domains per task and full discovery + enrichment requires a median of 18 different domains per task. In contrast, many popular existing evals nominally require width or depth but can actually be solved with very few retrieval steps due to memorization, contamination, or flawed benchmark design. For example, in our experiments we find that on WideSearch, a strong agent cites a single website for 48% of its tasks, against 0.2% on ATLAS.
Both the width and depth of ATLAS make it difficult. We find that increasing both makes tasks harder for all searchers tested.
Figure 6: Wider and deeper tasks are harder. (a) Row F1 by the number of gold rows to discover in the task. (b) Share of found rows that are fully correct, by the number of columns to enrich.