[于2026年7月14日提交（v1），最后修订于2026年7月15日（本版本，v2）]

标题：FinResearchBench II：基于共识金标准的金融报告质量区分深度研究基准

查看论文 PDF：《FinResearchBench II: 基于共识金标准的金融报告质量区分深度研究基准》，作者 Beidi Luan 等 7 人

查看 PDF HTML（实验性）

摘要：深度研究智能体越来越多地被用于生成长篇金融报告，然而大规模评估仍然受限于需要人类专家定义并执行高质量评估标准的瓶颈。我们通过提出一种可扩展的流程来解决这一问题，该流程无需人类专家参与最终环节即可生成高质量评估标准。我们基于104个真实用户查询构建了一个金融深度研究基准，并从模型生成的报告中自动合成了14,450条与查询相关的候选评估标准。为证明在评估标准执行阶段无需人类专家，我们将三位人类专家与一个由三个LLM组成的评判小组在抽样子集上的评估标准判断进行比较，结果表明基于LLM的评估与人类评估足够一致，可用于大规模评估标准筛选，包括在共同一致项上的标签级一致率达到98.67%。随后，我们通过两个过滤器推导出共识金标准：严格一致性过滤器，仅当三个LLM评判对同一查询下的每一份报告均一致同意时才保留该评估标准；可区分性过滤器，仅当该评估标准在受评系统之间至少给出一个多数“是”和一个多数“否”标签时才予以保留。此过程保留了3,687条通过一致性的评估标准，其中2,600条保持可区分性，构成最终的共识金标准集。使用这组最终评估标准，我们在10个深度研究系统上得到了明显区分的排名，条目级通过率从58.58%到22.23%不等。更广泛地看，由于该流程将人类专家执行从评估标准生成和评估中移除，它自然具有可扩展性，可用于基准评测、自动系统比较以及未来以评估驱动的系统改进研究。

提交历史

发件人：白左 查看电子邮件

[v1] 2026年7月14日（周二）01:31:20 UTC (6,335 KB)

[v2] 2026年7月15日（周三）01:47:03 UTC (6,335 KB)

[Submitted on 14 Jul 2026 (

v1

), last revised 15 Jul 2026 (this version, v2)]

Title:FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

View a PDF of the paper titled FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality, by Beidi Luan and 7 other authors

View PDF HTML (experimental)

Abstract:Deep research agents are increasingly used to produce long-form financial reports, yet large-scale evaluation remains bottlenecked by the need for human experts to define and execute high-quality rubrics. We address this problem by proposing a scalable pipeline for generating high-quality rubrics without human experts in the final loop. We build a financial deep research benchmark from 104 real-world user queries and automatically synthesize 14,450 query-specific candidate rubrics from model-generated reports. To justify removing human experts from rubric execution, we compare rubric judgments from three human experts with those from a three-LLM judge panel on a sampled subset, and show that LLM-based evaluation is sufficiently consistent with human evaluation to replace it for large-scale rubric screening, including 98.67% label-level agreement on jointly unanimous items. We then derive consensus-derived gold rubrics through two filters: a strict consistency filter, which keeps a rubric only if the three LLM judges unanimously agree on every report under the same query, and a distinguishability filter, which keeps a rubric only if it assigns at least one majority-yes and at least one majority-no label across the evaluated systems. This process retains 3,687 consistency-passed rubrics, of which 2,600 remain distinguishable and form the final set of consensus-derived gold rubrics. Using this final rubric set, we obtain clearly differentiated rankings across 10 deep research systems, with item-level pass rates ranging from 58.58% to 22.23%. More broadly, because the pipeline removes human-expert execution from rubric generation and evaluation, it is naturally scalable for benchmark evaluation, automatic system comparison, and future studies of evaluation-driven system improvement.

Submission history

From: Zuo Bai [

view email

]

[v1]

Tue, 14 Jul 2026 01:31:20 UTC (6,335 KB)

[v2]

Wed, 15 Jul 2026 01:47:03 UTC (6,335 KB)