AI News
商业产业 Hacker News AI 2026-10-28T15:46 正文 ok 1437 words

Surge AI 推出 DAYJOB: Tax 基准：在 50 项专业税务任务中评测 AI Agent，顶尖模型通过率仅 10.8%

英文标题：🔶 Dayjob: Tax – evaluating AI agents on 50 professional tax assignments

Surge AI 发布了全新专业评测基准 DAYJOB: Tax，涵盖个人、企业合伙、跨境税务等 8 个领域的 50 项真实复杂任务，重点评估 AI Agent 在模糊需求与杂乱材料下的端到端专业判断力。在 95% 标准的高门槛评测中，Claude Sonnet 5.5 以 10.8% 的通过率位居榜首，Claude Fable 5.1（9.6%）和 Claude Opus 5.5（9.2%）紧随其后。该基准包含近 1800 份包含手写收据、调度日志等非标准化凭证的真实工作区文件，测试结果表明，AI 虽能在单点计算与合规问答上达标，但在识别客户遗漏的关键隐患及整合复杂商业决策方面仍与人类专家存在巨大差距。
打开原文相关 3 · 机会 1
media3.6 KB media3.6 KB media1.1 MB media302.4 KB media218.8 KB media940.5 KB media300.2 KB media363.4 KB media1.8 MB media19.8 KB media38.8 KB media360.6 KB media212.5 KB media1.8 MB media149.8 KB media253.9 KB media259.9 KB

中文

安娜（Anna）正打算出售她的公寓。她在 2024 年 8 月搬出，并想知道是在 2026 年 10 月还是 2027 年 1 月完成交易对税务更有利。

一个 AI Agent 审查了这两个选项，并正确得出结论：这两个日期都能满足自住要求，从而享受最高 25 万美元资本利得免税额的税收优惠。

但它忽略了一个关键的截止日期。

根据常规规定，房主在出售房产前的五年内必须在该房产中至少居住满两年，才有资格享受该免税额。由于安娜是在 2024 年 8 月搬离的，她在 2027 年 8 月底左右将不再符合这一资格条件。如果她的交易延迟到 2027 年 8 月之后，她就可能失去该资格。

在五次尝试中，Astra 从未识别出这一截止日期。而 Opus 在全部五次尝试中均识别出了该问题。

该 Agent 对提议日期的评估是正确的，但其建议让安娜未能意识到一个可能影响重大财务决策的截止日期。一名优秀的税务顾问应当在客户根本不知道该提问时，仍能敏锐识别出可能带来重大后果的问题。

这种专业判断力正是我们通过 DAYJOB: Tax 所要测试的，它是 Surge AI 的 DAYJOB 系列基准测试 的最新成员。

DAYJOB: Tax 介绍

DAYJOB 旨在评估 AI Agent 处理专业税务任务的能力，这些任务需要专门知识、复杂材料，并涉及在初始指示中未完全明确的决策。我们早期的版本覆盖了医疗保健（Healthcare）和金融（Finance）领域。DAYJOB: Tax 将这一工作扩展到了报税准备、合规审查以及税务咨询服务中。

该基准包含跨越八个税务实践领域的 50 个任务环境。Agent 会收到客户的需求诉求和支持凭证，随后需要生成纳税申报单、工作底稿、技术审查报告或客户咨询建议。

这些任务在考察税务专业知识、计算、信息检索与指令遵循的同时，更全面测试了 Agent 的专业判断力。Agent 必须自主判断哪些问题值得深入调查、如何化解相互冲突的证据、何时信息尚不充分，并将这些裁量贯彻落实到最终交付成果中。

评测结果：排行榜与帕累托前沿（Pareto Frontiers）

在 DAYJOB: Tax 排行榜 上，Claude Sonnet 5.5 以 10.8% 的通过率位居榜首，紧随其后的是 Claude Fable 5.1（9.6%）和 Claude Opus 5.5（9.2%）。

我们以 95% 的评分标准阈值报告 pass@1。一次运行只有在至少满足该任务 95% 的评分准则时才算通过；这些准则综合评估了计算准确性、支持性证据、重大遗漏问题处理以及最终交付成果的完成度。

性能与成本对比

我们还对比了各模型的任务表现与单次尝试的平均成本。

Claude Sonnet 5.5 与 Opus 5.5 占据了当前测量帕累托前沿（Pareto frontier）的高性能区间。

专业税务工作的广度

DAYJOB: Tax 涵盖八大业务领域，每个领域都需要不同的法规知识、文件记录与专业经验：

  • 个人与家庭税务（Individual and family tax）
  • 小型企业税务（Small-business tax）
  • 合伙企业税务（Partnerships）
  • 财产与投资税务（Property and investments）
  • 遗产、赠与及信托（Estates, gifts, and trusts）
  • 国际税务（International tax）
  • 企业交易与重组（Corporate transactions）
  • 税务审查与争议（Tax review and disputes）

其中一些任务要求提交包含完备支持底稿的整套申报材料；另一些任务则要求 Agent 调查历史交易、审查过往申报，或针对客户尚未敲定的商业决策提供专业筹划建议。

任务工作区中包含什么？

该基准共包含 1,789 份工作区文件，每个任务中位数为 30 份，最多可达 123 份。其中包括 1,487 份 PDF、117 份 Excel 工作簿、38 份 Word 文档，以及额外的电子邮件、图像、CSV、JSON 记录和文本文件。

工作区资料涵盖已归档及草稿状态的纳税申报单、K-1 表、W-2 表、美国国税局（IRS）通知、总账流水、工资单记录、合同以及往来邮件。此外，还包含了税务专业人士日常打交道的非标准化凭证。

例如，一个任务包含一张手写采购收据的照片（购买用于转售的商品）。Agent 必须读取商品描述和金额，将其与支出流水进行对账，并判断这些采购对存货与主营业务成本（COGS）的具体影响。

另一个任务涉及一家工业筒仓维护服务公司，其客户误以为该公司仅在阿拉巴马州开展业务。然而其现场调度日志显示施工团队横跨了五个州开展作业。作业地点、团队规模和驻场天数构成了判断该公司多州纳税义务（multistate tax obligations）的关键证据。

其他环境同样复杂多样。例如一个跨国家庭的案例，必须将波兰的雇佣与财产记录，与加密货币交易所导出的流水、钱包历史记录以及历史采购批次进行全面对账核算。

由税务专家创建与审查

DAYJOB 任务源自真实世界的专业工作流。税务专家将这些真实情境转化为高度逼真的任务环境，构建相应的支持材料、预期产出及评分准则。每个任务都要经过三层专家审查，确保其准确性、完整性与可解答性。

参与构建的税务专家在企业税、合伙企业、个人申报、信托及多州合规方面拥有深厚背景，其履历包括：

  • 企业与私募股权税务： 一位拥有 10 余年从业经验的税务律师，曾在普华永道（PwC）、德勤（Deloitte）及埃克森美孚（Exxon）任职，专长涵盖税务规划、合规、审计及税务争议解决。
  • 合伙企业与企业税务： 一位 Baker Tilly 税务总监，曾任职于普华永道、Moss Adams 和 Windes，专注于复杂合伙企业利润分配、合并纳税申报、房地产投资信托（REITs）及高净值客户咨询。
  • 房地产与个人税务： 一位曾就职于德勤和安永（EY）的税务专业人士，专注于房地产合伙企业、个人所得税及信托规划。

建立在未核实假设上的退休规划

一对正办理分居的夫妻询问 AI Agent 应如何分割他们的 Roth IRA 和 401(k) 退休账户。Opus 5.5 告知他们并未建立合法婚姻关系，并建议各自保留独立账户，通过现金补偿来平衡差额。

其在一次回复的开头写道：

你们没有合法结婚，因此在不触发税款和罚金的情况下，这两个账户都无法直接分割。

但夫妻俩提供的原始材料对这一结论提出了质疑。虽然他们目前居住在通常不承认普通法婚姻（common-law marriage）的佛罗里达州，但他们此前曾共同居住在德克萨斯州，并在科罗拉多州举行过仪式。德州在特定条件下承认非正式婚姻，而在其他州合法确立的婚姻在佛州通常仍被视为有效。

这一事实上的细微差异，将彻底改变化解和分割退休账户的税务后果。

在全部五次测试中，Opus 始终基于夫妻双方未婚的预设给出建议，即便它在行文中意识到了不确定性。相反，Astra 在五次测试中有四次敏锐意识到必须首先核实两人的婚姻状况。其在一次回复中建议：

我们首先需要确认你们的合法婚姻状况。

客户的问题是如何分割账户，但 Agent 在给出行动路径前，必须先确认两人是否构成合法夫妻。而本应促成这一调查的证据线索，早已静静躺在附件文件中。

DAYJOB: Tax 的定位与差异

现有大多数税务评测基准通常始于一个明确定义的问题：回答一个具体的法规问题、计算一份报税单、或收集缺失的纳税人信息。这些评测虽然衡量了关键能力，但税务专家的绝大部分工作本质上是在弄清楚“真正的问题到底是什么”。客户提供的指示往往是不完整的，原始文件中夹杂着冲突的证据，而关乎重大利益的关键细节可能从未出现在初始需求中。

DAYJOB 衡量的是 Agent 能否承担这种端到端的全流程责任——包括主动识别客户忽视的问题、评估冲突证据，并首先理清到底需要做什么。

TaxBench

模型能解答高难度的税务法规问题吗？

TaxBench 评估了涵盖多个专业领域的 114 个税务问题，每个问题都包含预设的问题描述和评分标准。

而 DAYJOB 还会考察客户的问题本身是否问对了方向。 安娜想比较两个卖房日期。两个日期都能满足免税的居住要求，但 Agent 是否会警告她该免税资格可能在何时失效？

TaxCalcBench

模型能准确计算纳税申报表吗？

TaxCalcBench 基于逼真的税务文档和纳税人场景，评估纳税申报表相对于预期申报数值的计算准确度。

而 DAYJOB 还会追问底层的原始单据是否真正支持当前的税务处理。 例如，某个工作室的账本提及附有一张收据，但该收据实际缺失。Agent 是会直接指出缺乏佐证凭据，还是错误地将这笔报销款直接视作已核实？

TaxLab

Agent 能否主动收集缺失信息并正确申报？

TaxLab 测试 Agent 能否通过向模拟纳税人提问来获取遗漏信息，并完成联邦个人所得税申报表。

DAYJOB 则将这种自主权延伸到了更广泛的专业税务场景中。 最终的交付成果可能是一份合伙企业亏损分析报告、一份遗产税申报单、一份针对争议性税务立场的审查意见书，或是一封深度策略咨询函。

最核心的区别在于：Agent 到底承担了多大比例的端到端责任。 DAYJOB 评估 Agent 能否主动挖掘重大衍生问题、展现专业判断力，并将发现贯彻至最终的申报表、审查报告或专业筹划建议中。

DAYJOB: Tax 衡量的核心能力

DAYJOB 测试的是 AI Agent 能否从头至尾全盘接手一项专业委托。这意味着需要穿透杂乱无章的原始记录、发现客户从未主动提及的问题、在信息不完备时做出审慎合理的判断，并最终交付符合专业水准的工作成果。

评测结果表明，这对于当前的前沿模型依然是一道极难逾越的鸿沟。哪怕 Agent 能够完成某些单一环节，但在真实税务业务流中稳定运用专业判断并交付结果，目前依然难以企及。

AI Agent 们，目前尚未通过这场专业审计。

完整的评测排行榜及所有参测模型详情，请参阅 DAYJOB: Tax 基准测试主页。

欲了解更多信息或申请访问完整基准数据集，请联系 benchmarks@surgehq.ai。

English / 原文

Anna was planning to sell her condo. She had moved out in August 2024 and wanted to know whether closing in October 2026 or January 2027 would be better for taxes.

An AI agent reviewed both options and correctly concluded that either date would satisfy the residence requirement for a tax break worth up to $250,000 in excluded gains.

But it left out an important deadline.

Under the usual rules, homeowners must have lived in a property for at least two of the five years before selling it to qualify for the exclusion. Because Anna moved out in August 2024, she would stop satisfying that requirement around the end of August 2027. If her sale was delayed beyond August 2027, she could lose eligibility.

Across five attempts, Astra never identified the deadline. Opus identified it in all five attempts.

The agent evaluated the proposed dates correctly, but its advice left Anna unaware of a deadline that could affect a major financial decision. A tax adviser should recognize consequential issues even when the client doesn't know to ask about them.

This is the kind of professional judgment we're testing with DAYJOB: Tax , the latest addition to Surge AI's DAYJOB benchmark family.

Introducing DAYJOB: Tax

DAYJOB evaluates AI agents on professional tax assignments requiring specialized expertise, complex materials, and decisions that aren't fully specified in the initial instructions. Our earlier releases covered Healthcare and Finance. DAYJOB: Tax extends that work into tax preparation, compliance, and advisory services.

The benchmark contains 50 environments across eight areas of tax practice. Agents receive client requests and supporting records, then produce tax returns, workpapers, technical reviews, or client advice.

These assignments test professional judgment alongside tax knowledge, calculations, retrieval, and instruction following. Agents must decide what deserves investigation, resolve conflicting evidence, recognize when information is insufficient, and carry those decisions into their final work.

Results: Leaderboard and Pareto Frontiers

Claude Sonnet 5.5 leads with a 10.8% pass rate on the DAYJOB: Tax leaderboard, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%.

We report pass@1 at a 95% rubric threshold. An attempt passes when it satisfies at least 95% of the task's criteria, which assess calculations, supporting evidence, consequential issues, and the completed deliverable.

Performance versus cost

We also compared model performance against average cost per trial.

Claude Sonnet 5.5 and Opus 5.5 occupy the high-performance end of the measured Pareto frontier.

The breadth of professional tax work

DAYJOB: Tax covers eight areas, each requiring different rules, records, and expertise:

  • Individual and family tax
  • Small-business tax
  • Partnerships
  • Property and investments
  • Estates, gifts, and trusts
  • International tax
  • Corporate transactions
  • Tax review and disputes

Some assignments require complete filing packages with supporting workpapers. Others ask agents to investigate transactions, review previous filings, or prepare advice about decisions clients haven't yet made.

What's inside the assignments?

The benchmark contains 1,789 workspace files , with a median of 30 per assignment and as many as 123. These include 1,487 PDFs, 117 Excel workbooks, 38 Word documents, and additional emails, images, CSVs, JSON records, and text files.

The workspaces include filed and draft tax returns, K-1s, W-2s, IRS notices, general ledgers, payroll records, contracts, and correspondence. They also contain the less standardized evidence that tax professionals routinely work with.

One assignment includes a photograph of a handwritten receipt for merchandise purchased for resale. The agent must read the item descriptions and amounts, reconcile them with spending records, and determine how the purchases affect inventory and cost of goods sold.

Another assignment involves an industrial silo-services company whose client assumes it operates only in Alabama. But its field dispatch log shows crews working across five states. The job locations, crew sizes, and days on site provide evidence relevant to the company's multistate tax obligations.

Other environments are similarly varied. One involves a cross-border household whose Polish employment and property records must be reconciled with cryptocurrency exchange exports, wallet histories, and historical purchase lots.

Created and reviewed by tax experts

DAYJOB tasks begin with real-world professional workflows. Tax experts turn these situations into realistic assignments, developing the supporting records, expected outcomes, and grading criteria. Each task undergoes a three-layer expert review for correctness, completeness, and solvability.

Tax experts bring experience across corporate taxation, partnerships, individual returns, trusts, and multistate compliance. Their backgrounds include:

  • Corporate and private-equity taxation: A tax attorney with over 10 years of experience at PwC, Deloitte, and Exxon, covering tax planning, compliance, audits, and controversy.
  • Partnership and corporate taxation: A Baker Tilly Tax Director with experience at PwC, Moss Adams, and Windes, specializing in complex partnership allocations, consolidated corporate returns, REITs, and high-net-worth clients.
  • Real estate and individual taxation: A tax professional with experience at Deloitte and EY, specializing in real-estate partnerships, individual taxation, and trusts.

A retirement plan built on an unverified assumption

A separating couple asked an AI agent how to divide their Roth IRA and 401(k). Opus 5.5 told them they weren't legally married and recommended keeping their accounts separate, with a cash payment to even out the balances.

It opened one response:

You aren't legally married, so neither account can be split without triggering taxes and penalties.

But the couple's own records raised doubts about that conclusion. Although they now lived in Florida, where new common-law marriages generally aren't recognized, they had previously lived together in Texas and held a ceremony in Colorado. Texas recognizes informal marriages under certain conditions, and a valid marriage established elsewhere may remain valid in Florida.

That distinction could change the tax consequences of dividing their retirement accounts.

Across all five attempts, Opus based its recommendations around the assumption that the couple was unmarried , even when it acknowledged uncertainty. Astra recognized the need to verify their marital status in four of five attempts. One response advised:

We first need to confirm your legal marital status.

The client asked how to divide the accounts. The agent needed to establish whether they were married before recommending a course of action. The evidence that should have prompted that investigation was already in the files.

Where DAYJOB: Tax fits

Most existing tax benchmarks begin with a defined problem: a legal question to answer, a return to calculate, or missing taxpayer information to collect. These evaluations measure important capabilities, but much of a tax professional's work involves figuring out what the problem actually is. Clients provide incomplete instructions, records contain conflicting evidence, and consequential issues may never appear in the original request.

DAYJOB tests whether agents can handle that end-to-end responsibility , including identifying issues the client overlooked, evaluating conflicting evidence, and deciding what needs to be done in the first place.

TaxBench

Can the model answer a difficult tax question?

TaxBench evaluates 114 tax questions across multiple specialties, each with a defined problem and grading rubric.

DAYJOB also asks whether the client asked the right question. Anna wants to compare two dates for selling her condo. Both satisfy the residence requirement for a tax exclusion, but does the agent warn her when that eligibility could expire?

TaxCalcBench

Can the model calculate an accurate return?

TaxCalcBench evaluates tax-return accuracy against expected filing values, using realistic tax documents and taxpayer scenarios.

DAYJOB also asks whether the underlying records actually support the tax treatment. A studio's ledger references an attached receipt, but the receipt is missing. Does the agent flag the missing evidence or incorrectly treat the reimbursement as verified?

TaxLab

Can the agent gather missing information and file correctly?

TaxLab tests whether agents can question a simulated taxpayer, obtain missing information, and complete a federal individual tax return.

DAYJOB extends that responsibility across the wider tax profession. The deliverable might be a partnership loss analysis, an estate return, a review of a disputed tax position, or a strategy letter.

The central distinction is how much responsibility remains with the agent. DAYJOB evaluates whether agents can discover consequential issues, exercise professional judgment, and carry their findings through to a completed filing, review, or recommendation.

What DAYJOB: Tax measures

DAYJOB tests whether AI agents can take ownership of a professional assignment from start to finish. That means navigating messy records, identifying issues the client never raised, making sound judgments with incomplete information, and delivering work that meets professional standards.

The results show how difficult that remains for frontier models. Even when agents can perform the individual steps, exercising professional judgment and reliably carrying out real-world tax workflows remain out of reach.

Agents haven’t yet passed the audit.

The full leaderboard, with every model we evaluated, is on the DAYJOB: Tax benchmark page.

To learn more or request access to the full benchmark, contact benchmarks@surgehq.ai.