[提交于 2026年7月7日]

标题：超越排行榜：大型语言模型智能体工具使用、规划与推理失败的综合分析

查看论文 PDF：超越排行榜：大型语言模型智能体工具使用、规划与推理失败的综合分析

摘要：大型语言模型（LLM）智能体越来越多地在其使用工具、规划多步骤任务、与其他智能体协调以及长时间跨度下运行的能力上进行评估。所报告的基准分数提升往往掩盖了在互不相关的评估工作中反复记录的失败模式。本文综合了 27 篇基准测试、分类法和审计论文（2023-2026），涵盖 19 个不同的基准，形成一个跨领域的智能体局限分类体系。据我们所知，这是首次将工具使用、规划、长程推理、多智能体协调、安全性和测量有效性方面的证据整合为一个统一的 LLM 智能体局限分类体系。我们识别出六大失败集群：(1) 工具调用和参数级错误，(2) 规划和约束满足失败，(3) 由上下文累积导致的长程退化，(4) 多智能体协调失败，(5) 对抗性或不充分指定条件下的安全失败，以及 (6) 测量有效性问题。该分类体系通过将独立报告的错误类别按智能体从推理到执行流水线的不同阶段进行主题归类而迭代得出。纵观文献，我们发现失败随任务长度非线性累积，单个子任务的出色表现不能可靠地转化为端到端成功，并且额外的脚手架并不能持续提高可靠性。与此同时，在单轮工具使用、短程网页导航和范围狭窄的编码任务上已经展现出显著进展。

提交历史

来自：Wael Albayaydh [ 查看电子邮件 ]

[v1] 2026年7月7日（周二）03:05:13 UTC (175 KB)

[Submitted on 7 Jul 2026]

Title:Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents

View a PDF of the paper titled Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents, by Wael Albayaydh and 2 other authors

View PDF

Abstract:Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons. Reported benchmark gains often obscure recurring failure modes documented across otherwise unrelated evaluation efforts. This paper synthesizes 27 benchmark, taxonomy, and audit papers (2023-2026), spanning 19 distinct benchmarks, into a cross-cutting taxonomy of agent limitations. To our knowledge, this is the first synthesis that integrates evidence across tool use, planning, long-horizon reasoning, multi-agent coordination, safety, and measurement validity into a single, unified taxonomy of LLM agent limitations. We identify six failure clusters: (1) tool invocation and parameter-level errors, (2) planning and constraint-satisfaction failures, (3) long-horizon degradation from context accumulation, (4) multi-agent coordination failures, (5) safety and security failures under adversarial or underspecified conditions, and (6) measurement validity problems. The taxonomy was derived iteratively by grouping independently reported error categories into themes corresponding to distinct stages of the agent reasoning-to-action pipeline. Across the literature, we find that failures compound nonlinearly with task length, that strong performance on individual sub-tasks does not reliably translate into end-to-end success, and that additional scaffolding does not consistently improve reliability. At the same time, substantial progress has been demonstrated in single-turn tool use, short-horizon web navigation, and narrowly scoped coding tasks.

Submission history

From: Wael Albayaydh [

view email

]

[v1]

Tue, 7 Jul 2026 03:05:13 UTC (175 KB)