摘要：基于 LLM 的 Agent 日益被认为具备了实际推理的能力：规划、监控以及纠正自身动作。它们的成功通常归功于围绕模型构建的提示策略与 Agent 架构，例如思维链（Chain-of-Thought）、基于搜索的提示、自我修正（Self-Refinement）、反思（Reflection）、审查机制（Critics）以及工具调用循环（Tool-Using Loops）。然而，这些宣称通常仅由端到端成功率支撑（该指标仅显示任务是否完成），或依赖 LLM 裁判（其本身并不是严谨的验证器）。我们基于形式语义对 LLM 的规划能力进行了评测。任务以自然语言呈现，但每个任务都锚定于一个底层的隐式 PDDL 模型中，该模型能够判定每个动作是否适用、目标是否达成，以及解法计划是否存在。在五个领域（其中两个为全新提出，均随论文开源）、四种模型、六种提示策略和三种 Agent 架构的测试中，最佳配置在离线状态下解决了 78% 的可解任务，但在在线状态下仅能解决 40%，在在线模式下 Agent 多数时候甚至不检查当前状态就直接行动。不可解任务在低于 10% 的轮次中被识别出，而更沉重的提示策略所消耗的 Token 是直接提示的 5 到 22 倍，却未见明显收益。在我们的实验中，规划表现的差异更多取决于模型本身，而非提示策略或 Agent 架构。要宣称 Agent 具备规划能力，必须提供来自经过验证的动作轨迹的实据证据。

Abstract:LLM-based agents are increasingly credited with the competences of practical reasoning: planning, monitoring, and correcting their own actions. Their success is attributed to the prompting strategies and agent architectures built around the model, such as chain-of-thought, search-based prompting, self-refinement, reflection, critics and tool-using loops. Yet these claims are usually supported by end-to-end success rates, which show only whether a task was completed, or by LLM judges, which are not verifiers. We test LLM planning against formal semantics. Tasks are posed in natural language, but each is grounded in a hidden PDDL model that decides whether every action is applicable, whether the goal is reached, and whether a plan exists at all. Across five domains (two of them new, all released with the paper), four models, six prompting strategies and three agent architectures, the best configuration solves 78% of the solvable tasks offline but only 40% online, where agents mostly act without inspecting the state. Unsolvable tasks are recognised in under 10% of episodes, and heavier prompting strategies use 5 to 22 times as many tokens as direct prompting with no clear gain. In our experiments, planning performance varied more across models than across prompting strategies or agent architectures. Claims that agents can plan require evidence from validated action traces.