摘要：LLM 裁判（LLM judges）正日益被用于评估和改进 AI 生成的内容，但它们在复杂专业工作中的可靠性仍不明确。我们通过“Vibe Patenting”——一个用于评估 AI Agent 的端到端专利撰写测试基准——来研究这一问题。独立调用的 LLM 裁判负责评估生成的专利草案，并提供结构化反馈以支持迭代修改。在多种发明案例和撰写 Agent 配置中，裁判引导的修改持续提升了经裁判评估的草案质量，而无引导的修改往往迅速趋于饱和。值得注意的是，迭代的裁判反馈使低推理 Agent 能够逼近成本明显高得多的高推理 Agent 的性能。更强大的模型和更深度的推理通常能提高裁判评估的撰写质量，而特定领域的 Agent 工作流则带来了进一步的增益。我们通过专业专利律师的独立评估对 LLM 裁判进行了验证，发现了具有实际意义但高度依赖于具体指标的一致性，以及系统性的校准差异。这些结果凸显了 LLM 裁判作为复杂专业工作流的评估器与优化信号时的实用性与局限性。

Abstract:LLM judges are increasingly used to evaluate and improve AI-generated outputs, yet their reliability for complex professional work remains unclear. We study this problem through Vibe Patenting, an end-to-end patent-drafting testbed for AI-agent evaluation. A separately-invoked LLM judge evaluates generated patent drafts and provides structured feedback for iterative revision. Across multiple inventions and drafting-agent configurations, judge-guided revision consistently improves judge-assessed quality, while unguided revision tends to saturate. Notably, iterative judge feedback enables a low-reasoning agent to approach the performance of a substantially more expensive high-reasoning agent. Stronger models and increased reasoning generally improve judge-assessed drafting quality, while domain-specific agentic workflows provide further gains. We validate the judge against independent evaluation by a professional patent attorney and find meaningful but strongly metric-dependent agreement and systematic calibration differences. These results highlight both the utility and limitations of LLM judges as evaluators and optimization signals for complex professional workflows.