数据驱动解析：AI Agent 为何仍频频失败

AI Agent 进入主流已有大约两年。根据 Capgemini 2025 年的报告，预计到 2028 年，AI Agent 将在所调查的 14 个国家创造约 4500 亿美元的经济价值。距离这个时间点已不足一年半。那么，AI Agent 现在发展得怎么样了？

编程 Agent：表现极其出色。Claude Code、Codex 等产品已经跨越了从自动补全到端到端交付真实工作的门槛。根据一些非正式观察，我认识许多人甚至已经不再打开他们的 IDE。但事实证明，编程 Agent 是例外。

根据斯坦福大学 2026 年 AI Index，组织对 AI 的采用仍在持续上升，但几乎所有业务职能中的 AI Agent 部署率仍处于个位数。专门针对 AI Agent，来自斯坦福、伯克利、IBM 等机构的研究人员发现，68% 的已部署 Agent 在需要人工介入前执行的步骤不超过 10 个。

人们很容易在脑中外推 AI 的进步：“如果 AI 能处理反驳雅可比猜想（一个已有 87 年历史的数学猜想）这样困难的问题，那么它当然也能处理我的后台工作流。”这种推理方式实际上极其常见，以至于它有一个专门的术语：“AI 的锯齿状前沿（Jagged Frontier）”，用于描述 AI 能力的不均衡。

一个有趣的例子是：Gemini Deep Think 的表现足以让它在 IMO 获得金牌，但它正确读取模拟时钟的准确率却只有 50.1%。

不过，这些都无法解释为什么企业 Agent 的实际表现不及预期。简短的答案是：要打造始终稳定高效的 AI Agent 非常困难。

总体而言，四个因素解释了大部分差距：方差、基准测试质量（数据集与评估）、Agent 特有错误，以及对齐。下面将以数据为依据，深入探讨这些因素，以及为什么我们曾被承诺的惊人企业 AI Agent 价值，与如今实际交付的有限价值之间存在如此巨大的鸿沟。

---

关于方差与可靠性

在非确定性系统中解决可靠性问题，本身就是一个极其困难的问题。为说明这一点，我们将 AI Agent 建模为一个需要自主执行若干步骤来完成任务的系统。假设任意单一步骤成功的概率为 90%。

当你的 Agent 尝试一个包含 10 步的工作流时，只有 34% 的 Agent 运行能够成功完成全部 10 步（0.9¹⁰）。当流程增加一倍、变为 20 步工作流时，端到端通过率下降至 12%。30 步工作流的成功率则为 4.2%。

端到端可靠性会随着步骤数量增加而衰减。

由于每一个步骤如今都是非确定性的，系统整体可靠性会随着工作流步骤数量增加而呈指数下降。这是一个会被复杂性进一步放大的重大问题。

即便我们把单一步骤成功的概率提高到 99%，经过 30 步后，端到端运行成功完成的预期比例也只有 73.97%；相较于 99% 的单步成功率，这仍是大幅降级。

当然，真实 Agent 并没有这么简单。上述计算假设每个步骤彼此独立，但后续步骤实际上并不独立。现实中，错误会不断累积（某一步出错会污染此后每一步的上下文）。但另一方面，Agent 也可以从错误中恢复，你可以构建护栏来保证特定输出格式，等等。这个例子只是一个一阶近似，以极其简化的方式对 AI Agent 进行建模。遗憾的是，这个例子所说明的观点并不只是一阶近似：步骤越多，可靠性下降得越快。

我们刚刚描述的问题是“单次运行内方差（intrarun variance）”；换言之，即一次运行内部的方差。

但你的 Agent 不会只运行一次。它需要很多很多次地解决相同或相似的任务。

单次运行内方差对应的是跨运行方差（interrun variance）——即多次运行之间的方差。Sierra 知道这是个问题，因此当他们构建τ-bench（客户支持领域的行业标准基准之一）时，将 pass^k（pass-to-the-k）设为一项一等指标。简而言之，pass^k 是对跨运行方差的度量：将数据集中的每项任务运行 k 次；只有当它在全部 k 次中都成功时，该任务才算通过。换句话说，pass^k 问的是你的 Agent 每一次都能做什么，而不只是某一次能做什么。

为了用一个更具体的例子说明跨运行方差问题，让我们看一看 τ³ bench（改进版 τ-bench）。在τ-knowledge（银行领域）中，表现最好的模型是 gpt-5.2（高推理），其 pass^1 为 25.52%（这里的 pass^1 指的是：他们在该领域中将每项任务运行一次，其中 25.52% 通过）。

在 pass^4 下，这一数值约减半至 13.40%（这里的 pass^4 指的是：他们将数据集中的每项任务运行 4 次，只有 13.40% 的任务在全部 4 次中均通过）。请注意，这并不是旧模型的问题：gpt-5.2 是前沿推理模型，而跨运行方差仍然让其性能减半。

pass^k 告诉你 Agent 的一致性有多高。一个自然的后续问题是：它有多大能力？你可以用 pass@k 来衡量：在 k 次运行中，只要至少有 1 次运行通过，就算成功。你可以将 pass^k 想象为成功运行的交集，而将 pass@k 想象为成功运行的并集。

观察下图，所有模型都遵循相同模式，因此我们只关注 gpt-4o（橙色）。虚线展示了在 k 次试验中的 pass@k 准确率（当前的最大能力）。如你所见，虚线起点约为 60%，但很快接近约 95%。

另一方面，实线展示了在 k 次试验中的 pass^k 准确率（一致性衡量指标）。与上面的银行案例非常相似，实线迅速衰减至原始准确率的大约一半。

但这为什么重要？

可以将 pass@k 看作上限，即模型在其最佳一次尝试中能做到什么。pass^k 则是下限；也就是说，在生产环境中你真正可以依赖的能力。对于 τ-bench 上的 gpt-4o，上限约为 95%，下限约为 30%。这很重要，因为退款被处理错误的客户并不在乎：这个 Agent 另外 70% 的时间原本能做对。

此时你可能会想：如果上限是 95%，为什么不让 Agent 运行 k 次，然后选出成功的那一次？在某些领域，这实际上可行。编程再次是幸运儿：你可以运行测试，并知道哪一次尝试是正确的。但大多数 Agent 任务并没有答案钥匙。在生产环境中，没有任何东西能告诉你 k 次运行中哪一次是正确的。如果你能自动检测成功运行，你就已经解决了 Agent 评估问题（后文会进一步说明）。更糟的是，Agent 会对现实世界采取行动，而现实世界并不总是如此宽容。你无法将一笔退款处理 4 次，然后保留其中最好的一次。

因此，当你看到 gpt-4o 在 τ-bench 上的准确率约为 60% 时，你的第一反应或许是：我们需要让 Agent 变得更好。但 pass@k 表明，这 40% 的性能差距几乎完全可以由方差解释，而不是由 Agent／模型能力不足解释。

换句话说，gpt-4o 已经能够解决 95% 的问题，只是无法稳定地解决它们。

这种模式几乎可以在所有公开基准中观察到，也可以在我们合作过的客户所使用的 AI Agent 中观察到。方差是 AI Agent 的巨大瓶颈之一——既包括单次运行内方差，也包括跨运行方差。

它还有一个我们尚未讨论的后果：如果单次运行如此嘈杂，那么单个基准分数也是一个嘈杂的样本。方差不仅使你的 Agent 不可靠，也使你对 Agent 的测量不可靠。这就引出了下一个重大问题：数据集和基准测试。

---

关于基准测试

方差会让你的分数充满噪声。但还有第二个、更隐蔽的问题：你用于训练和评估 Agent 的数据集本身可能就是错误的。

2016 年，一名 19 岁的 MIT 学生和一名 21 岁的卡内基梅隆大学辍学生登上了飞往旧金山的航班——这趟航班将永远改变他们的人生。他们刚刚被 Y Combinator 录取，而他们的公司逐渐聚焦于一个简单的理念：机器学习不仅由算力驱动，也由更优质的训练数据驱动。他们最初在自动驾驶汽车公司中获得成功，后来扩展至机器人、国防、政府等其他行业。九年后，Alexandr Wang 和 Lucy Guo 的公司 Scale AI 估值约为 290 亿美元——这证明在 AI 时代，数据与算力同样有价值。

我们同样相信，AI Agent 开发中的数据也具有这样的价值。一个常见误解是：如果你没有对模型进行微调，你就没有训练数据。但你的 Agent 仍然在一个数据集上接受训练（只是并非通过梯度下降）。每当你根据 Agent 在数据集上的表现调整 Prompt、重写工具描述或重构工作流时，你都在用这些数据训练 Agent。数据集是整个迭代循环所依赖的信号：它决定哪些实验看起来有希望、哪些回归会被发现，以及哪些“改进”会被上线。

不用说，你用来训练的数据非常重要；事实上，有些人会认为，你的 Agent 只能和其训练数据一样好。即便 Agent 在你的数据集上表现出色，如果数据集已经陈旧、过时，或者与生产环境不属于同一分布，它在生产中仍会表现不佳。因此，数据集条目的质量和分布都极为重要。

遗憾的是，做好数据集很难。就连 Sierra 的行业标准 τ-bench 数据集，也被重新修订为一个τ³ 数据集。在原始数据集和改进后数据集之间，164 项任务中有 53 项因“无法完成或相互矛盾的约束条件”“预期操作错误”等原因需要更新。这占原始基准任务的 30% 以上。

WebArena 也存在类似问题：其最初的 812 项任务需要进行大规模修复（例如，评估器未对齐、澄清模糊指令、替换脆弱的匹配规则）。在此之后，这些问题导致的假阴性在其基线 Agent 中减少了 11%。同样的数据集质量问题还出现在许多其他基准中。

修复原始基准中的问题是巨大进步，但这些问题会导致性能大幅波动，足以彻底改变决策，并影响 Agent 的调整方式。

在实践中，如果 30% 的任务存在问题，那么行为正确的 Agent 可能比恰好匹配基准错误的 Agent 得分更低。我们见过团队陷入这一陷阱，即使使用的是内部基准和数据集：他们看到某个数字变化，就得出某项改动有效（或无效）的结论，并据此调整方向。数据集中的这些系统性问题，会让工程团队耗费数月追逐错误的指标、为错误的目标优化，最终离可靠、可投入生产的 Agent 越来越远。

此时，预期中的回应可能是：“即使大多数公开基准存在问题，我也不用它们。我根据自己的生产数据构建了自己的数据集。”这当然是正确的做法，但要做好它远比听起来困难。下面我们将简要讨论三个原因（这是一个非常复杂的话题，另一篇博客将进一步展开）。

你的数据集会在上线的那一刻开始过时。 大多数团队在生产上线前构建一次数据集，随后要么

1. 从不更新它，或

2. 煞费苦心地手动更新它

但生产数据与你最初的数据集差异很大，就像产品发布后会不断积累许多任何上线前 QA 都无法预见的 Bug。因此会形成一个循环：你需要可观测性来了解 Agent 在哪里出错；你为修复这些失败而做出改动；然后你需要测试这些改动……但测试所用的数据集并不包含你所针对的生产案例。你的数据集无法告诉你修复是否有效，或它是否引入了回归，因为其中没有你的生产测试案例。于是，你必须永远手动将生产追踪记录导入数据集。这恰恰是应当成为自动化流水线的事情，而这又引出了一个问题：流水线中究竟该纳入什么？

选择哪些内容加入数据集，比听上去更难。 我们合作过的许多公司都抱怨：他们拥有数百万条生产追踪记录，却不知道该如何处理。

你应该利用它们让 Agent 变得更好。但具体怎么做？把所有记录盲目塞进数据集会带来更多问题。迭代时间和成本大致会随数据集规模线性增长，而无法快速迭代的数据集，最终会成为你不再使用的数据集。这意味着，你希望拥有的是一组尽可能小、但仍能代表生产分布的追踪记录。

天真的答案是随机抽样，但想想这样会得到什么：如果你的 Agent 已经在 80% 的生产流量中成功，那么随机样本中约 80% 都是你已知 Agent 能通过的案例。你会将大部分评估预算花在重复确认已经有效的事情上，而非探索你真正关心的失败案例。只抽取失败案例同样不对，因为那样你就无法发现那些修复了失败、却在其他地方造成大规模回归的改动。

而且，选择本身还有一个更深层的限制：如果你的数据集只包含你已经见过的追踪记录，你就只能事后修复问题。这并没有用。想象一下，如果你只能从自己已经犯过的错中学习

It’s been about two years since AI agents went mainstream. According to Capgemini’s 2025 report, AI Agents were estimated to generate roughly $450 billion in economic value across 14 surveyed countries by 2028. That’s less than a year and a half from now. So, how are AI agents doing now?

Coding agents: remarkably well. Claude Code, Codex, and others have crossed the threshold from autocomplete to shipping real work end-to-end. Anecdotally, I know many people who don’t even open their IDEs anymore. But it turns out, coding agents are the exception.

According to Stanford’s 2026 AI Index, organizational AI adoption continues to rise, but AI agent deployment remains in the single digits across almost all business functions. For AI agents specifically, researchers from Stanford, Berkeley, IBM, etc. found that 68% of deployed agents execute no more than 10 steps before human intervention.

It’s tempting to mentally extrapolate improvements across AI: “if AI can handle something as hard as disproving the Jacobian conjecture (an 87-year-old math conjecture), surely it can handle my back-office workflow.” This line of reasoning is in fact so common that there is a term for it: the “Jagged Frontier” of AI, which describes the imbalance of AI capabilities.

An interesting example to illustrate this is how Gemini Deep Think performed well enough to earn a gold medal at IMO, but can only read an analog clock correctly 50.1% of the time.

None of that explains why enterprise agents underdeliver, though. The short answer is that making AI agents that consistently perform well is very hard.

In general, four things explain most of the gap: variance, benchmark quality (datasets & evaluations), agent-specific errors, and alignment. What follows is a data-driven deep dive into each and why there’s such a massive gap between the incredible enterprise AI Agent value we were promised and the little value that has been provided.

---

On Variance & Reliability

Solving reliability in a non-deterministic system is inherently a really hard problem. To illustrate this, let’s model an AI agent as a system that performs some number of steps autonomously to complete a task. Let’s assume the probability that any step succeeds is 90%.

When your agent attempts a workflow that takes 10 steps, only 34% of your agent runs will succeed on all 10 steps (0.9¹⁰). When that process doubles to a 20-step workflow, the end-to-end pass rate decreases to 12%. A 30-step workflow has a 4.2% success rate.

End-to-end reliability decays as the number of steps grows.

Because every step is now non-deterministic, the overall reliability of the system decreases exponentially with the number of steps in your workflow. This is a big problem that gets exacerbated by complexity.

Even if we increase the probability of a single step succeeding to 99%, after 30 steps, you can only expect 73.97% of your end-to-end runs to complete successfully, which is a massive downgrade from the 99% individual step success rate.

Of course, real agents aren’t this simple. The calculations above assume that each step is independent of each other. Subsequent steps are not independent. In reality, errors compound (a bad step corrupts context for every step after it). But on the flip side, agents can also recover from their mistakes, you can build guardrails to guarantee specific output formats, etc. This example is just a first-order approximation, modeling an AI agent extremely simply. Unfortunately, what is not just a first-order approximation is the idea this example illustrates: more steps means less reliability, fast.

The problem we just described was “intrarun variance”; in other words, the variance within a run.

But your agent doesn’t just run once. It will need to solve the same or similar tasks many, many times.

Intrarun variance’s counterpart is interrun variance—the variance across multiple runs. Sierra knew this was a problem, which is why when they built τ-bench (one of the industry standard benchmarks for customer support), they made pass^k (pass-to-the-k) a first-party measurement. Pass^k, simply put, is just a measurement for interrun variance: run each dataset task k times; a task only counts as passed if it succeeds all k times. In other words, pass^k asks what your agent can do every single time, not just once.

To solidify the issue of interrun variance in a more concrete example, let’s take a look at τ³ bench (an improved τ-bench). For τ-knowledge (banking domain), the highest performing model was gpt-5.2 (high reasoning) with a pass^1 of 25.52% (pass^1 here means they ran each task in the domain once, and 25.52% of them passed).

At pass^4, it dropped by about half to 13.40% (pass^4 here means they ran each task in the dataset 4 times and only 13.40% of tasks passed all 4 times). And note that this is not an old-model problem: gpt-5.2 is a frontier reasoning model, and interrun variance still cuts its performance in half.

Pass^k tells you how consistent your agent is. A natural follow-up question is: how capable is it? You can measure this with pass@k: of k runs, if at least 1 of the k runs passed, it counts as a success. You can imagine pass^k as an intersection of the successful runs and pass@k as a union.

Looking at the graph below, all the models follow the same pattern, so let’s just focus on gpt-4o (orange). The dotted line shows the accuracy for pass@k (max current capability) across k trials. As you can see, the dotted line starts at ~60% but quickly approaches ~95%.

On the other hand, the solid line shows the accuracy for pass^k (consistency measurement) across k trials. Much like the banking example above, the solid line quickly decays to around half of the original accuracy.

Why does this matter though?

Think of pass@k as the ceiling. What the model can do on its best attempt. pass^k is then the floor; i.e., what you can actually count on in production. For gpt-4o on τ-bench, the ceiling is ~95%, and the floor is ~30%. This matters because a customer whose refund gets botched doesn’t care that the agent would have gotten it right the other 70% of the time.

At this point, you might be thinking: if the ceiling is 95%, why not just run the agent k times and take the run that succeeded? In some domains, this actually works. Coding is (again) the lucky one: you can run the tests and know which attempt was correct. But most agentic tasks don’t come with an answer key. In production, nothing tells you which of the k runs was the right one. If you could automatically detect successful runs, you would have already solved agent evaluation (more on that later). And worse, agents act on the world, which isn’t always so forgiving. You can’t process a refund 4 times and keep the best one.

So, when you see that gpt-4o has an accuracy of ~60% on τ-bench, maybe the first instinct is that we need to make the agent better. But the pass@k shows that the 40% performance gap is almost all explained by variance, not agent/model capability.

In other words, gpt-4o can already solve 95% of the problems. Just not consistently.

This pattern can be seen across almost all public benchmarks as well as with AI agents of customers we have worked with. Variance is one of the massive bottlenecks of AI agents—that is both intrarun and interrun variance.

And it has one more consequence we haven’t talked about yet: if a single run is this noisy, then a single benchmark score is a noisy sample too. Variance doesn’t just make your agent unreliable; it makes your measurements of your agent unreliable. This brings us to the next big problem: datasets and benchmarks.

---

On Benchmarks

Variance makes your scores noisy. But there’s a second, sneakier problem: the dataset you’re training and evaluating your agent on can itself be wrong.

In 2016, a 19-year-old MIT student and a 21-year-old Carnegie Mellon dropout hopped on a flight to San Francisco—a flight that would change their lives forever. They had just gotten accepted into Y Combinator, and their company converged on a simple idea: machine learning was not only driven by compute, but also by better training data. They found early success with autonomous-vehicle companies, and eventually grew into other industries like robotics, defence, government, and more. Nine years later, Alexandr Wang and Lucy Guo’s company, Scale AI, was valued at ~$29 billion—proof that in the age of AI, data is just as valuable as compute.

We believe the same about data for AI agent development. A common misconception is that if you’re not fine-tuning a model, you don’t have training data. But your agent is still trained on a dataset (just not through gradient descent). Every time you adjust a prompt, rewrite a tool description, or restructure a workflow because of how your agent performed on your dataset, you are training your agent on that data. The dataset is the signal your entire iteration loop is steered by: it decides which experiments look promising, which regressions get caught, and which “improvements” ship.

Needless to say, the data you train on is quite important; in fact, some would argue that your agent will only be as good as the data that it’s trained on. Even if your agent performs well on your dataset, it will perform poorly in production if the dataset is stale/outdated, or if it’s not from the same distribution. Because of this, both the quality of and the distribution of dataset items matter a lot.

Unfortunately, datasets are hard to make well. Even Sierra’s industry-standard τ-bench dataset was revamped into a τ³ dataset. Between the original and the improved dataset, 53 tasks out of 164 total tasks needed to be updated for reasons like “Impossible or Contradictory Constraints”, “Incorrect Expected Actions”, etc. That’s over 30% of the tasks of the original benchmark.

WebArena had a similar problem: its 812 original tasks needed a substantial round of repairs (e.g. misaligned evaluators, clarified ambiguous instructions, replaced brittle matching rules). Following this, false negatives from these issues were reduced by 11% in their baseline agent. The same dataset quality problem has been seen across many other benchmarks.

It’s a massive step forward that the issues in the original benchmarks were fixed, but these issues cause massive swings in performance, which can completely flip decisions and influence how agents are adjusted.

In practice, if 30% of your tasks are broken, an agent that behaves correctly can score worse than one that happens to match the benchmark’s mistakes. We’ve seen teams fall into this trap, even with internal benchmarks & datasets: they see a number move, conclude a change worked (or didn’t), and steer accordingly. These systemic issues in datasets cost engineering teams many months chasing the wrong numbers and optimizing for the wrong things, leading them farther and farther away from reliable, production-ready agents.

At this point, an expected response may go something like this: “even if most public benchmarks are broken, I don’t use them. I built my own dataset from my own production data.” That’s definitely the right thing to do, but it’s also much harder to do well than it sounds, for three reasons we will briefly survey (it’s a really complex topic. More on this in another blog).

Your dataset goes stale the moment you ship. Most teams build their dataset once, before production, and then either

1. never update it, or

2. painstakingly update it by hand

But production data differs a lot from your original dataset. The same way a product accumulates bugs after release that no amount of pre-launch QA predicted. So a loop forms: you need observability into where your agent is messing up, you make changes to fix those failures, and then you need to test those changes... against a dataset that doesn’t contain the production cases you were targeting. Your dataset can’t tell you whether the fix worked or whether it introduced regressions because it doesn’t have your production test cases. Now you’re porting production traces into your dataset by hand, forever. This is exactly the kind of thing that should be an automated pipeline, which raises the question of what goes into it.

Selecting what is added to the dataset is harder than it sounds. A lot of the companies we’ve worked with complain that they have millions of production traces and no idea what to do with them.

You should be using them to make your agent better. But how? Blindly chucking all of them into your dataset causes more problems. Iteration time and cost grow roughly linearly with dataset size, and a dataset you can’t iterate against quickly is a dataset you’ll stop using. This means you want the smallest set of traces that is still representative of your production distribution.

The naive answer is random sampling, but think about what that gets you: if your agent already succeeds on 80% of production traffic, then ~80% of a random sample are cases you already know your agent passes. You’d be spending most of your eval budget re-confirming things that work instead of probing the failures you actually care about. Only sampling failures isn’t right either, since then you can’t catch changes that fix your failures but cause massive regressions elsewhere.

And selection alone has a deeper limit: if your dataset only contains traces you’ve already seen, you can only fix problems retroactively. That’s not useful. Imagine if you could only learn from mistakes after you’ve made them. You’d have to get hit by every single car of every make and model and year before you learn that you shouldn’t step in front of a moving car.

A bit of a silly example, but it illustrates that the goal here is beyond fixing your agent on specific test cases. The goal is continuous learning.

Covering the edge cases you haven’t hit yet means generating synthetic dataset items, too. Just like selection, this shouldn’t be done manually. How do you automate choosing which cases to generate while ensuring representative test cases, quality, and minimizing the added test cases? Synthetic dataset item creation is its own hard problem—and one that needs the same thing your selected traces need: somewhere to run, which is the next problem.

Production isn’t static, so you can’t just replay it. Suppose your agent answered a customer’s question last Tuesday: “how many flights did I take in the last 7 days?” The correct answer to that trace is pinned to what your production database looked like last Tuesday. The database has changed since. If you replay that trace today against production, the “right answer” is different; if you froze the trace, you now need a mock database, mock tools, and an environment that reproduces the state the agent originally saw. But that’s not the hard part. The hard part is building an accurate evaluator for this testing environment. As illustrated above, even the most trusted public benchmarks struggle to get their test cases and evaluators right, a problem almost no team has fully solved so far (we’ll talk about a big contributing factor in the final section on alignment).

Creating a good dataset of your own with a reliable and accurate evaluator is a complicated enough topic to warrant its own deep-dive blog later.

So datasets are noisy, they rot, and they’re expensive to rebuild. But suppose you get all of it right: variance accounted for, dataset clean and current. Your agent still fails sometimes. What do those failures actually look like? It turns out they typically follow patterns.

---

On Agent Errors

Let’s give credit where it’s due: reasoning models have meaningfully cut down on agent mistakes. The flailing, obviously-broken behavior of early agents is mostly gone. But agents still fail, and when you actually categorize the failures, two interesting things show up: the mistakes follow patterns, and those patterns are different for every model.

Let’s start with a 2024-era baseline. When the τ-bench authors manually analyzed a random sample of 115 tasks (40 of which failed), this was the breakdown of gpt-4o’s errors:

A third of failures were the agent calling the right tool with the wrong arguments. Another quarter were wrong decisions, and about a fifth each were providing wrong information and only partially resolving the task. Notice what these have in common: they’re largely mechanical. The agent knew roughly what to do and fumbled the execution.

Now fast-forward to a more recent analysis of three frontier models on the same benchmark. Instead of categorizing failed runs, the researchers manually analyzed 131 cases where the agent passed the benchmark while violating procedure along the way. They call these “corrupt successes”: the agent reaches the right final state, but on the way it misquotes a fee, invents a policy that doesn’t exist, or tells the user it executed an action it never actually took. The outcome checker sees a completed task, but in reality, it’s “corrupt”. By their audit, 27–78% of benchmark-reported “successes” (depending on the model) contained at least one violation like this. Here’s how those violations break down:

Two things stand out. First, the categories themselves changed since 2024. Of course, the analysis was not done by the same researchers, so this is not completely unexpected. But the shift is still there: the mechanical-execution failures that dominated the 2024 breakdown are now a small slice, and the dominant failure modes are things like policy faithfulness and data faithfulness. Essentially, the agent took actions that quietly deviate from the rules it was given, or from the data in front of it. The mechanical fumbles shrank; the failures moved up the stack, from “called the tool wrong” to “didn’t stay grounded in the policy and data.” This is dangerous because mechanical errors fail the task, so your metrics catch them. Grounding errors pass the task, so they ship to production without getting caught. Better reasoning didn’t completely eliminate mistakes. But it did change which mistakes were left while making them harder to detect.

Second, every model seems to have a different dominant failure mode. GPT-5’s violations are led by policy compliance (35.1%), meaning it executes actions the rules didn’t allow. Kimi-K2-Thinking’s biggest issue is policy faithfulness, at nearly half of its violations (47.8%): confident statements about policies that don’t exist. Mistral’s is data faithfulness (28.4%), a category where GPT-5 barely appears (5.4%), which manifests in fabricating prices, flight details, and confirmation numbers.

These distributions come from one domain in one study, so treat the exact numbers as illustrative. The pattern, though, matches what we see across agents in production.

Why does this matter? If agent errors were purely a model-capability problem, you’d expect them to look roughly the same across frontier models and to shrink uniformly with each generation. Instead, each model has its own failure fingerprint. That means “wait for the next model” is not a strategy: the errors are partly a property of the system around the model (the prompts, tools, context, and guardrails), and that system has to be tuned to the specific model running inside it. When you change to a different or “better” model, you’re mostly decreasing the number of mechanical mistakes, which is good progress, but you’re still left with the higher-level failures: staying faithful to policy, staying grounded in data, understanding intent.

There’s also another problem that remains, even after the AI agent is “perfect”: alignment. Alignment is tricky because most of the time, the agent didn’t technically make a mistake at all. It followed the policy, used the tools correctly, stayed grounded in the data, and still did something you didn’t want.

---

On Alignment

Let’s start somewhere concrete: LLM judges. When teams try to automate evaluation, the standard move is to have an LLM grade the agent’s runs. And the standard complaint we hear is that the quality just isn’t there. The skeptic’s argument goes like this: if the judge can tell what’s wrong, the agent should have been able to figure out the right answer in the first place. You’re using the same “brain” with the same information; in other words, if you change nothing, why would you expect a different result?

We think this argument has some merit but isn’t entirely true. There are creative ways to get better evaluations out of the same initial information: give the judge more compute (e.g., compare a reasoning model to a non-reasoning model), add a second model critiquing the first, and more. In our testing, this resolves a small but real chunk of the quality problems. But it doesn’t come close to resolving all of them. The residual is the interesting part. When a well-architected judge with plenty of compute still grades a run “wrong,” the most common reason isn’t a reasoning error. It’s that nobody ever told the judge what you actually wanted. The judge isn’t failing to reason. It’s missing information that only exists in your head.

That’s alignment. Here, we mean alignment between AI agents and human preferences. In our opinion, it’s an “eternal” problem. What we mean by that: the development of technology will not solve it. Training a better model will not solve it. The gap between what a human wants and what they’ve actually communicated will always have to be closed directly, because no amount of capability lets a model read a preference that was never expressed.

To break the problem down, abstract the specifics of any AI agent away, and you’re left with three things: inputs, a process, and an outcome.

Let’s assume that humans control the inputs into an agent, so we’re left with the process and the outcome.

Preferences in the outcome: when scheduling a meeting, you might ask an AI agent to “find a 30-minute slot with Sarah after 10 am.” The query already has preferences that you remembered to include, such as the “30-minute” meeting and “after 10 am”. But here are a few preferences that you didn’t realize until the agent violated them:

1. The agent books the meeting in Thursday’s one gap: the gap you deliberately left between two 3-hour blocks of meetings so you could eat lunch.

2. The agent books the meeting in a slot that is technically open, but it occupies the 30 minutes right before an important board meeting, a 30 minutes you always keep free to prepare.

3. The agent books the meeting in the middle of a completely free afternoon, but you would prefer to have all your meetings back to back in the morning to get them over with (or maybe you’re the opposite and you want them spread out as natural break points throughout the day).

Everyone carries hundreds of preferences like these (some of which are easier to concretize than others), and so do companies (e.g., document naming conventions, tone guidelines, who gets cc’d). A more general model doesn’t dissolve these since they’re not facts to be learned but rather choices to be communicated. In some form, these preferences will always need to be communicated.

Preferences in the process: say there are two viable methods for a task. Method 1 is more reliable but slower; method 2 is faster but less reliable. Which one you want depends on a lot of factors. For a personal project, maybe you just want to spin something up fast. For a production-ready project, you would prefer reliability while also not being overkill. Even here, only you know what “overkill” means. Only you know what “prefer reliability” means. How much do you prefer it? Under which circumstances would you prefer speed over reliability? You can imagine that even for just the two preferences of speed and reliability, there is a continuous spectrum, of which your preference is a small, fuzzy zone. In reality, for any given task, there are probably dozens of these small preferences. Combining two preferences like speed and reliability gives a continuous 2D plane. Three preferences become a 3D coordinate system. Four becomes a 4D vector space.

The point is that with more preferences, not only is it harder to communicate them all accurately, but also it’s harder to remember every single preference that needs to be communicated in the first place.

If you’ve used AI at all, you’ve almost definitely felt this. You prompt it to do a task you think you’ve explained extremely clearly, and it comes back with something that isn’t what you imagined. Usually it didn’t make a mistake but rather made a choice on a dimension you didn’t know you had an opinion about until you saw the wrong option. You never told it which conventions matter or what “done” looks like to you. Correcting it is you doing alignment work, one preference at a time. Needless to say, this scales with task complexity: the bigger the task, the more unstated preferences it touches, which is why alignment reliably takes more time than anyone budgets for it.

As models keep getting better, capability problems shrink: the mechanical errors go first, then more of the grounding errors. What’s left over doesn’t shrink, because it was never a capability problem. The remaining gap between what agents do and what we want them to do will increasingly be alignment: not “can the AI do the task,” but “does the AI know what doing the task means to you.”

Daunting as this sounds, it’s tractable. ‘Eternal’ doesn’t mean hopeless. It means the work never fully disappears. But preferences can be mined, encoded, and reused instead of rediscovered one correction at a time. More on this in another post.

---

So let’s return to where we started. We were promised $450 billion of agent value by 2028, and what we have instead is deployment stuck in the single digits and agents that get about 10 steps before a human steps in. The gap isn’t mysterious, and it mostly isn’t a model-capability gap: it’s compounding variance, datasets that lie, failure modes that hide inside “successes,” and preferences that were never communicated. The good news is that every one of those is a system problem, and system problems can be engineered. The next post is an introduction to how AI agents are being improved.

The cover art image in this piece is by Kirell Benzi. We take no credit for them.