🔶 从数百万次 Agent 运行中提取任务

通常，了解一次 Agent 运行时，你最先想知道的事情是：它被要求完成什么任务。答案在用户消息中，但很少以独立形式存在。消息到达模型之前，Agent 的运行框架往往已经在其中加入时间戳、环境状态、系统通知和用户详情；而且每个 Agent 的封装方式都不同。本文将介绍我们如何从这些内容中找出任务：以较低成本覆盖每天数百万次运行，并且无需询问开发者应该从哪里找。

为什么要提取任务？#

Laminar 每天处理数百万次 Agent 运行，而每一次运行都始于一个任务。这里所说的任务，是用户原样写下的请求，而不是对意图的摘要或重构。它是了解一次运行时最有用的信息之一。有了任务，你可以：

• 打开 trace 时，一眼看出 Agent 被要求做什么
• 了解用户实际上在提出哪些请求
• 找出 Agent 会在哪些任务上失败
• 基于生产数据，为不同类型的任务构建 eval

挑战#

最直接的做法，是从 Agent 第一次 LLM 调用中提取 role: "user" 的消息。有时这已经足够：许多 Agent 会把任务作为一条普通用户消息直接发送。

但同样常见的是，任务被埋在其他上下文中。一个常见原因是 prompt caching：缓存命中需要稳定的前缀，因此许多 Agent 会保持 system prompt 固定，并将每次运行都会变化的内容移到用户消息中。于是，用户轮次最终混合了以下内容：

• 静态脚手架：指令、章节标题和标记
• 动态请求上下文：谁发送了消息、正在审查的 pull request、客户资料
• 动态环境状态：当前时间、打开的浏览器标签页、磁盘上的文件、Agent 的待办事项列表
• 任务本身：通常是每次运行中唯一不同的部分

例如，下面是一个编程 Agent 某次运行的第一次 LLM 调用。在 system prompt 之后，用户轮次作为三条独立的用户消息到达：

其中只有一行是真正的任务：The checkout button still doesn't work on mobile.。其余内容都是脚手架：仓库状态、系统通知、消息发送者信息，以及对回复的指令。而且这些脚手架甚至并非固定不变，因为仓库状态会在每轮变化。

我们将这三条消息统称为用户轮次（user turn）：即出现在 Agent 上一次回复之后、role: "user" 的消息。在新对话中，这就是全部用户消息。在后续对话中，LLM 调用还会携带此前的对话内容，而用户轮次仅指自 Agent 上次回复后新增的消息。我们就是在用户轮次中寻找任务。

对于 Agent 开发者而言，这并不困难：他们确切知道任务在哪里，定制 UI 渲染器或一条简单的 SQL 查询就能将其提取出来。但一旦 Agent 发生变化，这种做法就失效了，而开发者每天都在修改 prompt。他们会增加或删除章节、重命名 XML 标签，并将上下文从一条消息移动到另一条。每次变更都可能悄无声息地破坏手写提取逻辑。与此同时，我们需要为平台上的每一个 Agent 提取任务，却没有开发者可供询问。

**简而言之：**LLM 会针对每个模板编写一条任务提取正则表达式；模板即 Agent 用来包裹每个任务的格式。因此，大多数运行根本不需要 LLM 调用。当格式发生变化时，我们会通过将每次运行与近期相似运行进行比较来发现变化，并为新格式编写新的正则表达式。本文余下部分将介绍促成这一设计的几次尝试。

尝试 #1：直接询问 LLM#

最简单的方案，是要求 LLM 从每一次运行中提取任务。理解单次运行的任务足够容易，即使是小型模型也能很好地完成。

问题在于成本。每天 100 万次 Agent 运行、每个用户轮次约 2,000 token，成本大约是每月 6,000 美元，并且会随流量线性增长。我们希望默认在每个 Agent 的每条 trace 中都提供任务，但以这个价格，我们无法承担。

尝试 #2：正则表达式#

想想你会如何在自己 Agent 的 trace 中寻找任务。你不会通读全部内容；你会直接跳到自己知道的位置：系统提醒之后、USER TASK: 那一行之后，或者轮次中的最后一条消息。

这就是一种模式，而模式无需在每次运行中都调用 LLM。它可以是一条正则表达式。因此，我们不再让 LLM 提取任务，而是每个 Agent 只询问它一次，让它写出能够定位任务的正则表达式；然后对后续每次运行应用该正则表达式，无需再次调用 LLM。

编写这样的正则表达式比阅读一个轮次更难，因此我们会并排向模型提供同一 Agent 的多个样本。其中保持不变的是脚手架；任务位于变化的内容中，模型负责判断究竟是哪一部分。有时正确答案只是“提取全部内容”——(?s)(.*)，因为许多 Agent 将任务作为一条裸消息发送。

对于上面的编程 Agent，正则表达式会提取 </system_reminder> 与 <user_info> 之间的任何内容：

这种方法在 Agent 发生变化之前效果良好。没有人会写完一个模板后永远不再修改，而模板变更可能以两种方式破坏正则表达式：

• **悄无声息地出错。**开发者开始在用户消息中附加文件：

  ...
  The checkout button still doesn't work on mobile.
  <attachments>
    - mobile-checkout.png (screenshot, 1.2 MB)
  </attachments>
  <user_info>
  ...
  

  正则表达式依然能够匹配，因此不会有任何报错；但现在每个提取出的“任务”末尾都多了一份附件列表。

• **明显地出错。**开发者将 <user_info> 移到第一条消息中，放在仓库状态旁边。现在任务之后不再有任何内容，正则表达式无法在系统提醒之后找到 <user_info>，于是每次运行返回的任务都为空。

无论哪种情况，问题都会从“这次运行的任务是什么？”变为“我们如何知道现有模式已经不再有效？”换言之：这次运行使用的是哪个模板版本，以及它何时发生了变化？

尝试 #3：解析结构#

正如示例所展示的，任务通常被某种标记包裹。Prompt 的编写目的是让模型理解清楚，因此通常具有可见结构——整个用户轮次往往由多个包含动态内容的区块组成。而且，这些区块很多时候是 XML 标签。

因此，第一个想法很简单：将 XML 标签视为 prompt 的结构。按其出现顺序找出用户轮次中的所有 XML 标签，将它们拼接并哈希，然后将结果视为模板版本。当开发者增加、删除或重命名标签时，版本就会改变，我们便根据最新样本生成新的正则表达式。

这能捕捉到上一节中的附件变更：新增的 <attachments> 标签改变了标签列表，因此版本发生变化，新版本会获得一条新的正则表达式：

它成本低廉，并且对许多 Agent 来说已经足够。但它会在两个方向上失效：

• **过于敏感。**XML 标签并不总是章节标记。浏览器 Agent 可能在用户轮次中包含页面交互元素（<button>、<input>、<a>），而编程 Agent 可能包含代码库中的 HTML。这样一来，签名几乎会在每次运行中变化，正则表达式需要每次重新生成，成本又回到了原来的水平。

• **过于迟钝。**许多模板根本不使用 XML：Agent 的上下文可能是纯文本、位于 Markdown 标题之下，或者使用开发者喜欢的任何其他标记。假设开发者改为以纯文本形式添加附件：

  The checkout button still doesn't work on mobile.
  Attachments:
  - mobile-checkout.png (screenshot, 1.2 MB)
  <user_info>
  ...
  

  标签列表完全没有变化，因此版本不会改变，旧的正则表达式会继续把附件作为任务的一部分返回。

核心问题在于：我们试图理解一个 prompt 的结构，而并不存在可供理解的通用结构。

尝试 #4：从大量运行中学习#

因此，我们不再解析结构，而是从运行本身学习模板。

只看单次运行时，你无法判断哪些内容是静态、哪些内容是动态；信息根本不足。但查看同一版本的大量运行后，答案就很明显：静态部分就是它们共同拥有的内容。

我们将每个用户轮次视为一系列行，并将版本定义为其所有轮次按相同顺序共同包含的行：即它们的最长公共子序列（LCS）。下面是编程 Agent 三次运行的结果，其中附件如前文所述以纯文本形式加入：

版本是这些公共行的哈希值。不需要对 XML、Markdown 或其他任何标记作出假设：动态行就是不同运行之间存在差异的行。任务几乎总是动态的，因为它会随运行而变化。LCS 只告诉我们一次运行属于哪个版本；在运行中找到任务，仍然是正则表达式的职责。

这同样能捕捉到欺骗了尝试 #3 的变更。Attachments: 在所有包含它的运行中都相同，因此它是静态行，添加它会改变版本。其下方的文件名则在不同运行中不同，所以它是动态内容，永远不会影响版本——这是正确的，因为它是内容而不是模板。

在实践中，我们会在一个近期运行窗口上计算：version = LCS(recent runs)。如果结果匹配某个已知版本，就复用该版本的正则表达式；如果不匹配，则是新版本，并为其生成新的正则表达式。当模板变化时，新版本的前几次运行仍会与旧版本共同存在于窗口中，因此它们的版本仅包含两者共有的部分。随着越来越多的新运行到来，窗口会在短暂延迟后自行收敛到新版本。在延迟期间，运行会继续使用旧版本的正则表达式。

不过，这里存在一个隐藏假设：近期运行都来自同一个版本。但如果开发者在进行 A/B 测试呢？

最终设计：Top-K 最近邻#

在 A/B 测试期间，运行会以 A B A B A B ……的形式到达。每个窗口都包含两个版本，因此 LCS 永远只保留二者的共同部分。开发者实际发布的版本中，至少有一个永远不会真正显现出来。

而 A/B 测试并非唯一情形。我们第一个示例中的编程 Agent 会接收用户发送的聊天轮次，也会接收来自 CI 集成的长自动化请求，例如“审查这个 pull request”。这就是两种不同的模板，全天交错出现。

**解决办法：**对于每个新运行，

1. 比较它与过去一小时内运行的相似度，依据是它们的用户轮次与该运行共享行的比例。
2. 选择最接近的 X%，例如 30%。如果窗口中存在其他 A 版本运行，那么 A 版本运行最接近的邻居也会是 A 版本运行。
3. 仅对这些运行执行 LCS。

因此，算法变为：version = LCS(X% closest of the last hour's runs)。

**数字的选择。**窗口是一个时间跨度，而不是运行数量：固定的“最近 100 次运行”在繁忙的 Agent 上可能全都来自同一个用户会话，这样该用户的名字就会看起来像静态内容；而一小时的流量几乎不会如此。窗口还会限制为每个 Agent 最多几百个不同的轮次，以保持排序成本低廉。K 使用比例而非数量，因此能随窗口规模变化：在 50/50 的 A/B 测试中，取 30% 也能让一次运行最接近的邻居来自其自身变体。

版本识别本身完全不需要 LLM，只需要哈希和 LCS。LLM 每个版本只介入一次：每条提取正则表达式都归属于一个 Agent 和一个模板版本；当新版本出现时，我们会根据该版本的新鲜样本生成其正则表达式。在正则表达式准备好之前，我们会回退为每次运行调用一次 LLM。结果是：LLM 成本主要随你修改 Agent 的频率增长，而不是随其流量增长。如今，我们每天在任务提取上花费约 10 美元；如果对每次运行都调用 LLM，则每天需要约 200 美元。

更多细节#

• **如何区分 Agent？**一条 trace 往往包含多个 Agent：主 Agent、它的子 Agent，以及负责总结对话或生成标题的小型辅助 Agent。我们通过它们 system prompt 的第一句话来区分 Agent，并为主 Agent 提取任务；主 Agent 的第一次 LLM 调用具有最多输入 token。
• **首轮与后续轮次。**后续轮次通常遵循与对话首轮不同的模板，因此会被分别进行版本识别。
• **消息边界。**我们使用一行标记来连接一个轮次中的消息，使正则表达式可以锚定一条消息在哪里结束。该标记会从正则表达式捕获的内容中移除。

仍然困难的问题#

仍然存在一些边缘情况。极长的轮次会在我们生成之前被截断

The first thing you usually want to know about an agent run is what it was asked to do. The answer is in the user's message, but rarely on its own. By the time the message reaches the model, the agent's harness has wrapped it in timestamps, environment state, system notices and user details, and every agent wraps it differently. This post is about how we find the task inside all that: cheaply, across millions of runs a day, and without asking developers where to look.

Why Tasks?#

Laminar processes millions of agent runs every day, and every one of them starts with a task. By task we mean the request as the user wrote it, verbatim, not a summary or a reconstruction of intent. It's one of the most useful things to know about a run. With the task in hand, you can:

• see at a glance what an agent was asked to do when you open its trace
• understand what your users actually ask for
• find which tasks an agent fails at
• build evals for different kinds of tasks from production data

The Challenge#

The obvious answer is to take the role: "user" messages from the agent's first LLM call. Sometimes that's enough: many agents send the task as a plain user message of its own.

But just as often, the task is buried among other context. One common reason is prompt caching: cache hits need a stable prefix, so many agents keep the system prompt fixed and move everything that changes per run into the user messages. The user turn ends up being a mix of:

• static scaffolding : instructions, section headers and markers
• dynamic request context : who sent the message, the pull request under review, a customer's profile
• dynamic environment state : the current time, open browser tabs, files on disk, the agent's todo list
• the task itself : usually the one part that's different in every run

For example, here is the first LLM call of a coding agent's run. After the system prompt, the user turn arrives as three separate user messages :

Only one line of this is the task: "The checkout button still doesn't work on mobile." The rest is scaffolding: the state of the repository, a system notice, who sent the message, and an instruction for the reply. And the scaffolding isn't even fixed, since the repository state changes on every turn.

We call these three messages together the user turn : the role: "user" messages that come after the agent's last reply. In a new conversation, that's all of them. In a follow-up, the LLM call also carries the earlier conversation, and the user turn is just the new messages since the agent last replied. The user turn is where we look for the task.

For the agent's developer, none of this is hard: they know exactly where the task is, and a custom UI renderer or a quick SQL query can pull it out. But that stops working once the agent changes, and developers change prompts every day. They add and remove sections, rename XML tags, and move context from one message to another. Every change can silently break a hand-written extraction. And we need to extract tasks for every agent on the platform, with no developer to ask.

TL;DR: an LLM writes a task extraction regex for each template, the format an agent wraps around every task, so most runs need no LLM call at all. When the format changes, we notice by comparing each run with similar recent runs, and write a new regex for the new format. The rest of this post walks through the attempts that led to this design.

Attempt #1 – Just Ask an LLM#

The simplest solution is to ask an LLM to extract the task from every run. Understanding the task of a single run is easy enough that even small models handle it well.

The problem is cost. At a million agent runs per day and about 2,000 tokens per user turn, that's roughly $6,000 a month, and it grows linearly with traffic. We want the task in every trace of every agent by default, and at that price we can't afford it.

Attempt #2 – Regexes#

Think about how you'd find the task in a trace of your own agent. You wouldn't read the whole thing; you'd jump straight to where you know it is: right after the system reminder, after the USER TASK: line, or the last message of the turn.

That's a pattern, and a pattern doesn't need an LLM on every run. It can be a regex. So instead of asking an LLM for the task, we ask it once per agent for a regex that finds the task, and then apply that regex to every following run without another LLM call.

Writing that regex is harder than reading one turn, so we give the model several samples from the same agent side by side. What stays the same between them is scaffolding; the task is somewhere in what changes, and the model decides which part it is. Sometimes the right answer is simply "take everything" ((?s)(.*)), since plenty of agents send the task as a bare message.

For the coding agent above, the regex takes whatever sits between </system_reminder> and <user_info>:

This works well until the agent changes. Nobody writes a template once and never touches it again, and a changed template can break a regex in two ways:

• Silently. The developer starts attaching files to the user's message:

  ...
  The checkout button still doesn't work on mobile.
  <attachments>
    - mobile-checkout.png (screenshot, 1.2 MB)
  </attachments>
  <user_info>
  ...
  

The regex still matches, so nothing fails, but every extracted "task" now ends with a list of attachments.

• Loudly. The developer moves <user_info> into the first message, next to the repository state. Now nothing follows the task, the regex finds no <user_info> after the system reminder, and every run comes back with no task at all.

Either way, the problem changes from "what's the task of this run?" to "how do we know that the pattern we have is no longer valid?" In other words: which version of the template does this run use, and when did it change?

Attempt #3 – Parsing the Structure#

As the examples show, the task is usually wrapped in some kind of marker. Prompts are written to be clear to a model, so they tend to have a visible structure – often the whole user turn is built from blocks with dynamic content inside. And very often, those blocks are XML tags.

So the first idea is simple: treat the XML tags as the prompt's structure. Find all of them in the user turn in the order they appear, join and hash them, and treat the result as the template version. When the developer adds, removes or renames a tag, the version changes, and we generate a new regex from the latest samples.

That catches the attachments change from the previous section: the new tag changes the list of tags, so the version changes, and the new version gets a new regex:

It's cheap, and for many agents it's enough. But it breaks in both directions:

• Too sensitive. XML tags are not always section markers. A browser agent may include the page's interactive elements (, , ) in its user turn, and a coding agent may include HTML from the codebase. The signature then changes on almost every run, the regex is regenerated every time, and the cost goes back up.

• Too blind. Many templates don't use XML at all: an agent may get its context as plain text, under Markdown headers, or with any other marker the developer likes. Say the developer had added the attachments as plain text instead:

  The checkout button still doesn't work on mobile.
  Attachments:
  - mobile-checkout.png (screenshot, 1.2 MB)
  <user_info>
  ...
  

The tag list is exactly what it was, so the version doesn't change, and the old regex keeps returning the attachments as part of the task.

The core problem is that we're trying to understand the structure of a prompt, and there is no universal structure to understand.

Attempt #4 – Learning from Many Runs#

So instead of parsing the structure, we learn the template from the runs themselves.

Looking at a single run, you can't tell what is static and what is dynamic; there's simply not enough information. But looking at many runs of the same version, it becomes obvious: the static part is whatever they all share.

We treat each user turn as a sequence of lines, and define a version as the lines that all its turns have in common, in the same order: their longest common subsequence (LCS). Here it is for three runs of our coding agent, with the attachments added as plain text, as above:

The version is a hash of those common lines. No assumptions about XML, Markdown or any other markers: a dynamic line is simply one that differs between runs. The task is almost always dynamic, since it differs from run to run. The LCS only tells us which version a run belongs to; finding the task in the run is still the regex's job.

This also catches the change that fooled Attempt #3. Attachments: is the same in every run that has it, so it is a static line, and adding it changes the version. The file name under it differs from run to run, so it's dynamic and never affects the version – which is right, because it's content, not template.

In practice, we compute this over a window of recent runs: version = LCS(recent runs). If the result matches a known version, we reuse its regex; if not, it's a new version and gets a new regex. When the template changes, the first runs of the new version still share the window with the old one, so their version is only what the two have in common. As more new runs arrive, the window resolves to the new version on its own, after a short lag. During the lag, runs keep the old version's regex.

This has a hidden assumption, though: that recent runs all come from the same version. But what if a developer runs an A/B test?

Final Design – Top-K Nearest Neighbors#

During an A/B test, runs arrive as A B A B A B …. Every window contains both versions, so the LCS is only what the two share, forever. At least one of the versions the developer actually shipped never shows up.

And A/B tests aren't the only case. The coding agent from our first example gets chat turns from its users, but also long automated requests from a CI integration, like "review this pull request". That's two different templates, interleaved all day.

The fix: for each new run,

1. Compare it with the runs of the last hour, by the share of lines their user turns have in common with it.
2. Select the closest X% of them, say 30%. If there are other A runs in the window, an A run's closest neighbors will be A runs.
3. Run the LCS only on those.

So the algorithm becomes version = LCS(X% closest of the last hour's runs).

Choosing the numbers. The window is a time span, not a count: a fixed "last 100 runs" could all come from one user session on a busy agent, and that user's name would look static; an hour of traffic almost never does. It's also capped at a couple hundred distinct turns per agent, so ranking it stays cheap. K is a share rather than a count, so it scales with the window: with 30%, a run's closest neighbors come from its own variant even in a 50/50 A/B test.

Versioning itself needs no LLM at all, only hashing and LCS. The LLM comes in once per version: each extraction regex belongs to an agent and a template version, and when a new version appears, we generate its regex from fresh samples of that version. Until it's ready, we fall back to an LLM per run. The result: LLM cost grows mostly with how often you change your agent, not with how much traffic it gets. Today we spend around $10 a day on task extraction, instead of the $200 a day it would take to call an LLM on every run.

A Few More Details#

• Which agent? A trace often holds several agents: the main one, its subagents, and small helpers that summarize the conversation or write a title. We tell agents apart by the first sentence of their system prompt, and extract the task for the main one, whose first LLM call has the most input tokens.
• First turns and follow-ups. Follow-up turns usually follow a different template than the first turn of a conversation, so they're versioned separately.
• Message boundaries. We join a turn's messages with a marker line, so a regex can anchor on where one message ends. The marker is stripped from whatever the regex captures.

What's Still Hard#

There are still edge cases. Very long turns are truncated before we generate a regex, so extraction can miss on them. Large context that barely changes between runs, or tasks that repeat verbatim like "Continue", can look static and split one template into extra versions, which costs a few more LLM calls. A variant smaller than the share, like a 10% rollout, still gets mixed in with the others. And a quiet agent needs a couple dozen runs before it gets a version.

We tried to keep such cases rare while keeping costs low. An explicit signal from the developer, like a CI step that tells Laminar when the template changed, would close the gap, but we didn't want to ask users for extra work.

Wrapping Up#

You can't learn a template by looking at one turn, but you can by looking at many: treat turns as lines, compare each one to its closest neighbors, and accept a short lag in exchange for a pipeline that corrects itself.

This is what powers task extraction in Laminar today. The agent's task is in every trace, and it's also a column you can query: agent_input on the traces table. One query in the SQL editor shows what your agents were asked to do, or which tasks end in errors. There's nothing to configure, and it keeps up as your agent changes.

Try Laminar and see what your agents are being asked to do.