在 Agent 中引入渐进式披露：速度提升 2.3 倍，Skyvern 3.0 架构实践

TL;DR - 我们以 Pi 为灵感从零重写了 Skyvern，使其速度提升了 2.3 倍，并在 Odysseys benchmark 上取得了 90.5% 的成绩。欢迎通过 开源版本 或 云服务版本 进行体验。

💡 回顾：什么是 Skyvern？ 这是一个开源框架，旨在帮助非技术团队通过 Prompt 提示词构建浏览器自动化流程。Skyvern 具备模型无关性（Model Agnostic），既可以使用 Claude 等闭源模型运行，也可以使用 Deepseek 等开源模型运行。我们已经协助数千家公司实现了各种场景的自动化，包括与医疗门户网站交互、获取发票或公用事业账单、填写政府表格、申请工作等。

RAG？你在说什么？

对于不熟悉 Agent 的人来说，RAG 代表检索增强生成（Retrieval Augmented Generation）。这是一种向 LLM 传递信息以便其做出更明智决策的方法。例如，在就某篇文档向 LLM 提问之前，你可以先解析该文档并将其输入给 LLM。你可以在这里阅读更多相关信息。

基于 RAG 的 LLM 框架运作方式：从知识库获取信息，将其转换为上下文，并传递给 LLM 以便做出决策

Skyvern 的第一个版本就是使用 RAG 构建的。我们会加载网页、截取屏幕截图并添加注释，将其浓缩为一个较小的元素列表，然后发送给 LLM 让其采取行动。

第一版 Skyvern 模仿了基于 RAG 的框架：从网站获取信息，将其转化为可操作元素，并询问 LLM 应采取什么操作

如果你能以 100% 的准确率识别网页上的每个可交互元素，这种方式效果非常好。我们原本以为自己最终能做到这一点。但我们错了。

现实是……网页环境极其混乱。准确识别网页上的每一个元素简直是一场噩梦。最初只是一个用于识别可交互元素的小工具，最终演变成了实质上构建一套全新的 DOM 解释器。

解析 Shadow DOM、iframe、select2 下拉框、Kendo UI 控件、Canvas 元素等等……这变成了一场无休止的战斗。

最糟糕的是，Agent 的性能并没有随着更优秀的 LLM 推出而提升。相反，它被这套 DOM 解析逻辑给卡死了。如果解析器未能正确识别某个元素，整个 Agent 运行就会彻底失败。

这是一个信号：我们需要给 LLM 更多自由。

给 LLM 自由，让它做想做的事

RAG 系统如果处理不慎，容易默认预处理层必须始终发生在任何 LLM 调用之前，而不让 LLM 自行决定何时需要这些信息。在以前 LLM 不擅长遵循指令的时代，这是一个好思路，因为你无法相信它们能在需要时可靠地提取信息；但自从 Opus 4.5 之后的模型发布以来，指令遵循能力已经不再是大问题。

我们注意到市场上出现了 2 个凸显这种思维转变的趋势：

1. 像 Claude Code 这样的代码编写 Agent 正在从 Embedding 风格的代码搜索转向类似 grep 的代码搜索，因为模型在使用基础原生工具按需获取上下文方面变得越来越出色
2. 像 Pi 这样的极简 Coding Agent 越来越受欢迎，Pi 的衍生项目如 Prime-intellect 通过调优测试框架并让 Agent 根据需要生成和执行 Python 代码，在 ARC-AGI-3 benchmark 上达到了 SOTA

于是我们问自己：同样的方法是否适用于浏览器 Agent？ 如果让 Agent 从所需的最小上下文开始，并赋予其按需拉取额外上下文的能力，结果会怎样？

在实践中，这意味着为 Agent 提供以下工具：

1. 使用模型生成的 JavaScript 扫描 HTML
2. 在网站上执行任何 rustwright 支持的操作，例如点击、输入文本、滚动、切换标签页等
3. 截取屏幕截图
4. 将运行标记为最终状态（成功，或附带原因的失败）

结果超出我们的预期

我们最初的假设是性能会提升，但代价是成本增加和处理速度变慢。因为新架构应该需要调用更多次 LLM 才能完成相同任务。

为了检验这个假设，我们将测试分为了两个分支：

1. 领先的公开浏览器 Agent 基准测试：通过 Odysseys 测试准确率
2. 覆盖约 100,000 次真实客户运行的线上 A/B 测试，用于测试速度和成本

Odysseys 基准测试结果

Skyvern 3.0 登顶了 Odysseys benchmark。它取得了 90.5% 的完美标准成功率（perfect rubric success rate），几乎完全符合我们的预期。但令我们惊喜的是，它的运行效率极高，平均仅需 65.4 步。

生产环境 A/B 测试结果

我们在大约 100,000 次运行中进行了生产环境 A/B 测试，并测量了影响：

生产环境 A/B 测试的结果令人意外。新架构竟然……更快、更便宜了？我们原本以为这两个维度中至少有一个会受到负面影响。

为什么会这样？深入探究后，我们发现了几处有趣的现象：

1. 截图使用率下降了 90%（占所有 LLM 调用的比例从 27% 降至 2.7%）。包含图像的 LLM 调用需要经过视觉编码器来协助 LLM“看懂”图像，而这往往比纯文本编码器慢 1.7 倍
2. Token 消耗量增加了 52%（单次运行从 188K tokens 增至 286K tokens）。平均轮次（即 LLM 调用次数）增加了 3.59 倍（6.4 轮次变为 23 轮次），但每轮消耗的 Token 减少了 59%（29K tokens 降至 12K tokens）
3. Prompt Cache 命中率提升了 195%（从 27% 提升至 79.7%），因为我们不再在每轮交互中都加载屏幕截图或完整 HTML，这意味着 Prompt 中有更大比例是可以被缓存的

尽管总 Token 使用量有所增加，但被缓存的 Token 享有 70% 的折扣，这解释了最终成本净下降的原因。

那么……浏览器 Agent 问题已经解决了吗？

答案仍然是：还没有。它们在成本、速度和准确率这 3 个维度上仍有提升空间。

我们的团队目前正在探索将类似 Jev 的“系统一模型”（system one models）应用于浏览器 Agent 的角色。当前这代模型依然存在很大局限性：64K 上下文窗口、无法处理图像，这会导致无法接受的精度损失，但我们预计这将在未来 6-12 个月内发生改变 🙂

快来试试 Skyvern 吧！

欢迎通过我们的 Skyvern 开源版 或 Skyvern 云服务 体验，并告诉我们你的想法！

TL;DR - We rewrote Skyvern from scratch usingPias inspiration, making it 2.3x faster and score 90.5% onOdysseys benchmark. Try it out the viaOpen SourceorCloud.

💡 Recap: What is Skyvern? It’s an open source framework that helps non-technical teams prompt to build browser automations. Skyvern is model agnostic, and can be run with closed models like Claude, or open ones like Deepseek. We have helped thousands of companies automate things like interacting with healthcare portals, fetching invoices or utility bills, filling out government forms, applying to jobs, and more.

RAG? What are you talking about?

For those unfamiliar with agents, RAG stands for Retrieval Augmented Generation. It’s a way to pass information to an LLM so they make smarter decisions. For example, before asking questions to an LLM about a document, you could parse that document and feed it into an LLM. You can read more about it here.

RAG-based LLM Frameworks work by fetching information from a Knowledge base, transforming it into context, and passing it to an LLM to make decisions on

The first version of Skyvern was built using RAG. We would load up a website, screenshot and annotate it, condense it into a smaller list of elements, and send it to an LLM to take actions on.

First version of Skyvern mimicked RAG based frameworks: It fetched information from the website, transformed it into actionable elements, and asked the LLM what action to take

This worked great if you could identify every interactable element on a website with 100% accuracy. We thought we could eventually get there. We were wrong.

Except.. the web is messy. Accurately identifying every element on a website proved to be a nightmare. What started off as a nice little utility to identify interactable elements essentially turned into building a new DOM interpreter.

Parsing things like Shadow DOMs, iframes, select2 dropdowns, Kendo UI widgets, Canvas elements, …. this became an endless battle.

The worst part was that agent performance did not improve with better LLMs. Instead, it was bottlenecked by this DOM parsing logic. If it failed to correctly identify an element, the entire agent run failed.

This is a sign that we need to give the LLM more freedom.

Give the LLM freedom to do what it wants

RAG systems, if you aren’t careful, can assume that the pre-processing layer always happens ahead of any LLM calls, and don’t let the LLM decide when it is necessary. This was a good idea when LLMs weren’t good at following instructions, which meant that you couldn’t trust them to reliably pull the information when needed, but instruction following hasn’t been a big problem since the models released after Opus 4.5.

We noticed 2 moves in the market that highlighted this change in thinking:

1. Coding agents like Claude code were moving away from embedding style code search to grep style code search because the models were increasingly getting better at using primitive tools to fetch the context they needed
2. Minimalist coding agents like Pi were becoming increasingly popular, with extensions of Pi like Prime-intellect hitting SOTA on the ARC-AGI-3 benchmark by tuning the harness and letting the agent generate and execute python code as needed

So we asked ourselves: would the same approach work for browser agents? What if the agent started with the minimum context required, and gave it the ability to pull additional context as needed?

In practice, it means giving the agent the following tools:

1. Scan the HTML with model-generated javascript
2. Take any rustwright-supported action on the website, for example Click, Type, Scroll, Switch tabs, etc
3. Take a screenshot
4. Mark the run as final (success, or failed with reason)

The results were better than we expected

Our hypothesis was that we would see an increase in performance, in exchange for added cost and a slower processing speed. The new architecture should result more calls to the LLM to get the same task done.

To test this hypothesis, we split the test up into two branches:

1. The leading public browser agent benchmark: Odysseys to test for accuracy
2. A live A/B test across ~100,000 customer runs to test for speed / cost

Odysseys benchmark results

Skyvern 3.0 topped the Odysseys benchmark. It reported a 90.5% perfect rubric success rate, pretty much exactly as we expected. What surprised us though was that it was highly efficient in its runs, at only 65.4 average steps.

Production A/B Test results

We ran the production A/B test on approx 100,000 runs and measured the impact

| Skyvern 2.0| Skyvern 3.0| Improvement
---|---|---|---
Mean run time| 593 s| 262s| 2.3x
Total cost per run| $0.039| $0.0302| -22.5%

The production A/B test results were surprising. The new architecture was.. faster and cheaper? We expected at least one of these dimensions to be a net negative

Why was that the case? Digging into it, we found a few interesting things:

1. Screenshot utilization dropped by 90% (27% — > 2.7% of all LLM calls). LLM calls with images go through a vision encoder to help the LLM “see” the image, which tend to be 1.7x slower than text-only encoders
2. **Token consumption increased by 52% (188K tokens / run — > 286K tokens per run) **Average number of turns (ie LLM Calls) increased by 3.59x (6.4 —> 23), but tokens per turn decreased by 59% (29K tokens —> 12K tokens)
3. **Prompt Cache hit rates increased by 195% (27% — > 79.7%) **because we stopped loading the screenshot or HTML on every turn, which meant a larger % of the prompt was cacheable

Despite the increased token usage, cached tokens carry a 70% discount, which explains the net reduction in cost.

So… are browser agents solved?

The answer is still no. They can still be improved across all 3 dimensions (cost, speed, accuracy).

Our team is now exploring the role of ”system one models” like Jev for browser agents. The current generation is still highly limited: 64K context window, no image processing, which causes unacceptable losses in accuracy, but we expect this to change over the next 6-12 months 🙂

Give Skyvern a try!

Run it via our Skyvern Open Source or Skyvern Cloud versions and let us know what you think!