AI News
商业产业 Hacker News Agent 2026-09-15T15:03 正文 ok 725 words

浏览器 Agent 的苦涩教训

英文标题：🔶 The bitter lesson of browser agents

Browser Use 在 2024 年推出时，GPT-4o 尚未针对计算机操作训练，因此团队必须预先定义页面状态以及点击、输入、滚动等固定动作。随着模型编程能力提升，系统先将固定动作替换为模型生成的代码，随后进一步让模型通过原始 CDP 自主决定观察什么页面信息。实测中，Hermes agent 将 12 个浏览器工具收敛为 `browser_exec` 后，Opus 4.8 和 Kimi K3 的平均 token 用量分别下降 60% 和 66%，且均完成全部 18 次任务运行。文章认为，开发浏览器 Agent 应复用经过验证的 Agent harness、暴露足够底层且简单的接口，并让模型自行选择观察与行动。
打开原文相关 3 · 机会 1
media29.1 KB media15.5 KB media36.8 KB media33.3 KB media41.1 KB media34.5 KB media42.9 KB media37.5 KB media40.2 KB media28.6 KB media45.7 KB media39.3 KB media38.8 KB media30.8 KB media13.6 KB media14.9 KB media330.6 KB media8.5 KB media67.7 KB media98.1 KB

中文

当我们在 2024 年 11 月发布 Browser Use 时，GPT-4o 尚未接受过计算机操作训练。它基于下一个 token 预测构建，无法可靠地理解自己在页面上能够做什么。我们不得不把这些能力明确告诉它。

随着模型的编程能力提升，我们从预定义的状态和动作转向了代码：起初由模型编写动作，之后再由模型编写观察。如今，它在一个现有的 Agent harness 中，通过原始 CDP 同时完成两者。

定义状态与动作空间

我们先定义状态：页面上有什么、哪些元素可点击。再定义动作空间：点击、输入、滚动。每一次交互都必须符合这些规则。

我们早期的 ReAct 风格循环：浏览器成为我们的页面状态，GPT-4o 进行推理，然后从固定动作菜单中选择点击、输入或滚动。浏览器返回下一次观察。我们早期的 ReAct 风格循环：浏览器成为我们的页面状态，GPT-4o 进行推理，然后从固定动作菜单中选择点击、输入或滚动。浏览器返回下一次观察。我们早期的循环：由我们决定模型能够看到和做到什么。

模型选择一个动作，我们执行它并返回下一个状态。但浏览器自动化是一场边缘情况的游戏。每一种例外都需要再补一个修复方案。

用代码替代固定动作

模型的编程能力正在显著提升。我们在 2025 年 9 月试验了 JavaScript 执行，并在 10 月试验了持久化 notebook。模型可以编写程序，而不是从动作中进行选择。

相同的循环，但动作菜单被模型编写的、包含 for 循环的程序取代。我们预定义的页面状态仍然保留。变量和辅助函数会在调用之间持续存在。相同的循环，但动作菜单被模型编写的、包含 for 循环的程序取代。我们预定义的页面状态仍然保留。变量和辅助函数会在调用之间持续存在。让它编写动作。页面状态仍由我们提供。代码仅用于示意。

绘制签名或循环处理评论，不再需要新增一个动作。变量可以持久保存，因此模型能够检查结果并继续执行。

Hermes agent 用由我们的 Browser Use CLI 驱动的 browser_exec 替换了 12 个浏览器工具。对于 Opus 4.8，平均 token 用量下降 60%；对于 Kimi K3，下降 66%。两个版本均完成了每个模型的 18/18 次运行（6 个任务，每个任务运行 3 次）。

预定义状态的局限

我们开放了动作，但仍然由我们决定模型可以看到什么。这成为了下一个问题。

Cookie 按钮可能就显示在屏幕上，却没有出现在我们发送的状态中。也许无障碍树没有暴露它，或者我们的处理流程把它漏掉了。无论哪种情况，模型都永远看不到这个按钮。

网站显示了一个 Accept 按钮，但我们提取的状态只列出 Article 和 Read more。缺失的按钮说明我们的处理流程已经成为瓶颈。网站显示了一个 Accept 按钮，但我们提取的状态只列出 Article 和 Read more。缺失的按钮说明我们的处理流程已经成为瓶颈。按钮就在那里。只是我们没能把它展示给模型。

某个 EHR 下拉菜单可能如下所示（简化的 DevTools 树）：

<ehr-app>
  #shadow-root (open)
    <iframe src="https://ui.example">
      #document <!-- other origin -->
        <dropdown-menu>
          #shadow-root (closed)
            <button>Choose...</button>

我们的状态启发式方法无法预见每一个网站。我们赋予了模型行动自由，但它仍然透过我们的假设来观察。

移除预定义状态

如果我们也让它编写观察呢？

我们让 LLM 直接访问 CDP，即 Chrome DevTools Protocol。没有必须提供的可点击元素列表。它可以编写代码来检查 DOM、截取屏幕截图，或查看某个 frame 的内部。

此前，我们的提取层向模型发送固定的可点击元素列表。现在，模型通过 CDP 命令自行选择观察内容，浏览器则返回它请求的视图。此前，我们的提取层向模型发送固定的可点击元素列表。现在，模型通过 CDP 命令自行选择观察内容，浏览器则返回它请求的视图。模型选择要检查什么。CDP 命令仅用于示意。

浏览器依然拥有状态。消失的是我们要求所有内容必须适配某一种预定义表示形式的限制。一张截图可能回答一个问题，一次 DOM 查询可能回答下一个问题。模型自行决定它需要看到什么。

为什么选择 CDP，而不是 Playwright？

CDP 是 Chrome 原生的控制协议。模型可以直接使用它。

使用 Playwright 时，Agent 代码需要通过 CDP 之上的额外一层才能抵达 Chrome。Browser Harness 将 CDP 直接暴露给 Agent 代码，在保留连接和运行时的同时移除了 Playwright 层。使用 Playwright 时，Agent 代码需要通过 CDP 之上的额外一层才能抵达 Chrome。Browser Harness 将 CDP 直接暴露给 Agent 代码，在保留连接和运行时的同时移除了 Playwright 层。对于 Chrome，CDP 让模型能够访问 Playwright 浏览器 API 之下的能力。

Playwright 会增加一些假设，例如其定位器无法看到关闭的 shadow root。CDP 可以直接检查这些 root。Playwright 同样暴露 CDP；我们只是将直接访问设为了默认选项。

一个简单的浏览器 harness

Pi、Codex 和 OpenCode 已经能够处理 Agent 循环。编程 Agent 已有数百万用户尝试各种类型的工作。这种测试覆盖正是我们应当复用其 harness 的原因。

一个持续进行 QA 五小时的 Agent，不应因为循环中的 bug 而失败。我们曾遇到一次压缩：它产生了大量上下文，以至于立刻再次触发压缩。我们的客户不应该被迫发现这类 bug。

我们新增了一个简单的浏览器工具：持久化代码，并将文本、截图和错误返回给模型。

Pi、Codex 或 OpenCode 通过 browser_exec 交换代码和结果，browser_exec 则经由 CDP 与 Chrome 通信。Pi、Codex 或 OpenCode 通过 browser_exec 交换代码和结果，browser_exec 则经由 CDP 与 Chrome 通信。一个 CLI 或工具调用将 Agent 连接至 CDP。

从我们在 2026 年 1 月推出的 Agent SDK，到 4 月推出的 Browser Harness，这成为了我们的发展方向。Browser Harness、BrowserCode 和 Browser Use Pi 采取了不同的方法。它们都让模型选择看什么、做什么。

这里真正的苦涩教训是什么？

Sutton 的苦涩教训倾向于那些能够随着更多计算资源而持续改进的通用方法。对我们而言，这意味着：

  • 复用经过验证的 Agent harness。
  • 暴露模型能够很好使用的、最简单的底层接口。
  • 让模型自行选择其观察和动作。

浏览器、计算机、文本、音乐：同一条原则都适用。随着模型变化，重新思考 harness。

English / 原文

When we launched Browser Use in November 2024, GPT-4o wasn't trained for computer use. Built on next-token prediction, it didn't reliably understand what it could do on a page. We had to spell it out.

As models got better at coding, we moved from predefined state and actions to code: first the model wrote its actions, then its observations. Now it does both through raw CDP, inside an existing agent harness.

Defining state and action space

We defined the state: here's what's on the page, here's what's clickable. Then the action space: click, type, scroll. Every interaction had to fit those rules.

Our early ReAct-style loop: the browser becomes our page state, GPT-4o reasons, then selects click, type, or scroll from our fixed action menu. The browser returns the next observation.Our early ReAct-style loop: the browser becomes our page state, GPT-4o reasons, then selects click, type, or scroll from our fixed action menu. The browser returns the next observation.Our early loop: we chose what the model could see and do.

The model picked an action, we ran it and sent back the next state. But browser automation is a game of edge cases. Every exception needed another fix.

Replacing fixed actions with code

Models were getting much better at coding. We experimented with JavaScript execution in September 2025 and persistent notebooks in October. The model could write a program instead of picking actions.

The same loop, but the action menu is replaced by a model-written program with a for loop. Our predefined page state remains. Variables and helpers persist between calls.The same loop, but the action menu is replaced by a model-written program with a for loop. Our predefined page state remains. Variables and helpers persist between calls.Let it write the actions. Still our page state. Code is illustrative.

Drawing a signature or looping through comments no longer needed a new action. Variables persisted, so the model could inspect results and keep going.

Hermes agent replaced 12 browser tools with browser_exec, powered by our Browser Use CLI. Mean token use fell 60% for Opus 4.8 and 66% for Kimi K3. Both versions solved 18/18 runs per model (six tasks, three runs).

The limits of predefined state

We'd opened up the actions, but we still decided what the model could see. That became the next problem.

A cookie button can be right there on the screen and missing from the state we send. Maybe the accessibility tree doesn't expose it, or our processing drops it. Either way, the model never sees the button.

The website shows an Accept button, but our extracted state lists only Article and Read more. The missing button shows how our processing became the bottleneck.The website shows an Accept button, but our extracted state lists only Article and Read more. The missing button shows how our processing became the bottleneck.The button is there. We just failed to show it.

An EHR dropdown might look like this (simplified DevTools tree):

<ehr-app>
  #shadow-root (open)
    <iframe src="https://ui.example">
      #document <!-- other origin -->
        <dropdown-menu>
          #shadow-root (closed)
            <button>Choose...</button>

Our state heuristics couldn't anticipate every website. We'd given the model freedom to act, but it was still looking through our assumptions.

Removing predefined state

What if we let it write the observations, too?

We gave the LLM direct access to CDP, the Chrome DevTools Protocol. No mandatory list of clickable elements. It could write code to inspect the DOM, take a screenshot or look inside one frame.

Before, our extraction layer sent the model a fixed list of clickable elements. Now the model chooses its observations through CDP commands, and the browser returns the view it asked for.Before, our extraction layer sent the model a fixed list of clickable elements. Now the model chooses its observations through CDP commands, and the browser returns the view it asked for.The model chooses what to inspect. CDP commands are illustrative.

The browser still has state. What disappears is our requirement that everything fit one predefined representation. A screenshot might answer one question. A DOM query might answer the next. The model decides what it needs to see.

Why CDP, not Playwright?

CDP is Chrome's native control protocol. The model can use it directly.

With Playwright, agent code reaches Chrome through an extra layer above CDP. Browser Harness exposes CDP directly to the agent's code, removing the Playwright layer while retaining the connection and runtime.With Playwright, agent code reaches Chrome through an extra layer above CDP. Browser Harness exposes CDP directly to the agent's code, removing the Playwright layer while retaining the connection and runtime.For Chrome, CDP gives the model access beneath Playwright's browser API.

Playwright adds assumptions, such as locators that cannot see closed shadow roots. CDP can inspect those roots directly. Playwright also exposes CDP; we made direct access the default.

A simple browser harness

Pi, Codex and OpenCode already handle the agent loop. Coding agents have millions of users trying all sorts of work. That testing is a reason to reuse their harnesses.

An agent five hours into QA shouldn't fail because of a bug in the loop. We once hit compaction that produced so much context it immediately triggered compaction again. Our customers shouldn't have to discover those bugs.

We add a simple browser tool: persistent code, with text, screenshots and errors coming back to the model.

Pi, Codex or OpenCode exchanges code and results with browser_exec, which communicates with Chrome through CDP.Pi, Codex or OpenCode exchanges code and results with browser_exec, which communicates with Chrome through CDP.A CLI or tool call connects the agent to CDP.

From our Agent SDK in January 2026 to Browser Harness in April, this became the direction. Browser Harness, BrowserCode and Browser Use Pi take different approaches. All let the model choose what to see and do.

What is actually the bitter lesson here?

Sutton's bitter lesson favors general methods that improve with more computation. For us:

  • Reuse a proven agent harness.
  • Expose the simplest underlying interface the model can use well.
  • Let the model choose its observations and actions.

Browsers, computers, text, music: the same principle applies. As models change, rethink the harness.