AI News
商业产业 Hacker News AI 2026-10-06T16:50 正文 ok 1969 words

我们从构建 Agent 运行沙箱中学到了什么

英文标题：🔶 What we learned from building the sandboxes our agents run in

QA Wolf 最初将 AI Agent 基础设施设计为短生命周期、可重试的云端任务，但发现 Agent 会等待用户、操作真实系统并持续保存进行中的工作，因此更像一位同事而非一次作业。该公司随后为每个 Agent 会话提供独立的云端机器、真实代码仓库、Shell、浏览器和开发工具，并通过预启动机器池降低冷启动延迟。为处理重复执行、状态丢失和潜在恶意命令等问题，系统采用单次执行、持久化自动保存、最小权限和默认拒绝等机制。该架构使 Agent 可并行拆分大型 QA 任务：单项请求最多可启动 50 个独立 Agent，而无需专门的内部协调 API。
打开原文相关 3 · 机会 1
media5.6 KB media464 B media226 B media734 B media581 B media1.6 KB media222 B media1.6 KB media5.6 KB media71.7 KB

中文

我们构建支撑 QA Wolf AI Agent 的基础设施时，采用了云端最常见的构建方式：将其作为任务运行，启动、完成工作、退出并被清理。事实证明，这完全是错误的模型。

Agent 会话的行为不像一个任务，更像一位同事。它会等待人类回复，会接触真实系统，还会执行事先无人审查过的命令。

我们第一版 Agent 只有一小组专门构建的工具：一个用于读取文件，一个用于搜索，一个用于运行测试。每增加一种能力，就意味着增加一个工具；而由于没有真正的磁盘，我们最终不得不构建一个模拟文件系统和一个模拟 bash，才能让它运行命令。它将大部分精力花在绕过这些缺口上，而每当我们希望它完成新事情时，都得手工再造一个工具。

与此同时，我们真正想完成的工作——让一个团队的端到端测试持续保持通过——本来就是 QA 工程师在普通笔记本电脑上已经会做的事情。举例来说，如果某个 PR 将“Place order”按钮重命名为“Complete purchase”，导致结账测试失败，工程师会阅读 PR，用 grep 在测试中搜索旧标签，打开应用查看新按钮，修复选择器，重新运行测试并提交修复。在这一过程中所使用的每种工具（git、Shell、浏览器和编辑器），任何工程师的机器上本来就都有。

我们曾尝试给 Agent 提供工具，但它真正需要的是一台电脑。

Agent 要成为有效 QA 工程师所需的每一种工具，笔记本电脑上本来就已经具备，而且在那里运行得更好。我们的文件读取器只是更差的 cat，搜索工具只是更差的 grep。我们一直在以一次一个命令的方式重建一台电脑。因此，我们停下了这件事，转而为每个 Agent 会话提供一台独立隔离的云端机器，其中包括：

  • 受版本控制的真实测试套件检出副本
  • 带有 git、gh、node、jq、prettier、Python 和 QA Wolf CLI 的 Shell
  • 在工作需要时提供的真实浏览器，并且你可以实时观看

而且，由于这台机器只属于一个会话、不会服务其他任何对象，一名 Agent 运行重负载任务不会拖慢其他客户；发生崩溃时，也只会影响发生故障的会话。

摒弃传统云端构建方式

在大多数云系统中，有工作时才启动机器，任务完成后立即关闭，因为机器运行的每一分钟都要付费，而典型任务并不需要保留上一次运行遗留的任何内容。

但在处理 Agent 时，这种方式并不真正适用，因为 Agent 会话并没有一个明确的“完成”时刻。它会暂停等待用户，然后在用户回复后继续，并始终保留进行中的工作。如果在这些暂停期间关闭机器，下一条消息就得等待新机器启动，或者进行中的工作会随旧机器一同消失。

我们不得不摒弃许多“正确且规范的工作方式”：

  • 冷启动。 对于普通任务，机器启动稍慢一些通常没人会注意到。对 Agent 而言，用户往往刚刚要求它做某件事，并正在等待回复。我们每天有超过 6,000 台机器服务于这类实时会话。在 Slack 和 GitHub 讨论串中，我们过去会在 Agent 回复一分钟后关闭其机器。但人们会开会、上洗手间，也会不知不觉沉迷于 YouTube 视频。因此，当他们终于回复时，往往要等待 25 秒甚至更久才能收到答复。
  • 重试。 普通任务失败时，你会再次运行它，因为运行两次通常不会造成损害。但 Agent 会产出内容，例如缺陷报告以及 Slack 和 GitHub 讨论串中的回复。如果机器在 Agent 已提交缺陷报告后宕机，而我们重试该任务，你的团队就会收到两份相同的缺陷报告。
  • 无状态任务。 普通任务不会将任何内容从一次运行带到下一次，因此任何机器都一样，丢失一台机器没有代价。对于 Agent 会话，机器本身就是工作：其中有你的测试检出副本、你和 Agent 正在编辑到一半的内容、停留在某个页面上的浏览器。在会话中途丢失它，失去的不是一台机器，而是工作成果。
  • 可信代码。 普通任务运行的是由某人编写、审查并部署的代码。Agent 则会根据其读取到的任何内容——被测应用、PR 描述、团队配置文件——自行编写并执行命令。因此，外部贡献者可以提交一个 PR，其描述要求接手该工单的人运行 cat .env，并将测试凭据粘贴到评论中；而拥有 Shell 和 GitHub Token 的 Agent 可能真的会这样做。

这四个问题中的每一个，都需要单独的解决方案。

我们是如何构建的

在你准备好之前就已就绪的机器

一台全新的机器最多需要 30 秒才能就绪，因此我们不会在会话打开时才启动机器。我们维护一个提前启动的机器池，并让它随一天内的流量增长和缩减。分配一台机器只需要几毫秒。在你向 Agent 发送第一条消息之前，系统就已经为你分配好机器。

我们不会对浏览器做同样的预启动。许多 Agent 工作根本不需要浏览器，因为规划测试覆盖、阅读代码和回答问题都在终端中完成。因此，只有当 Agent 想验证自己的工作时，例如重新运行它刚刚修复的测试，会话才会启动浏览器。在我们典型一天运行的数千个实时会话中，只有数百个会启动浏览器。

一旦会话获得机器，它就会在对话暂停期间保留该机器。无论你在哪里与 Agent 对话——Slack 讨论串、GitHub 讨论串或其他界面——每次回复后，它的机器都会继续运行五分钟；同时，对任何一台机器的存活时长设有硬性上限。这覆盖了大多数真实的来回交互。

只执行一次，以及对“完成”更严格的定义

自主工作的 Agent 只有一次尝试机会。假设它调查了一项失败的测试，判断失败是一个真实 Bug，在你的问题追踪器中提交了缺陷报告，然后机器在它能够将链接发到 Slack 之前宕机。任务失败了，但缺陷报告已经在你的追踪器中。再次运行后，你的团队会收到第二份报告。

我们本可以尝试从中断处恢复，但系统没有记录哪些步骤已经落地；而且对于其中许多步骤，也没有可靠的方法能在事后查询。我们选择的权衡是：当基础设施在这些任务底层发生故障时，让故障以你可见的方式发生，而不是悄悄地把工作做两次。这不是边缘情况，而是常见路径。在一个繁忙的工作日，我们的 Agent 现在会为数十个客户团队处理超过 1,300 个 Slack 和 GitHub 讨论串，没有人类在旁引导。

这刻意背离了我们自己的测试运行器：测试运行器会将失败的测试最多重试三次，才认定它是真正的失败。重新运行一次测试能够告诉你一些信息；重新运行一个 Agent，只会多出第二份缺陷报告。

比机器存活更久的工作

在普通云系统中，你不会保护进行中的工作，因为根本没有这种工作。机器上的所有内容，都是存储在其他地方的某个内容的副本，因此机器宕机后，只需启动另一台机器并再次运行任务。

但我们两点都做不到。机器保存着你和 Agent 正在编辑到一半内容的唯一副本，而且如上所述，我们不会重新运行 Agent。因此，我们必须让工作比机器活得更久。未保存的更改会在静默几秒钟后自动保存；当你持续输入时，至少每 30 秒保存一次，保存到不属于该机器的持久化存储中。当一台机器消失后，下一台机器会从那里继续，而不是从头开始。如果某项更改无法被干净地恢复，我们会保留你最近保存的代码，而不是猜测如何合并，因此最坏情况只是丢失几秒钟的输入内容。

我们不试图识别恶意命令

为每个会话提供独立机器，可以让不同会话彼此隔离。但这无法保护一个会话免受其自身 Agent 的影响——Agent 能够造成破坏的机器，恰恰保存着你的测试代码和凭据。

我们不试图区分一条合法命令和一条恶意命令。我们假定任何命令都可能具有敌意，并限制它们能够访问的范围：

  • 最小访问权限。 Agent 可以修改你的测试代码和它自己的临时空间。其他所有内容均为只读，或根本不存在。
  • 机密信息保持私密。 测试需要密码和 API Key，因此 Agent 的工具可以使用它们。它们会私密地传递给每条命令，而不是通过命令行传递，因此永远不会出现在进程列表、日志或错误消息中。
  • 没有云凭据。 Agent 无法访问其运行所在云端机器的凭据。
  • 只有你的权限，绝不更多。 在实时会话中，Agent 以你的身份行动，并且只拥有你在该工作区中的确切权限。
  • 默认拒绝。 如果无法完成上述任一项设置，会话就不会启动。不会退回到无保护状态下运行。

它为我们带来了什么

有些请求对一个 Agent 来说太大了。“修复昨晚运行中所有失败的测试”可能意味着 20 项彼此无关的修复，因此 Agent 会把工作拆分为独立部分，并为每一部分启动一个单独的 Agent，最多可同时启动 50 个。每个 Agent 都有自己的机器、自己的浏览器和自己的分支，因此它们不会相互干扰。

我们没有为此构建专门的内部 API。Agent 会写一份简短说明，然后从其 Shell 中运行公开的 QA Wolf CLI，这也是任何客户都可以运行的同一条命令：

qawolf --json agent send "$(cat /tmp/brief.md)"

随后，它会做你在自己的终端中运行少量 Agent 时会做的事：保留每个会话的链接，检查它们的状态，并回答它们的问题。由于每个 Agent 都有自己的机器，同时运行数十个 Agent 只需要更多机器，而不需要协调层。

这些都不是什么新东西。预启动机器、最小权限和自动保存已经存在多年，这四项修复中没有一项特别巧妙。真正花了我们一些时间的是意识到：我们的默认做法有一个共同前提——它们都假设另一端没有人在等待。没有人在等待回复，没有人的半成品工作放在机器上，也没有人的凭据被任务持有。

我们并没有为了修复这一点而放弃一次性基础设施。机器仍然是一次性的：它从池中启动，按计时器停止，我们不在乎你拿到的是哪一台。我们从机器上移走的是会话。提前启动、将工作保存到别处、拒绝将任务运行两次、假定每一条命令都具有敌意：这四种方式使机器可以低成本地丢失，同时让会话得以存活。

一旦我们不再将 Agent 会话视为任务，而是开始思考 QA 工程师需要从自己的笔记本电脑上获得什么，大部分设计都从这个问题中自然推导出来。

每个 QA Wolf Agent 会话都运行在这样的机器上：无论你是在 QA Wolf 应用中、Slack 讨论串中启动它，还是通过 Claude 或 ChatGPT 配合 QA Wolf MCP 启动它。连接后，要求 Agent 修复一个失败的测试。

English / 原文

We built the infrastructure that underpins QA Wolf's AI agents the way you build most things in the cloud: as jobs that start, do the work, exit and get cleaned up. Turns out, that was entirely the wrong model.

An agent session doesn't behave like a job. It behaves more like a coworker. It waits on people, it touches real systems, and it runs commands nobody reviewed ahead of time.

The first version of our agent got a short list of purpose-built tools: one to read a file, one to search, one to run a test. Every new ability meant another tool, and without a real disk we ended up building a pretend file system and a pretend bash just so it could run a command. It spent most of its effort working around the gaps, and every new thing we wanted it to do meant hand-building another tool.

Meanwhile, the job we wanted done — keeping a team's end-to-end tests green — is one QA engineers already do on an ordinary laptop. When a PR renames the "Place order" button to "Complete purchase" and breaks the checkout test, for example, an engineer reads the PR, greps the tests for the old label, opens the app to look at the new button, fixes the selector, reruns the test and commits the fix. Every tool they use along the way (git, a shell, a browser and an editor) is already on any engineer's machine.

We tried giving the agent tools but it needed a computer.

Every tool the agent needed to be an effective QA engineer was already on a laptop, and worked better there. Our file reader was a worse cat. Our search tool was a worse grep. We'd been rebuilding a computer one command at a time. So we stopped, and gave every agent session its own isolated machine in the cloud, with:

  • a real checkout of the test suite, under version control
  • a shell with git, gh, node, jq, prettier, Python and the QA Wolf CLI
  • a real browser when the work calls for one, which you can watch live

And because that machine belongs to one session and nothing else, one agent running something heavy can't slow down another customer, and a crash only takes down the session it happened in.

Unlearning the way you build for the cloud

In most cloud systems, you boot a machine when there's work to do and shut it down the moment the job is done, since you pay for every minute a machine runs and a typical job doesn't need anything left over from the last one.

That doesn't really work when you're dealing with agents, because an agent session doesn't have a clean "done." It pauses to wait on a person, picks back up when they reply, and holds work in progress the whole time. Shut the machine down during one of those pauses and the next message waits for a new machine to boot, or the work in progress goes down with the old one.

We had to unlearn so many of the Right And Proper Ways Of Working:

  • Cold starts. With a normal job, nobody notices if a machine takes a while to boot. With an agent, someone has usually just asked it to do something and is watching for the reply. More than 6,000 of our daily machines go to live sessions like that. In Slack and GitHub threads, we used to shut down an agent's machine a minute after it replied. But people have meetings and use the bathroom and get sucked into YouTube videos. So when they did get around to responding, they’d wait 25 seconds or more for a reply.
  • Retries. When a normal job fails, you run it again, because running it twice does no harm. But an agent produces things, like bug reports and replies to threads in Slack and GitHub. If the machine dies after the agent has filed a bug report and we retry the job, your team gets the same bug report twice.
  • Stateless jobs. A normal job carries nothing over from one run to the next, so any machine is as good as any other and losing one costs nothing. For an agent session, the machine is the work: a checkout of your tests, the edits you and the agent are halfway through, a browser sitting on some page. Lose it mid-session and you haven't lost a machine, you've lost the work.
  • Trusted code. A normal job runs code someone wrote, reviewed and deployed. An agent writes its own commands as it goes, based on whatever it reads: the app under test, PR descriptions, a team's config files. So an outside contributor can open a PR whose description asks whoever picks up the ticket to cat .env and paste the test credentials into a comment — and the agent, holding a shell and a GitHub token, could do it.

Each of those four problems needed its own fix.

How we built it

Machines that are ready before you are

A fresh machine takes up to 30 seconds to get ready, so we don't start one when a session opens. We keep a pool of machines booted ahead of demand, and it grows and shrinks with traffic through the day. Handing one over takes a few milliseconds. By the time you send your first message to the agent, you've already been assigned a machine.

We don't do the same for browsers. A lot of agent work never needs one, since planning coverage, reading code and answering questions all happen in a terminal. So a session only starts a browser when the agent wants to check its work, like rerunning a test it just fixed. Of the thousands of live sessions we run on a typical day, only a few hundred ever start one.

Once a session has a machine, it keeps it through the pauses in a conversation. Wherever you're talking to the agent, whether that's a Slack thread, a GitHub thread or another surface, its machine stays up for five minutes after each reply, with a hard cap on how long any one machine can live. That covers most real back-and-forth.

One attempt, and a stricter definition of done

An agent working on its own gets one attempt. Say it investigates a failed test, decides the failure is a real bug, files a bug report in your tracker, and then the machine dies before it can post the link in Slack. The job failed, but the bug report is already in your tracker. Run it again and your team gets a second one.

We could try to resume where it left off, but nothing recorded which steps already landed, and for a lot of them there's no reliable way to ask after the fact. That's the tradeoff we picked: when our infrastructure fails underneath one of these jobs, it fails where you can see it, instead of quietly doing the work twice. And this is the common path, not an edge case. On a busy weekday, our agents now work through more than 1,300 Slack and GitHub threads for dozens of customer teams, with nobody steering.

It's a deliberate break from our own test runner, which retries a failing test up to three times before calling it a real failure. Rerunning a test tells you something. Rerunning an agent just adds a second bug report.

Work that outlives the machine

In a normal cloud system you don't protect work in progress, because there isn't any. Whatever's on the machine is a copy of something that lives somewhere else, so when the machine dies you start another one and run the job again.

We don't get either of those. The machine holds the only copy of the edits you and the agent are halfway through, and as above, we don't rerun agents. So we had to make the work outlive the machine. Unsaved changes are saved after a few seconds of quiet, and at least every 30 seconds while you're typing, to persistent storage the machine doesn't own, and when a machine goes away the next one picks up from there instead of from scratch. If a change can't be restored cleanly, we keep your latest saved code rather than guess at a merge, so the worst case is losing a few seconds of typing.

We don't try to spot a bad command

Giving every session its own machine keeps sessions away from each other. It doesn't protect a session from its own agent — the machine the agent can do damage on is the one holding your test code and your credentials.

We don't try to tell a legitimate command from a malicious one. We assume any command could be hostile and limit what all of them can reach:

  • Least access. The agent can change your test code and its own scratch space. Everything else is read-only or isn't there.
  • Secrets stay private. Tests need passwords and API keys, so the agent's tools can use them. They're passed to each command privately rather than on the command line, so they never show up in a process list, a log or an error message.
  • No cloud credentials. The agent can't reach the credentials of the cloud machine it runs on.
  • Your permissions, nothing more. In a live session, the agent acts as you, with exactly your permissions in that workspace.
  • Fail closed. If any of this can't be set up, the session doesn't start. There's no fallback to running unprotected.

What it bought us

Some requests are too big for one agent. "Fix every failing test from last night's run" might mean twenty unrelated fixes, so the agent splits the work into independent pieces and starts a separate agent for each one, up to 50 of them. Each gets its own machine, its own browser and its own branch, so they can't trip over each other.

We didn't build a special internal API for this. The agent writes a short brief and runs the public QA Wolf CLI from its shell, the same command any customer can run:

qawolf --json agent send "$(cat /tmp/brief.md)"

Then it does what you'd do running a handful of agents from your own terminal. It keeps the link to each session, checks in on them and answers their questions. Since every agent has its own machine, running dozens at once only needs more machines, not a coordination layer.

None of that is new. Pre-booted machines, least privilege and autosave have all been around for years, and not one of the four fixes is clever. What took us a while was noticing what our defaults had in common: every one of them assumed nobody was on the other end. Nobody waiting on a reply, nobody whose half-finished work was sitting on the machine, nobody whose credentials the job was holding.

We didn't abandon disposable infrastructure to fix that. The machine is still disposable — it boots from a pool, it dies on a timer, we don't care which one you get. What we moved off the machine is the session. Booting before you ask, saving the work elsewhere, refusing to run the job twice, assuming every command is hostile: four ways of keeping the machine cheap to lose while the session survives.

Once we stopped treating agent sessions like jobs and started asking what a QA engineer needs from their own laptop, most of the design fell out of that question.

Every QA Wolf agent session runs on one of these machines, whether you start it in the QA Wolf app, in a Slack thread, or from Claude or ChatGPT with QA Wolf MCP. Connect it and ask an agent to fix a failing test.