如何构建 AI 软件工厂：能够创建、审查并合并 PR 的 Agent

TL;DR

AI 软件工厂由五个阶段构成，每个阶段都有一道关卡。Agent 是其中成本最低的部分。

阶段	决定什么	已公开案例
任务接入	哪些工作值得启动	Sentry 的 Seer会先为每个新进入的问题评估可操作性
隔离	Agent 在哪里运行才能避免冲突	Stripe 可在约 10 秒内启动预热的 devbox
工具	Agent 能访问什么	Stripe 的 Toolshed 通过 MCP 暴露约 500 个内部工具
验证	变更是否正确	Spotify 的 LLM 裁判会否决约 25% 的 Agent 会话
合并关卡	谁承担责任	Faire 要求 Agent 创建的 PR 必须经过两名人工审查者审核

简而言之：所有成功做到这一点的公司，都是先建立关卡，再部署 Agent 集群。Spotify 的 Fleetshift 于 2023 年上线，比它拥有可放入其中的 Agent 早了两年。生成能力能够随着投入而扩张，审查能力却不能，这种不对称正是整个设计问题的核心。

**Firecrawl 的定位。**Agent 需要代码仓库中并未携带的实时 Web 上下文。面向开放互联网使用搜索 + 抓取，面向代码使用经整理的开发者索引，二者均位于同一个 MCP 模块之后，并且每次抓取都启用提示注入检测。

---

2026 年 1 月 6 日，Stephen Toub 在 3.5 万英尺高空通过手机创建了 9 个 pull request，其中 7 个被合并。他参与 dotnet/runtime 项目，并写下了这段经历带给他的结论：

AI 改变了代码生产的经济学。一名具备良好判断力、手持手机的人，能够生成 PR 的速度会超过一个团队审查它们的速度。

这句话就是全文的主题。现在，一名使用编码 Agent 的工程师坐在飞机座位上，就能让整个团队的审查能力达到饱和。值得关注的问题，已经不再是如何让 Agent 编写代码，而是如何消化其产出。

AI 软件工厂是企业逐渐趋同的答案，它将自主编码 Agent 从演示产品转化为团队能够吸收的实际吞吐量。本指南将其拆分为五个阶段，分别介绍每一阶段背后的公开架构及其构建配置。若想先了解 AI Agent 如何推理、调用工具及获取 Web 上下文，可以先阅读该入门文章，再回到这里了解编码 Agent 集群究竟如何被交付上线。

假设你已经选定了 Agent；我们已在最佳 AI 编码 Agent盘点中介绍过这一环节。软件工厂指的是围绕 Agent 的一切。

什么是 AI 软件工厂？

AI 软件工厂是围绕编码 Agent 构建的系统，而不是 Agent 本身。工作从队列进入，Agent 在隔离工作区中运行，验证自动完成，人工位于一个明确的合并关卡上。它也被称为 Agentic 软件工厂。

一个有用的区分是 Agent 与软件工厂之间的区别。在笔记本电脑上运行编码 Agent，只是在使用 Agent：你选择任务、看着它工作、阅读 diff、执行合并。除了打字以外，其余工作仍然由你完成，而你的注意力就是上限。

软件工厂则将这些步骤迁移到基础设施中。没有人决定 Agent 要接手哪个问题，因为接入规则会做出决定；没有人设置工作区，因为隔离环境会被自动配置；没有人检查变更能否编译，因为在人类介入之前，验证就已经运行。人只会在流程末端出现，处理那些需要承担责任的决策。

Addy Osmani 在一篇文章中更简练地概括道：

软件工厂是在大规模利用循环。

他所说的循环，正是我们在循环工程指南中讨论的机制：Agent 运行、检查自己的工作、再次运行，直到验证器判定停止。软件工厂则是让大量这类循环同时运行、且无需任何人盯守的机器。

真正的软件工厂与一堆脚本之间，有三个特征将二者区分开来：

• **它由队列驱动，而不是由提示词驱动。**工作来自 issue、告警或 Slack 频道，系统决定哪些任务值得启动。没有人在手动输入提示词。
• **环境是一次性的。**每个 Agent 都拥有一个可以被销毁的干净工作区，因此一次失败的运行无需付出额外代价，并行运行也无法相互破坏。
• **验证先于审查运行。**当一个 diff 到达人类面前时，它已经完成编译、通过测试，并经过范围检查。

如果缺少第三点，你构建的就不是软件工厂，而是一台生成审查工作速度快于你消化速度的机器。这正是本文余下部分试图避免的失败模式。

Vercel CEO Guillermo Rauch 在 Vercel 开源其面向云端编码 Agent 的参考平台时，阐述了这一战略逻辑，并点名了本文所引用的同类系统：

你已经听说，Stripe（Minions）、Ramp（Inspect）、Spotify（Honk）、Block（Goose）及其他公司都在构建自己的“AI 软件工厂”。为什么？……从商业层面看，软件公司的护城河将从“它们写出的代码”，转向生产这些代码的“生产资料”。真正的 alpha 在于你的工厂。

@rauchg，2026 年 4 月 14 日

每个已公开软件工厂都具备的五个阶段

这些系统大多基于后台编码 Agent，即在独立环境中无人值守运行、而非在编辑器中运行的 Agent。阅读足够多这类架构后，无论公司如何命名，都会看到相同的骨架。Mastra 在 Mastra Factory 中将其作为六个命名阶段交付；Spotify 将其描述为嵌套反馈循环；Stripe 则将各部分称为蓝图。它们的形态是相同的。

阶段	Stripe（Minions）	Spotify（Honk）	Shopify（River）	Ramp（Inspect）
任务接入	Slack 消息、表情回应	Fleetshift 跨仓库选择目标	在公开频道中使用 @river	被分配的任务
隔离	预热的 EC2 devbox	Kubernetes Pod，访问受限	在持久会话上运行的一次性 harness	来自文件系统快照的 Modal 沙箱
工具	Toolshed，约 500 个内部 MCP 工具	通过 MCP 访问内部系统	凭据代理与网关	测试、遥测、功能开关、截图
验证	5 秒内完成 lint 和测试，随后运行受限 CI	确定性检查、LLM 裁判、CI	自动化 PR 审查模式	视觉与遥测验证
合并关卡	两轮 CI 运行后进行人工审查	人工审查	人工审查	人工审查

阶段顺序比工具选择更重要。每一道关卡都会阻止工作流向下一阶段，而成本高昂的阶段位于流程末端。

在展开细节前，先说明一个架构要点。Anthropic 的托管 Agent 架构将软件工厂划分为大脑（模型和 harness，无状态）、双手（一次性沙箱）以及会话（持久的只追加事件日志）。Shopify 在 Under the River 中直接引用了这一设计。如果你从本文中只构建一项能力，那就构建会话日志，因为它是让其他一切都能够一次性化的基础。

在单个会话中并行运行 Agent，与运行一个 Agent 集群是不同的问题。我们的 Codex 多 Agent 编排指南介绍了该层面的子 Agent、扇出和 worktree 机制。

阶段 1：工作如何到达 Agent

任务接入决定哪些工作值得启动。如果这一环节做错，之后的每个阶段都会在本不该开始的工作上消耗 token。

最朴素的做法是为每个开放 issue 分配一个 Agent。所有公开方案都会先进行筛选。Sentry 的 Seer会对每个新进入的错误评估可操作性，只调查达到门槛的错误。Shopify 采取了不同做法，将任务接入路由至 Slack，并只有一条规则：Agent 在公开频道工作，绝不在私信中工作。Shopify CEO Tobi Lütke 将这一约束描述为有意为之：

River 不会回应私信。她会礼貌地拒绝，并建议你创建一个公开频道，与你和她一起开始工作。……因此，每段对话都可以被搜索。Shopify 的任何人都可以参与。

@tobi，2026 年 5 月 9 日

其论点是组织层面的，而非技术层面的。一次私有 Agent 会话只会教会一个人，并随着窗口关闭而消失。

一个最小的任务接入过滤器由标签加查询构成。下面的命令会提取那些由人工明确标注为适合 Agent 处理、且规模较小的 issue：

gh issue list \
  --label "agent-ready" \
  --state open \
  --limit 20 \
  --json number,title,labels,body \
  --jq '.[] | select(.labels | map(.name) | index("needs-design") | not)'

Microsoft 的数据说明了为何规模过滤应该放在这里。在 dotnet/runtime 的十个月实践中，涉及 1 至 50 行变更的 Agent PR，成功率为 76% 至 80%；而性能优化工作落地率为 54.5%。公开总结直白地描述了这种规律：Copilot 的编码 Agent“非常擅长实现规格明确的变更，非常善于调查问题，但在架构解决方案方面相对较弱”。

去重关卡，以及为何它的大多数版本都是失败即放行

任务接入还应该提出第二个问题，但几乎没有人这样做：是否已经有人在上游修复了这个问题？

对于一名工程师而言，这只是锦上添花。但在 Stripe 每周合并 1,300 个 Agent PR的规模下，让 Agent 着手已经解决的问题，恰恰是软件工厂应当消除的浪费；若没有机制进行检查，这种浪费就不会浮现。

Firecrawl 的开发者索引适合回答这类问题，因为它索引的是 issue 讨论、已合并 pull request、README 和文档，而不是关于它们的博客文章。将搜索范围限定到相关依赖：

curl -X POST https://api.firecrawl.dev/v2/search/developer \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"connection pool exhausted under high concurrency",
       "k":3,"types":["issue","pull_request"],
       "repos":["magicstack/asyncpg"],"passages":1}'

这里与关卡相关的部分至关重要。当你使用 repos 限定范围时，响应会新增顶层 repos 区块，并区分两种原本看起来完全相同的情形：

{ "success": true, "partial": false,
  "repos": [ { "repo": "magicstack/asyncpg", "indexed": true,
               "types": { "issue": true, "pullRequest": true, "readme": true } } ] }


{ "success": true, "partial": false,
  "repos": [ { "repo": "acme/internal-fork", "indexed": false,
               "types": { "issue": false, "pullRequest": false, "readme": false } } ] }

两种情况都会返回 HTTP 200 及 success: true。第二种情况会返回零条结果。因此，写成 if not results: spawn_agent() 的关卡无法区分“没有人报告过这个问题”与“我们没有覆盖该仓库”，从而让两者都通过。这道关卡会在失败时放行。

读取 repos[0].indexed，并将 false 视为未知而非已清除，只需一行修复，就能将搜索框变成关卡。未知状态应路由给人工，或在不限定范围的情况下重试。

对于单个 issue 的分诊，即问题在于一个生产环境 bug 是否确实属于你的情况，我们已在使用编码 Agent 修复生产环境 bug中深入介绍过该工作流。

阶段 2：选择隔离模型

两个 Agent 编辑同一个工作目录，是浪费一天时间最快的方法。这也是大多数自建软件工厂停滞的地方，因为显而易见的方案在大约第四个并发 Agent 出现之前都能正常工作。

以下是三种模型，按成本和能力递增排列：

模型	隔离内容	不隔离内容	适用场景	真实案例
Git worktree	文件、分支	端口、数据库、已安装依赖、网络	单机运行 2 至 5 个 Agent	Claude Code 的 --worktree
容器	文件、依赖、网络、进程	主机资源	冲突依赖、不受信任的变更	container-use、Sculptor
云沙箱	一切，包括并发能力	你所需要的没有任何东西	集群规模、无限并行	Stripe devbox、Ramp on Modal、Spotify on Kubernetes

Worktree 是适合开始的选择。Git worktree 是位于自己分支上的第二个工作目录

TL;DR

An AI software factory is five stages with a gate at each one. The agent is the cheap part.

Stage	What it decides	Published example
Intake	Which work is worth starting	Sentry's Seer scores every incoming issue for actionability first
Isolation	Where the agent runs without colliding	Stripe boots pre-warmed devboxes in about 10 seconds
Tools	What the agent can reach	Stripe's Toolshed exposes roughly 500 internal tools over MCP
Verification	Whether the change is right	Spotify's LLM judge vetoes about 25% of agent sessions
Merge gate	Who is accountable	Faire requires two human reviews on agent-authored PRs

The short version: every company that made this work built the gates before the fleet. Spotify's Fleetshift shipped in 2023, two years before it had an agent to put in it. Generation scales with spend, review does not, and that asymmetry is the whole design problem.

WhereFirecrawl fits. Agents need live web context the repo does not carry. Search + scrape for the open web and a curated developer index for code, both behind one MCP block, with prompt injection detection on every fetch.

---

On January 6, 2026, Stephen Toub opened nine pull requests from his phone at 35,000 feet. Seven of them merged. He works on dotnet/runtime, and he wrote up what the experience told him:

AI changes the economics of code production. One person with good judgment and a phone can generate PRs faster than a team can review them.

That sentence is the entire subject. A single engineer with a coding agent can now saturate a team's review capacity from an airplane seat. The interesting question stopped being how to make agents write code and became how to absorb the output.

An AI software factory is the answer companies have converged on, and it is what turns autonomous coding agents from a demo into throughput a team can absorb. This guide breaks it into five stages, each with the published architecture behind it and the configuration to build it. For the broader picture of how AI agents reason, call tools, and pull in web context, start with that primer, then come back here for how a coding-agent fleet actually gets shipped.

Assume you have already picked an agent, which we covered in our roundup of the best AI coding agents. The software factory is everything around it.

What is an AI software factory?

An AI software factory is the system around a coding agent rather than the agent itself. Work arrives from a queue, agents run in isolated workspaces, verification happens automatically, and a human sits at an explicit merge gate. Also called an agentic software factory.

The useful distinction is between an agent and a software factory. Running a coding agent on your laptop is an agent: you choose the task, you watch it work, you read the diff, you merge. Everything except the typing is still you, and your attention is the limit.

A software factory moves those steps into infrastructure. Nobody decides which issue an agent picks up, because intake rules do. Nobody sets up a workspace, because isolation is provisioned. Nobody checks whether the change compiles, because verification runs before a human is involved at all. The person shows up at the end, on the decisions that carry accountability.

Addy Osmani puts it more compactly:

A software factory is harnessing loops at scale.

The loops he means are the ones we covered in our guide to loop engineering: an agent that runs, checks its own work, and runs again until a verifier says stop. The software factory is the machinery that runs many of them at once without anyone watching.

Three properties separate a real software factory from a pile of scripts:

• It is queue-driven, not prompt-driven. Work enters from issues, alerts, or a Slack channel, and the system decides what is worth starting. Nobody is typing prompts.
• Environments are disposable. Every agent gets a clean workspace it can destroy, so a bad run costs nothing and parallel runs cannot corrupt each other.
• Verification runs before review. By the time a diff reaches a person, it has already compiled, passed tests, and been checked for scope.

Miss the third and you have not built a software factory. You have built a machine that generates review work faster than you can absorb it, which is the failure mode the rest of this article is organized around avoiding.

Guillermo Rauch, Vercel's CEO, made the strategic case when Vercel open sourced its own reference platform for cloud coding agents, and he named the same systems this article draws on:

You've heard that companies like Stripe (Minions), Ramp (Inspect), Spotify (Honk), Block (Goose), and others are building their own "AI software factories". Why? [...] On a business level, the moat of software companies will shift from 'the code they wrote', to the 'means of production' of that code. The alpha is in your factory.

@rauchg, April 14, 2026

The five stages every published software factory shares

Most of these systems are built on background coding agents, meaning agents that run unattended in their own environment rather than in your editor. Read enough of these architectures and the same skeleton appears, whatever the company calls it. Mastra ships it as six named stages in Mastra Factory. Spotify describes it as nested feedback loops. Stripe calls the pieces blueprints. The shape is the same.

Stage	Stripe (Minions)	Spotify (Honk)	Shopify (River)	Ramp (Inspect)
Intake	Slack message, emoji reaction	Fleetshift picks targets across repos	@river in a public channel	Assigned task
Isolation	Pre-warmed EC2 devboxes	Kubernetes pods, constrained access	Disposable harness on durable sessions	Modal sandboxes from filesystem snapshots
Tools	Toolshed, ~500 internal MCP tools	Internal systems over MCP	Credentials proxy and gateway	Tests, telemetry, feature flags, screenshots
Verification	Lint and tests in under 5s, then capped CI	Deterministic checks, LLM judge, CI	Automated PR review mode	Visual and telemetry verification
Merge gate	Human review after two CI runs	Human review	Human review	Human review

The stage order matters more than the tooling. Each gate stops work from reaching the next stage, and the expensive stages sit at the end.

One structural note before the details. Anthropic's managed agents architecture splits a software factory into a brain (the model and harness, stateless), hands (disposable sandboxes), and a session (a durable append-only event log). Shopify cites it directly in Under the River. If you build nothing else from this article, build the session log, because it is what lets everything else be disposable.

Running agents in parallel inside a single session is a different problem from running a fleet. Our guide to multi-agent orchestration with Codex covers subagents, fan-out, and worktree mechanics at that level.

Stage 1: how work reaches an agent

Intake decides what is worth starting. Get this wrong and every downstream stage burns tokens on work that should never have begun.

The naive version assigns an agent to every open issue. The published versions all filter first. Sentry's Seer scores each incoming error for actionability and only investigates the ones that clear the bar. Shopify made a different call and routed intake through Slack, with one rule: agents work in public channels, never DMs. Shopify CEO Tobi Lütke described the constraint as deliberate:

River does not respond to direct messages. She politely declines and suggests to create a public channel for you and her to start working in. [...] Every conversation is therefore searchable. Anyone at Shopify can jump in.

@tobi, May 9, 2026

The argument is organizational rather than technical. A private agent session teaches one person and dies with the window.

A minimal intake filter is a label plus a query. This pulls issues that a human has explicitly marked as agent-eligible and small:

gh issue list \
  --label "agent-ready" \
  --state open \
  --limit 20 \
  --json number,title,labels,body \
  --jq '.[] | select(.labels | map(.name) | index("needs-design") | not)'

Microsoft's data says why the size filter belongs there. Across ten months on dotnet/runtime, agent PRs of 1 to 50 changed lines succeeded 76 to 80% of the time, while performance work landed at 54.5%. The published summary is blunt about the shape of it: Copilot's coding agent is "excellent at implementing well-specified changes, very good at investigating issues, and relatively poor at architecting solutions."

The dedup gate, and why most versions of it fail open

There is a second question intake should ask, and almost nobody does: has someone already fixed this upstream?

For one engineer that is a nice-to-have. At Stripe's 1,300 merged agent PRs a week, spawning agents onto already-solved problems is exactly the waste a software factory exists to remove, and nothing surfaces it unless something checks.

Firecrawl's developer index covers this shape of question, since it indexes issue threads, merged pull requests, READMEs, and docs rather than blog posts about them. Scoped to the dependency in question:

curl -X POST https://api.firecrawl.dev/v2/search/developer \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"connection pool exhausted under high concurrency",
       "k":3,"types":["issue","pull_request"],
       "repos":["magicstack/asyncpg"],"passages":1}'

Here is the part that matters for a gate. When you scope with repos, the response adds a top-level repos block, and it distinguishes two cases that otherwise look identical:

{ "success": true, "partial": false,
  "repos": [ { "repo": "magicstack/asyncpg", "indexed": true,
               "types": { "issue": true, "pullRequest": true, "readme": true } } ] }


{ "success": true, "partial": false,
  "repos": [ { "repo": "acme/internal-fork", "indexed": false,
               "types": { "issue": false, "pullRequest": false, "readme": false } } ] }

Both return HTTP 200 with success: true. The second returns zero results. So a gate written as if not results: spawn_agent() cannot tell "nobody has reported this" from "we have no coverage of that repo," and waves both through. That gate fails open.

Reading repos[0].indexed and treating false as unknown rather than clear is a one-line fix that turns a search box into a gate. Unknown routes to a human or retries unscoped.

For single-issue triage, where the question is whether a production bug is even yours, we covered the workflow in depth in fixing production bugs with a coding agent.

Stage 2: pick an isolation model

Two agents editing one working directory is the fastest way to lose a day. This is where most homegrown software factories stall, because the obvious answer works until roughly the fourth concurrent agent.

Three models, in ascending order of cost and capability:

Model	Isolates	Does not isolate	Good for	Real example
Git worktrees	Files, branch	Ports, databases, installed deps, network	One machine, 2 to 5 agents	Claude Code's --worktree
Containers	Files, deps, network, processes	Host resources	Conflicting dependencies, untrusted changes	container-use, Sculptor
Cloud sandboxes	Everything, plus concurrency	Nothing you need	Fleet scale, unlimited parallelism	Stripe devboxes, Ramp on Modal, Spotify on Kubernetes

Worktrees are where to start. A git worktree is a second working directory on its own branch, sharing one repository. The examples below use Claude Code, though the pattern is the same in Codex; we compared how the two behave on longer tasks in Claude Code vs Codex. Claude Code creates one per session:

claude --worktree feature-auth

That lands in .claude/worktrees/feature-auth/ on a branch named worktree-feature-auth. You can pin the isolation to a specific subagent instead, so a refactoring agent always gets its own tree:

---
name: refactorer
description: Applies mechanical refactors across many files
isolation: worktree
---
 
Apply the requested refactor across every affected file, then run the tests
and report the results.

The part every worktree guide skips

A worktree is a fresh checkout, which means your .env is not in it. Neither is node_modules, and neither is a free port. This is the thing that actually breaks at agent four, and it is why teams conclude worktrees "do not scale" when what they hit was an unconfigured checkout.

Claude Code handles the gitignored-file half with a .worktreeinclude file at the project root, using gitignore syntax:

.env
.env.local
config/secrets.json

Ports and databases are still yours to solve. Assign them from the worktree name rather than hardcoding, and give each agent its own database. Add .claude/worktrees/ to .gitignore while you are there.

Two more settings worth knowing. New worktrees branch from the repository's default branch, which is usually what you want for independent tasks; set worktree.baseRef to "head" when agents need your in-progress work. And you can branch a worktree straight from a pull request, which is how you point an agent at review feedback:

claude --worktree "#1234"

Cleanup is the failure mode nobody plans for. Interactive sessions prompt on exit, but non-interactive runs with -p do not clean up at all, and each one holds a lock until a later sweep releases it. If you are scripting a fleet, clean up explicitly:

git worktree list
git worktree remove .claude/worktrees/feature-auth
# if git refuses because the tree is locked
git worktree unlock .claude/worktrees/feature-auth

Graduate to containers when agents start needing conflicting dependency versions, and to cloud sandboxes when concurrency matters more than the cost of running infrastructure. Ramp's Inspect makes the case for the top rung plainly: "When background agents are fast, they're strictly better than local: same intelligence, more power, and unlimited concurrency."

Stage 3: give the agent hands, not just a brain

A model with a repository is an autocomplete. A model with your test runner, your telemetry, your feature flags, and your deploy tooling is a colleague. The gap between those two is the tool layer, and it is the least glamorous and most decisive stage in the software factory.

The published numbers say how seriously the leaders take it. Stripe's Toolshed hosts roughly 500 internal tools behind one MCP server, with controls that block destructive actions. Cloudflare runs an internal MCP Portal and generated AGENTS.md across more than 3,900 repositories. Ramp wires tests, telemetry queries, feature flags, and screenshot verification into every sandbox.

The leverage is that this is fleet-wide configuration, not per-agent setup. One MCP block reaches every agent:

{
  "mcpServers": {
    "firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": { "FIRECRAWL_API_KEY": "${FIRECRAWL_API_KEY}" }
    }
  }
}

A representative tool layer for a software factory looks roughly like this:

Capability	Answers	Reached via
Test and lint runners	Does it build and pass	Shell in the sandbox
Telemetry	Did it break in production	Observability MCP server
Feature flags	Is this path even live	Internal MCP tool
Ecosystem and docs	Is this API still real	firecrawl_developer_search
Deploy and rollback	Can this be undone	Internal MCP tool, gated

Rauch pushes the same point further, arguing that the context itself should live in one place:

Your software factory should be a monorepo. All your company context (design, marketing, sales, engineering, support…) in one place for agents to build upon

@rauchg, August 18, 2026

We arrived at the same place at Firecrawl. We run an internal monorepo called Firebrain that holds the complete company context, from product and engineering to marketing, sales, and support, in one place agents can read. The payoff is agents that already know how the company works before they start a task. It matters more than usual for us because the team is fully remote, spread across five continents and many time zones, so the repo is often the only colleague awake.

On instructions files, resist the urge to write a manual. OpenAI's Codex team runs a repository of roughly a million lines with an AGENTS.md of about 100 lines, which "serves primarily as a map, with pointers to deeper sources of truth elsewhere." Microsoft's data backs the general principle from the other direction: adding .github/copilot-instructions.md and firewall configuration moved agent success on dotnet/runtime from 41.7% to a sustained 71%. Preparation was worth more than any model upgrade in the same period.

For the tradeoffs between wiring tools over MCP versus letting agents shell out, see our MCP vs CLI comparison and the case for why agents prefer the CLI. To package a role so every agent in the fleet inherits it, see our overview of Claude plugins and skills.

Stage 4: verification is where software factories actually differ

Every software factory generates code. What separates them is what happens between generation and a human's eyes, and this is the stage with the most counterintuitive published results.

Spotify's feedback loops post describes the clearest structure: nested loops, cheapest first.

1. Inner loop. Deterministic verifiers selected automatically from what is in the repo. Format, compile, test. No model involved.
2. The judge. A model evaluates whether the diff stayed in scope. It vetoes roughly 25% of agent sessions, and about half of those are recoverable by course-correcting the agent rather than discarding the work.
3. Outer loop. CI and PR checks.

Spotify also ranks its failure modes, and the ranking is the useful part. A failed PR generation is an annoyance. A PR that fails CI is a burden on an engineer. A PR that passes CI and is functionally wrong erodes trust in the whole system. Design toward the first failure and away from the third.

The confidence-score trap

Here is the most valuable negative result published in this space, and it will save you a quarter.

Faire built an internal reviewer called faire-review and tried the obvious quality lever first: filter comments by the model's own confidence score. It did not work. At the strictest gate, 65% of all comments were thrown away and the acceptance rate moved by 3%. Their write-up shows a comment scored 0.93 that was dismissed sitting beside one scored 0.35 that was accepted and fixed.

What actually lifted acceptance to 73% was better context about the change, better targeting of where a comment is worth making, and model-graded gates in place of a deterministic threshold. Self-reported confidence is not a quality signal. A second model asking "is this comment worth a human's time" is.

AI code review is the most widely deployed piece of the software factory, and Uber reached the same conclusion from a different direction on uReview, which now covers more than 90% of roughly 65,000 weekly diffs. Its principle is that precision beats volume, because developers lose confidence fast when suggestions are noisy. The payoff is measurable: 65% of uReview's comments get addressed in the same changeset, against 51% for human reviewers.

The check that tests do not cover

Deterministic verifiers answer whether the code compiles and the tests pass. Neither question covers whether the page still looks right. A CSS change can clear every check in the inner loop and quietly break the phone view, because almost nobody writes tests asserting that a layout is reasonable.

That is worth closing, because it is precisely the failure Spotify ranks as most damaging: a change that passes CI and is wrong. Give the agent a browser, and pin the routine as a skill file so every agent in the fleet runs it the same way instead of improvising:

---
name: visual-verify
description: Screenshot affected pages before and after a change, then attach the evidence to the PR.
---
 
## Verify
1. Read the git diff to identify which pages changed
2. Screenshot each affected page at 1280px and 375px before applying the change
3. Apply the change, then screenshot the same pages at the same two widths
4. Capture anything the browser console printed
 
## Report
Attach every screenshot to the pull request, labelled before and after.

Step 1 is the load-bearing one. Reading the diff keeps the check scoped to what actually changed, rather than testing the whole app badly. We covered the fuller routine, including Devin's three-phase testing pattern and how the instruction wording decides whether any of it helps, in fixing production bugs with a coding agent.

The last step used to be where this broke. Producing screenshots is easy. Getting them onto the pull request meant a person dragging files into a comment box, which puts a human back in the loop at exactly the point you were trying to automate. GitHub closed that on September 1, 2026 with a repeatable --attach flag, in gh v2.99.0 and later:

gh pr comment "$PR" \
  --body "Visual verification: checkout flow, desktop and mobile." \
  --attach './before-1280.png#Checkout at 1280px, before' \
  --attach './after-1280.png#Checkout at 1280px, after' \
  --attach './before-375.png#Checkout at 375px, before' \
  --attach './after-375.png#Checkout at 375px, after'

Text after # becomes alt text, and gh falls back to the filename without it. The flag behaves the same on gh issue create, edit, and comment, and on gh pr create, edit, and comment. Images cap at 10MB; video is 10MB on free plans and 100MB on paid ones. GitHub Enterprise Server is not supported in this release.

Distributing the skill is its own fleet problem, and gh skill handles it in preview. Publish once, and every agent installs the same pinned version instead of carrying a local copy that drifts:

gh skill install your-org/agent-skills visual-verify@v1.2.0 --agent claude-code

One honest limit. This proves the page rendered and the flow ran. It does not prove the change is well built or that it holds up next quarter. It kills the "said it was fixed, never actually ran it" failure, and it does not replace review.

Order your checks by what they cost

Stripe caps agents at a maximum of two CI runs before handing off to a human. At more than 3 million tests, a CI cycle is the expensive resource, so the question becomes which checks earn a slot ahead of it.

The ordering falls out of unit economics. Lint and tests return in under five seconds locally. A scoped ecosystem lookup costs 2 credits per 10 results as of September 2026. A CI cycle across a suite that size costs vastly more than either. So when an agent writes against an API it half-remembers, checking that call against real issues, PRs, and migration guides before pushing is close to free:

curl -X POST https://api.firecrawl.dev/v2/search/developer \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"createClient options changed breaking rename in v5",
       "k":3,"types":["issue","pull_request","doc"],"passages":1}'

Run live, that returns actual migration guides, including Migrating-v4-to-v5.md from the Deepgram JavaScript SDK, with citation URLs pinned to a commit SHA rather than a branch, so the reference does not drift.

Be realistic about the hit rate. On Firecrawl's own DevDex benchmark the index returns a correct answer in its top ten 63% of the time, against 45% for ordinary web search. That is a meaningful edge and roughly a third of queries still miss. It narrows the search, it does not end it. We covered the retrieval mechanics and the build-versus-buy decision in giving coding agents up-to-date documentation.

Stage 5: the merge gate

An agent merge policy is the least written-about stage and the one that determines whether the software factory is an asset or a liability. The question is not whether an agent may open a PR. It is what has to be true before one lands.

Published policies cluster into tiers by blast radius:

Change class	Example	Gate
Mechanical, reversible	Dependency bump, lint fix, test migration	Automated checks, auto-merge
Scoped feature or fix	Bug fix under 50 lines, new test coverage	One human review
Agent-authored, non-trivial	Anything touching product behavior	Two reviews, per Faire's rule
Domain-sensitive	Auth, payments, cryptography, data deletion	Named domain owner, regardless of author

Faire's rules are the most concrete published set, and two of the three are about review sequencing rather than review itself. Require two reviews on Copilot-authored PRs, the assignee's plus another human's. Do not request review from code owners until the PR already has one review, which stops agents from pulling domain experts into unfinished work. Mark ready for review only when the agent requests it.

Mastra's Factory separates the gate into three distinct approvals, which is a clarifying way to think about it: accepting the work into the queue, approving the plan, and merging the PR. Its docs are explicit that these are independent, and "acceptance authorizes the task to enter work. It's separate from approving the plan and from merging the eventual PR."

Then there is attribution. The Linux kernel's 2026 policy states that AI agents must not add a Signed-off-by: line, since that trailer is a legal assertion a model cannot make, and requires an Assisted-by: AGENT_NAME:MODEL_VERSION trailer instead. GitHub went the other way with Agent HQ, managing agent identity the way it manages human developers. Either is defensible. Silence is not, because six months later nobody can answer which changes came from where.

The far end of the spectrum is worth knowing about. OpenAI's Codex team reports that on its internal project "humans may review pull requests, but aren't required to," agents "often squash and merge their own pull requests," and the team has "pushed almost all review effort towards being handled agent-to-agent." That is a real position, taken on a greenfield codebase with no external users, by the team that builds the agent. Read the next section before adopting it.

Source:Stephen Toub, "Ten Months with Copilot Coding Agent in dotnet/runtime", retrieved September 10, 2026. 396 agent pull requests received human commits; 482 did not.

What the numbers actually say

The vendor pages for this category cite the same five statistics. Here is the fuller picture, including the parts that do not flatter the idea.

Agent PRs merge less often than human ones. Microsoft's ten-month dotnet/runtime review is the only public dataset with a like-for-like comparison. Agent PRs merged 67.9% of the time against 87.1% for Microsoft engineers. The autonomy split is sharper still: PRs that received human commits merged 86.2% of the time, fully autonomous ones 55.1%. Revert rates held steady at 0.6% against 0.8%, so what does land is not noticeably worse.

Source:Stephen Toub, "Ten Months with Copilot Coding Agent in dotnet/runtime", retrieved September 10, 2026. Across 6,181 pull requests.

Passing tests is not the same as being mergeable. METR asked maintainers from scikit-learn, Sphinx, and pytest to review 296 AI pull requests that had already passed SWE-bench's automated grader. Roughly half would not have been merged, a gap of about 24 percentage points against the grader. The gap did not shrink across models from mid-2024 to mid-2025.

Throughput and instability rise together. DORA's 2025 report found AI adoption correlated with more delivery throughput and more delivery instability at the same time, concluding that AI is an amplifier of whatever an organization already is. Its 2026 ROI modeling puts first-year return near 39% with an eight-month payback, but only through a J-curve dip it attributes partly to a verification tax. Gains run 35 to 40% on simple tasks and often under 10% on legacy code.

Volume has a maintainability cost. GitClear's analysis of 211 million changed lines found duplicated code blocks rose eightfold in 2024, refactored lines fell from about 25% of changes to under 10%, and churn roughly doubled from 3.3% to 7.1%.

Set against that, the adoption numbers are real and large. Shopify reports 59,918 River sessions and 3,536 River-coauthored pull requests merged in a single 30-day window, and Lütke put the share plainly: "About one in eight pull requests merged into our codebase last week was authored by River, reviewed by us." Ramp reached about 30% of merged PRs with no mandate at all. Airbnb migrated about 3,500 test files in six weeks against an estimated 1.5 years by hand, with 75% done in the first four hours.

Notice what that last one is. It is a migration. So is Spotify's fleet work, and Amazon's Java version upgrades. The proven workload is mechanical change at scale, not greenfield product design.

Where software factories break

Review capacity, first and always. On dotnet/runtime, merged agent PRs drew 16.5 review comments against 12.4 for human PRs, and the top two reviewers produced 36% of all feedback. That is a load-bearing pair of people. Faire names the same problem directly: PR volume went up while the number of human reviewers stayed flat.

The last stretch is expensive. Airbnb's long-tail files needed 50 to 100 retries each. The most-quoted line from Hacker News practitioners puts it well: "Agentic software development delivers if you're not too picky. The last 10% might last a very long time and cost many tokens."

Parallel branches collide. Worktrees cannot check out the same branch twice, and merge queues degrade as agent throughput rises. Unglamorous, and where fleets actually stall.

Cost is not obviously worth it, and the honest advocates say so. When one team published a manifesto arguing code should be neither written nor reviewed by humans and recommended spending $1,000 per engineer per day on tokens, the top Hacker News response was: "The site has zero benchmarks, zero defect rates, zero cost comparisons, zero production outcomes. The only metric offered is 'spend more money.'"

Slop accumulates unless something eats it. OpenAI's Codex team is candid that before they automated the loop, "our team used to spend every Friday (20% of the week) cleaning up 'AI slop.' Unsurprisingly, that didn't scale."

Open source is absorbing the externality. Steve Ruiz, who founded the canvas library tldraw, summarized the shift when his project paused external contributions: "Writing code is now easy; the real work is reviewing it." The most instructive data point is curl's. After years of rising low-quality reports, the maintainers took all of July 2026 off from vulnerability reporting and called it "possibly our best project decision in a long while." Daniel Stenberg notes the flood has not slowed for the other security teams he sits on. The only intervention that worked was closing intake, which is a gate argument.

Give your software factory live web context with Firecrawl

The workload that pays for a software factory is migration and maintenance. Spotify's fleet-wide changes, Airbnb's test migration, Amazon's Java upgrade: all the same shape, all mechanical, all at a scale humans resent.

Every one of those is triggered by something that does not live in your repository. A release note. A deprecation. A breaking change three dependencies deep. Fleetshift-style tooling finds the targets inside your code, but what to migrate to is an external fact, and a software factory that only reads its own repo cannot see the change that created the work.

Firecrawl's developer index puts that live web context behind one search call: 70M+ issue threads, merged pull requests, READMEs, and documentation pages, most refreshed daily. Filters narrow it to the current answer rather than the popular one:

npx -y firecrawl-cli@latest setup developer-index


curl -X POST https://api.firecrawl.dev/v2/search/developer \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"modern replacement for moment.js, tree shakeable",
       "k":5,"types":["readme"],"language":"TypeScript","min_stars":2000}'

Same server, same firecrawl_developer_search tool, available to every agent in the fleet from one MCP block.

When the answer is not in a developer artifact, the same MCP exposes the general Firecrawl search and scrape pipeline: firecrawl_search returns ranked results across the open web, and firecrawl_scrape turns any URL, including JavaScript-rendered pages, into clean markdown the agent can reason over. That is the shape most factory-side lookups take. Search to find the two or three URLs that could plausibly answer the question, scrape to pull the current text off each one, then let the agent decide. A vendor status page, a partner API's changelog, a pricing table, a government schema: none of it lives in the repo, all of it lands in the same tool call, and none of it goes stale between runs.

A scraped page is also a place an attacker can plant instructions aimed at your agent. Firecrawl's prompt injection detection runs on every extraction, flags suspected injection attempts in the response, and lets a fleet operator drop or quarantine the offending content before it reaches the model. It is the piece that makes an open-web tool safe to hand a background agent.

Start with one gate

Pick the stage where your agents currently produce the most rework, and put a gate there. Not a fleet, not an orchestrator, one gate. Every architecture in this article was assembled that way, and the companies that skipped straight to volume are the ones writing the honest posts about cleaning up on Fridays.