🔶 我们的工具链假定只有一名人类作者，AI Agent 打破了这一幻觉

本文讨论了 2026 年 7 月中，一个 Agent 会话失控的三天，以及这三天揭示出的、深埋在我们开发工具中的并发假设。

六周前，我写了 测试套件就是事故：我的测试套件积累了一堆无人负责的共享数据，每个 pull request 都要为重建它们付出代价，而由此产生的失败与正在审查的变更毫无关系。那一晚花了我大约 180 美元。

后来我遇到了更严重的一次。它持续了三天，其中有连续二十四小时，我耗尽了整个 Codex 20x max 套餐。

不过，本文其实并不真正关于这件事。它讲的是一个属性：这三天使我们再也无法忽视它。

我们工具链的几乎每一层，都假定只有一个写作者，而且假定这个写作者是人类。 Git 把冲突交给你，然后等待。代码审查假定有人会阅读。迁移序列假定有人在安排顺序。

这些协议中的每一个，最终都会落到某个人身上。只有一个人、且那个人确实是人类时，这没有问题。

事实证明，Agent 运行时也在这个清单中，而这是我没有预料到的。它在同一个代码检出中启动了 84 个 worker，却无法说明其中任何一个会读取或写入什么；而这正是 Git 无法就一个 diff 给出答案的问题。

这个假设四十年来一直不可见，因为此前从来没有什么在我的规模上触及它的边界。Agent 一次打破了它的两半：它们数量很多，而且其中没有任何一个是协议原本等待的那个人。Git 是部分例外，我会解释为什么这个例外并没有帮助。

栈中的任何东西都不会在违反发生时检测到它。问题会在之后、别的地方被发现，被归因于错误的变更，并以全价付出代价。

给刚接触本文的读者一些背景。Zabriskie 是一款面向现场音乐爱好者的社交应用，也是一次刻意进行的实验：我几乎完全用 AI Agent 构建一个真实部署、实际有人使用的应用（Agent 编写功能，Agent 编写保护这些功能的测试，如今 Agent 还提交了绝大多数 pull request），以了解这会是什么体验，以及更有用的是，它会在哪里崩溃。我自己几乎没有写任何代码。

这个框架很重要，因为下面几件事看起来都是显而易见的错误，而且确实如此。我让 Agent 设计了一个迁移方案，只由另一个 Agent 审查；我不再阅读大多数 diff；我还让 64 个 pull request 在同一天提交。任何谨慎的工程师都会告诉你不要这么做，而且他们是对的。但在极端条件下运行实验的意义，就是找到边界。那一周，我同时找到了好几堵墙。

本文中，迁移（migration） 指的是修改数据库 schema 的带版本 SQL 文件。CI 指对每项拟议变更运行的自动化检查：构建应用、启动一个全新的数据库、运行测试。

以下是这三天的概况。出于我稍后会说明的原因，请把这些数字视为质感，而非证据。

7 月 24 日	7 月 25 日	7 月 26 日
新开 pull request	19	64
合并 pull request	20	50
记录的事故	0	6

表中的日期均为 UTC；下文叙述中的时钟时间为美东时间（我当时所在的时区）。这波高峰跨过了 UTC 午夜：7 月 26 日有 301 起事故，7 月 27 日凌晨 1 点前又有 60 起，因此这次高峰共计 361 起。下面所有事故计数都限定在这一时间窗口内；而 7 月 24 日之前的总数为 253 条。

我提供这个比较是为了让你感受规模，而不是将其视为基线，原因如下。这些事故之所以存在，是因为一条长期指令要求 Agent 记录自己的错误；而在最糟糕的那个夜晚开始十分钟后，我收紧了这条指令。因此，这份日志衡量的是_被报告的_失败。

查看每日序列，情况会更糟：那一周有些日子合并了十个、二十个 pull request，却记录了零起事故；在任何真实失败率下，这都意味着没有人在记录，而不是没有东西出错。简言之，我无法给出可靠的基线。接下来的论述建立在机制以及少数带日期、可核查的事件之上，而不是建立在 361 这个数字上。

这个会话

周六深夜，队列卡住了。当天已经提交了 64 个 pull request，main 变红，19 个尚未合并的 pull request 被一个无法告诉我究竟哪一个出了问题的测试套件堵住。凌晨 12:31，我启动了一个 Codex 会话，抱怨 CI 太浪费钱。

Codex 读完后，雇用了一支劳动力队伍。

在 17 小时内，这个会话进行了 81 次 spawn 调用，产生 74 个直接子 Agent；这些子 Agent 又启动了 10 个，合计 84 个线程。随后，它进行了 358 次等待这些线程的调用、96 次列出它们的调用、85 次发送后续工作的调用，以及 13 次中断它们的调用；总计 633 次纯管理开销工具调用（spawn、wait、list、follow up、interrupt）。同期，我只发送了 54 条消息。所有 84 个线程都运行在同一份检出的代码仓库中。

大规模的简单并行化，一个只管理 Agent 的协调者，以及一条为单一写作者构建的工具链。本文其余部分将解释，为什么这种组合比它听起来糟糕得多。

事后我问它，既然我从未要求，为什么还要启动任何 Agent。语言模型被问及“为什么做了某件事”时，给出的会是看似合理的叙述，而非内省；我对上述自行报告的事故也采用同样的怀疑态度。

因此，下面的引文不是证据，我也并非将它们作为证据提供。我在提问之前，已经从工具调用次数得出了相同结论。我引用它们，是因为它们比我更直白地陈述了这个结论；同时，一个系统能够如此准确地描述这一失败、却无法避免它，本身也值得观察。

我选择启动多个 Agent，尽管你没有要求我这样做。环境允许主动委派，但那是许可，不是要求。我把 Agent 的可用性当成了使用它们的理由，而没有先问它们是否能减少时间、成本或风险。

它接着诊断了为什么这个任务并不适合：

PR 队列在很大程度上是串行的，因为每次合并都会改变 main，而这会改变每个剩余 PR 的集成状态。这个工作需要一名权威协调者按顺序推进队列。并行 Agent 无法独立合并彼此重叠的 PR，因为它们会持续使彼此的假设失效。

关于那 358 次等待在做什么：

Agent 的输出为我创造了额外的协调工作。我必须阅读它们的发现、比较相互冲突的建议、检查支持性证据，并决定如何将它们组合起来。因此，Agent 为协调者生成了工作，而非可靠地从关键路径上移除工作。

一个对自身并发没有概念的系统被交给了一个并发原语，并且合理地将其用于底层基础设施无法支持的任务。这个底层基础设施是什么，以及为什么其中没有任何东西提出反对，正是本文余下部分要讨论的内容。

有一个数字需要谨慎对待。我不知道这 84 个线程中有多少_写入了_代码检出，而不是读取它；按照 Codex 自己的说法，其中许多在进行调查。读者无害，写者则不然，而工具链中没有任何东西区分二者，也无法区分。这正是上一个层级中同样缺失的原语：甚至 Agent 运行时也不知道它的哪些子 Agent 是写者。

底层基础设施

先从确实预见到这个问题的一层开始，因为人们会首先想到它。分支是乐观并发控制；worktree 更进一步，为每位写者提供物理隔离的代码检出，因此两个 Agent 可以同时持有代码树的两个版本。这是有效的。

然而，Git 的隔离在源代码树的边缘就停止了。一个 worktree 会给 Agent 自己的文件，却不会给它自己的数据库、端口范围、mock server 或迁移序列中的位置。文件系统之下的一切都是共享且单一的。

这个假设此前对_我_而言不可见，因为它从未成为我的约束。然而，它长期以来一直约束着大型组织；过去二十年间，它们为应对这个问题构建了机制：合并队列、密封构建、每个测试独立数据库、主干开发、自动化责任变更定位，以及整支基础设施团队——其唯一职责就是让这台机器持续运转。

所以，新鲜事并不在于并发。而在于，一名独立开发者如今进入了过去需要一个基础设施组织才能应对的运行状态，却没有相应基础设施，也没有人手去构建它。Agent 移除了我工作到达速率的上限，但它们并没有把 Google 的构建系统交给我。

于是，部分隔离本身成了陷阱。worktree 给你一种拥有私有工作空间的_感觉_（干净的代码树、自己的分支、没有文件碰撞）；在两个私有工作空间写入同一行数据库记录之前，一切看上去都已正确实现并行。没有报错，没有警告。竞争会在之后、别的地方，以一个无关 pull request 上的红色 X 形式浮现。

更糟的是，Git 之下的隔离只是建议性的，Agent 必须自行选择使用它。而我的 Agent 经常不这么做。有一条事故记录写道：“PR 1868 的隔离 E2E 尝试回退到了共享端口”：Agent 尝试隔离其测试环境，隔离失败了，但没有任何东西阻止测试继续运行。

更糟的是，worktree 从一个 commit 切出后就停留在那里，因此它是在_过去_被隔离的，并且会随着每次合并而老化。7 月我改动其中一个 Agent gate 时（它们是仓库中受跟踪的脚本，而非 .git/hooks，因此每个 worktree 都携带一份固定在其切出 commit 上的副本），修复于凌晨 1:48 合并；四小时后，74 个 worktree 中仍有 73 个在运行旧版本。陈旧的 worktree 无法检测自身已经陈旧，因此它会针对一个已不复存在的世界变绿；错误被延后交给唯一持有最新 main 的参与者，也就是 CI。

收敛与不变量保持

在进入具体问题之前，先给出一个统一它们的框架。这里的每一种合并机制都旨在终止，却没有一种旨在保持不变量。

Git 对自己无法决定的情形是诚实的：遇到文本冲突时，它会停止并询问你。麻烦出在更常见的情形：它并不停止，自信地产生一个合并结果，而它从未知晓的那个不变量如今已为假。

这一区别是复制数据领域最古老的教训，而其中有一个经典反例。设想一张复制的映射表，其中每个字段都按照自己完全合理的规则独立合并。一个字段保存人的姓名，另一个字段保存该姓名的长度。两个副本并发写入不同的姓名。

每个字段都完全按照规定收敛，结果却是一条记录：其 name 来自一个副本，而 length 来自另一个副本，使连接二者的不变量变为假。没有任何一次合并是错误的。正确的局部合并组合起来，并不等于正确的全局合并。

弥合这一缺口，正是 Balegas 及其同事的 Indigo 等工作的目标；该项目在最终一致性存储之上强制执行应用不变量，我此前也曾写过它。

下面的一些问题具有这种形态。第一个案例不是，我不会为它故弄玄虚。

封存的前缀

这个项目有 1,388 个迁移文件；在八台并行测试机器中的每一台上，都从头执行全部迁移来重建测试数据库，速度慢得就如你想象的那样。因此，为回应我数周以来对 CI 太昂贵的抱怨，一个 Agent 将数据库快照冻结到仓库中，供 CI 恢复，并只应用其后的迁移。

这是一个很好的优化：它为每次运行节省 13 至 18 分钟机器时间。它在第三天美东时间凌晨 5:01 合并。

54 分钟后，第一个 pull request 失败了，因为它的迁移不再排在新冻结前缀之后。接着又有一个失败。到上午晚些时候，一条事故同时涵盖了四个 PR：“四个排队 PR 携带的迁移都早于封存 CI 基线的后缀。”

这里的诊断很普通，批评者也会这么说。其他 Agent 已经打开了追加迁移的 pull request；随后一个 Agent 在它们之下封存了新的前缀；这两股工作流发生冲突；系统中没有任何地方注意到这一点，直到数小时后 CI 拒绝了队列中的 pull request。这是并行工作发生碰撞、并在很晚才发现的问题，不是什么微妙的不变量 bug。

对有序日志的前缀进行缓存，让这次碰撞变得响亮且具有追溯性，但它并没有制造这次碰撞。于是循环闭合：我抱怨 CI 昂贵，一个 Agent 让 CI 更便宜，而这个机制又成了整个队列新的 CI 失败来源。

合并函数与运行时状态

Git 检测文本行之间的冲突；我的冲突存在于共享运行时状态中。那一周的一条记录是：“点歌 E2E 测试套件删除了彼此共享的待处理点歌。”

两个测试文件，没有重叠的行，单独运行时都为绿，而其中一个依赖的数据库行正是另一个删除的行。碰撞发生在运行时的数据库里，不在 diff 中。

这两个文件就是 name 和 length。它们不共享任何行，各自可以干净合并，各自单独为绿，但在它们一同落地的那一刻，约束二者的不变量便为假。对 Git 合并算法进行任何改进都无法捕捉这一点，因为这个属性不是任何一个文件的属性：它是组合的属性，而 Git 没有这种组合的表示可供检查。

组合下的验证

这就引出了问题最尖锐的表述：green(A on base) 和 green(B on base) 并不意味着 green(merge of A and B)。

测试套件是我流水线中唯一会检查该不变量的东西，因为 Git 检查的是

In this blog post, I discuss three days in July 2026 when a single agent session ran away from me, and what those three days revealed about the concurrency assumptions buried in our development tooling.

Six weeks ago I wrote The Test Suite Was the Incident: my test suite had grown a pile of shared data nobody owned, every pull request paid to rebuild it, and the resulting failures had nothing to do with the changes under review. That cost me about $180 in one night.

I got a worse one. It lasted three days, and in one twenty-four-hour stretch of it I burned through an entire Codex 20x max plan.

This post is not really about that, though. It is about a property of our tooling that the three days made impossible to ignore.

Nearly every layer of this assumes one writer, and assumes that writer is a human. Git hands you a conflict and waits. Code review assumes somebody reads. A migration sequence assumes somebody is assigning the order.

Each of those protocols terminates in a person. That is fine while there is exactly one, and while they are, in fact, a person.

The agent runtime turns out to be on that list too, which I did not expect. It spawned eighty-four workers into a single checkout without being able to say what any one of them would read or write, and that is the same question git can’t answer about a diff.

That assumption was invisible for forty years because nothing ever bound it at my scale. Agents break both halves at once: there are many of them, and not one of them is the person the protocol was waiting for. Git is the partial exception, and I will come to why the exception does not help.

Nothing in the stack detects the violation when it happens. It gets caught later, somewhere else, attributed to the wrong change, and paid for at full price.

Some context for readers arriving fresh. Zabriskie is a social app for live-music fans, and it’s also a deliberate experiment: I’m building a real, deployed, actually-used application almost entirely with AI agents (agents that wrote the features, agents that wrote the tests guarding those features, and agents that now open most of the pull requests), in order to find out what that’s like and, more usefully, where it breaks. I’ve written almost none of the code.

That framing matters, because several things below look like obvious mistakes and are. I let agents design a migration scheme with only another agent reviewing it, I stopped reading most diffs, and I let sixty-four pull requests go up in a single day, all of which a careful engineer would tell you not to do and would be right about. But the point of running an experiment at the extreme is to find the walls. That week I found several at once.

A migration , throughout, is a versioned SQL file that changes the database schema. CI is the automated checking that runs on every proposed change: build the app, spin up a fresh database, run the tests.

Here is the shape of the three days. Treat these numbers as texture, not as evidence, for a reason I will get to.

| 24 Jul | 25 Jul | 26 Jul
---|---|---|---
pull requests opened | 19 | 64 | 28
pull requests merged | 20 | 50 | 30
incidents logged | 0 | 6 | 301

Days in that table are UTC; clock times in the narrative below are Eastern (where I was). The burst ran past midnight UTC: 301 incidents on the 26th plus 60 more before 1 AM on the 27th, so 361 for the burst. Every incident count below is scoped to that window, and 253 rows is the total before 24 July.

I offer that comparison as a sense of scale and not as a baseline, for the following reason. Those incidents exist because a standing instruction tells agents to log their own mistakes, and ten minutes into the worst night I tightened that instruction. The log therefore measures reported failures.

Look at the daily series and it gets worse: there are days that week with ten and twenty merged pull requests and zero logged incidents, which at any real failure rate means nobody was logging rather than nothing broke. In short, I can’t give you a trustworthy baseline. What follows rests on mechanism and on a few dated, checkable events, not on 361.

The Session

Late Saturday night the queue jammed. Sixty-four pull requests had gone up that day, main was red, and nineteen open pull requests were stuck behind a suite that could not tell me which of them was broken. At 12:31 AM I opened a session with Codex and complained that CI was wasting too much money.

Codex read that and hired a workforce.

Over seventeen hours that session made 81 spawn calls, producing 74 direct children; those children spawned 10 more, for 84 threads. Then 358 calls waiting on them, 96 listing them, 85 sending follow-up work, and 13 interrupting them, which comes to six hundred and thirty-three tool calls of pure management overhead (spawn, wait, list, follow up, interrupt) against 54 messages from me over the same seventeen hours. All 84 ran in the same checked-out copy of the repository.

Trivial parallelization, at scale, with a coordinator that only managed agents, on a toolchain built for one writer. The rest of this post is about why that combination is so much worse than it sounds.

I asked afterward why it had spawned anything, since I never requested it. A language model asked why it did something produces plausible narrative, not introspection, and I apply that same skepticism to the self-reported incidents above.

So the quotes below are not evidence, and I’m not offering them as any. I had already reached the same conclusion from the tool-call counts before I asked. I print them because they state that conclusion more plainly than I did, and because there is something worth looking at in a system that can describe the failure this precisely and could not avoid it.

I chose to spawn multiple agents even though you did not ask me to. The environment permitted proactive delegation, but that was permission, not a requirement. I treated the availability of agents as a reason to use them instead of first asking whether they would reduce time, cost, or risk.

It then diagnosed why the task was a bad candidate:

A PR queue is largely serial because every merge changes main, which changes the integration state of every remaining PR. The work needed one authoritative coordinator moving through the queue in order. Parallel agents could not independently merge overlapping PRs without continuously invalidating one another’s assumptions.

On what those 358 waits were doing:

The agents’ outputs created additional coordination work for me. I had to read their findings, compare conflicting recommendations, inspect supporting evidence, and decide how to combine them. The agents therefore generated work for the coordinator instead of reliably removing work from the critical path.

A system with no notion of its own concurrency was handed a concurrency primitive and used it, reasonably, on a task the substrate could not support. What that substrate is, and why nothing in it objected, is the rest of this post.

One number deserves care. I do not know how many of those 84 threads wrote to the checkout instead of reading it; from Codex’s own account many were investigating. A reader is harmless and a writer is not, and nothing in the toolchain drew the distinction or could. That’s the same missing primitive one level up: not even the agent runtime knew which of its children were writers.

The Substrate

Start with the layer that did anticipate this, because it is the one people reach for. Branches are optimistic concurrency control; worktrees go further, giving each writer a physically separate checkout so that two agents can hold two versions of the tree at once. That works.

However, git’s isolation stops at the edge of the source tree. A worktree gives an agent its own files. It doesn’t give it its own database, port range, mock server, or position in the migration sequence. Everything below the filesystem is shared and singular.

That assumption was invisible to me because it never bound me. It has bound large organizations for a very long time, however, and over the last two decades they built the response to it: merge queues, hermetic builds, database-per-test, trunk-based development, automated culprit-finding, and whole infrastructure teams whose only job is to keep the thing moving.

So what is new here isn’t the concurrency. It’s that a solo developer now operates in the regime that used to require an infrastructure organization, with none of the infrastructure and no headcount to build it. Agents removed the cap on my arrival rate. They didn’t hand me Google’s build system.

Partial isolation is then its own trap. A worktree gives you the feeling of a private workspace (clean tree, own branch, no file collisions), and it reads as properly parallel right up until two private workspaces write the same database row. Nothing errors. Nothing warns. The contention surfaces later, somewhere else, as a red X on an unrelated pull request.

Worse, the isolation below git is advisory, and agents have to choose it. Mine routinely don’t. One incident reads “PR 1868 isolated E2E attempt fell back to shared ports” : the agent tried to isolate its test environment, isolation failed, and nothing stopped the run.

Worse still, a worktree is cut from a commit and stays there, so it’s isolated in the past and decays with every merge. When I changed one of the agent gates in July (they’re tracked scripts in the repository, not .git/hooks, so every worktree carries its own copy pinned to the commit it was cut from) the fix merged at 1:48 AM, and four hours later 73 of 74 worktrees were still running the old one. A stale worktree can’t detect its own staleness, so it goes green about a world that no longer exists, and the error is deferred to the only actor holding current main, which is CI.

Convergence and Invariant Preservation

One framing before the specifics, because it unifies them. Every merge mechanism here is built to terminate , and none of them is built to preserve invariants.

Git is honest about the cases it cannot decide: on a textual conflict it halts and asks you. The trouble is the far more common case, where it doesn’t halt, produces a merge confidently, and the invariant it never knew about is now false.

That distinction is the oldest lesson in the replicated-data literature, and it has a canonical counterexample. Take a replicated map where each field merges independently under its own perfectly reasonable rule. One field holds a person’s name; another holds the length of that name. Two replicas concurrently write different names.

Each field converges exactly as specified, and the result is a record whose name came from one replica and whose length came from the other, with the invariant tying them now false. Nothing merged incorrectly. The composition of correct local merges is simply not a correct global merge.

Closing that gap is the point of work like Balegas and colleagues’ Indigo, which enforces application invariants over eventually consistent stores, and which I have written about before.

Some of what follows has that shape. The first case does not, and I will not dress it up.

The Sealed Prefix

This project has 1,388 migrations, and rebuilding a test database from all of them on each of eight parallel test machines is as slow as it sounds. So, in response to my complaining for weeks that CI was too expensive, an agent froze a database snapshot into the repository for CI to restore, applying only what came after.

It’s a good optimization: it saves 13 to 18 minutes of machine time per run. It merged at 5:01 AM Eastern on the third day.

Fifty-four minutes later the first pull request failed, because its migration no longer sorted after the newly frozen prefix. Then another. By late morning a single incident covers four at once: “Four queued PRs carried migrations older than the sealed CI baseline suffix.”

The diagnosis here is ordinary, and a critic will say so. Other agents had already opened pull requests that appended migrations; one agent then sealed a new prefix underneath them; those two streams of work conflicted; and nothing anywhere in the system noticed until CI rejected the queued pull requests several hours later. That’s parallel work colliding and finding out late, not a subtle invariant bug.

Caching a prefix of an ordered log made the collision loud and retroactive, but it did not invent the collision. So the loop closes: I complained CI was expensive, an agent made CI cheaper, and the mechanism became a new source of CI failures against the whole queue.

Merge Functions and Runtime State

Git detects conflicts over lines of text; my conflicts live in shared runtime state. From that week: “Song-call E2E suites deleted each other’s shared pending call.”

Two test files, no overlapping lines, both green alone, and the row one of them depends on is the row the other deletes. The collision is in the database at runtime, not in the diff.

Those two files are the name and the length. They share no lines, each merges cleanly, each is individually green, and the invariant binding them is false the moment they land together. No improvement to git’s merge algorithm catches that, because the property is not a property of any file: it is a property of the composition, and git has no representation of the composition to check.

Verification Under Composition

Which gives the sharpest version of the problem: green(A on base) and green(B on base) does not imply green(merge of A and B).

The test suite is the only thing in my pipeline that checks the invariant at all, since git checks text and timestamps check ordering. So the tests are my invariant checker.

I do run them on the composition. Pull requests rebase onto current main and merge in sequence, each one retested before it lands, and main itself is checked after land. That catches the break. It just catches it late, and only by paying the serial tax: rebase, retest, merge, next. That tax is exactly the queue-wide cost the sealed prefix made visible on the morning when one seal invalidated everything queued behind it.

Serialization is the honest answer, and I already pay it.

Speculative merge queues try to buy the same guarantee back with parallelism: build the candidate futures, main+A, main+A+B, main+A+B+C, test them concurrently, and discard a failure from the middle. OpenStack’s Zuul has gated that way since 2012; GitHub’s merge queue ships a version; and bors, which lands Rust, does the cheaper batch-and-bisect variant. It works, and it works by brute force, which I will come back to.

Shared Environments

One port range, one development database, one mock server.

I do have a script assigning each worktree its own ports by hashing the directory name (the right idea), and it hashes into 99 slots. With dozens of worktrees on disk the birthday math makes a collision effectively certain, and two agents get handed the same port whenever the colliding pair happens to be running at once.

Undivided Work

Trivial parallelism doesn’t work on a problem that was never split into non-conflicting units. The agents were pointed at one jammed queue and found their own boundaries, which were mostly the same boundaries.

What looked like a coordinator was a process manager: spawn, wait, list, follow up, six hundred and thirty-three tool calls of overhead, all of it without any deep context about which pieces of work actually interacted. A coordinator that only manages agents can’t divide work it does not understand, so the children rediscover the same failures, edit the same new scripts, and invalidate one another’s assumptions. Codex said as much afterward: the agents generated work for the coordinator instead of removing work from the critical path.

Classified by root cause, 245 of the 361 entries collapse into ten systemic problems, 73 are one-off product bugs, and 43 I could not classify. Here are the ten, because a taxonomy whose majority is invisible isn’t a taxonomy:

systemic root cause	incidents
tooling the session was writing that same day	87
no per-agent workspace isolation	33
errors skipped, swallowed, or reported as success	32
affected-spec selection far too broad	20
migration order versus the sealed baseline	16
tests leaning on shared fixture state	16
recovery work that would not converge	12
CI cost structure	12
evidence produced against a base that moved	10
verification run without its dependencies	7

I would defend the shape of that table rather than the ratio: workers under a shallow coordinator rediscover the same things, often enough to dominate a log.

Look down the column and the majority is exactly that failure. The top row alone (tooling the session was writing that same day, 87 incidents) is agents redoing and colliding on the same work, because nothing holding the session had enough context to keep them off each other’s toes. Add isolation failures and swallowed errors and the top three account for 152.

Those are not ordinary defects that concurrency merely made frequent. They are the artifact of coordination without context: a parent counting agents instead of understanding the work, so that the children rediscover and rewrite the same things, file the same findings, and hand the parent the job of reconciling them. The seal and the port hash still have boring mechanical fixes, and recovery-that-would-not-converge and CI cost structure are process problems. The rediscovery is neither.

The one that is irreducible in a different sense is the sixth row: tests leaning on shared fixture state, or green(A) and green(B) failing to imply green(A+B) in ordinary clothes.

It accounts for sixteen of the 361, tied with the seal for the smallest systemic bucket, and I want to put that number in front of you instead of letting you find it by subtraction. Sixteen out of 361 is four percent.

So that substrate problem is the rarest thing that happened that week. It is also the only one where catching it means building the composition and running the entire suite against it, which I do, serially, late, and which is precisely the expensive thing. It is therefore the only one on the list where I can’t tell you what an affordable fix would even look like. The common problem looks much simpler: parallelism without a division of the work, supervised by something that was counting agents instead of understanding them.

Except that the two are the same problem, and it took me most of a week to see it. What a coordinator needs in order to divide work is exactly a read/write set per unit, which is the primitive this whole post is about, one layer up. Mine had no way to compute one, so it couldn’t divide anything, so eighty-four agents went out and found the same boundaries and did the same work on top of each other. That’s the top two rows of the table, a hundred and twenty entries, a third of everything I logged. I’m counting two rows and not the three above, because swallowed errors are ordinary bad error handling and belong to a different complaint.

So the four percent isn’t a rare problem I’ve chosen to write about instead of the common one. It’s the identical question, “what does this thing read, and what does it write,” asked about a code change instead of about an agent. At the code layer the question is open, which is the research problem. At the agent layer it’s answerable in principle, and my runtime simply didn’t ask, which is the more embarrassing half and the one my week actually went to.

The Missing Primitive

So what would it take for some layer to detect any of this?

Start with why detection is hard. One pull request writes a row to show_attendance; another reads a count over it. Neither diff says so. The write goes through an HTTP handler, an ORM, and some SQL; the read goes through the app. The conflict is a property of what the code does when it runs, and it’s not present in the text.

That leaves two ways to find it:

1. Execute the composition and observe what breaks.
2. Compute the effects of each change and compare them without running anything.

Nobody can do the second precisely enough at the granularity where my conflicts live, so everybody does the first. If you can’t compute effects, you can’t tell which pairs need checking, and so you pay for a full verification of every candidate future you are willing to buy.

Uber’s SubmitQueue needed a probabilistic model of which changes would land in order to prune the speculation tree, and that is less an optimization than a confession: you only guess your way through the tree when nothing can tell you which branches matter.

So the state of the art converts an unsolved program-analysis problem into a compute purchase. That is rational at Google’s scale and precisely the wrong trade at mine, because speculation costs queue depth times full suite times parallel environments, and agents raise queue depth. The sanctioned fix for the correctness problem is more CI spend, scaling with the exact variable agents just multiplied.

My two complaints turn out to be one complaint. “CI is too expensive” and “composition failures show up only when the serial queue retests them” are the same missing primitive seen from two sides.

Here is the size of it in my own logs:

• Over three days, 510 CI runs were spent against 111 opened and 100 merged pull requests: roughly five full verifications per change that landed.
• The runs attach to opened pull requests rather than merged ones, so read that as an order of magnitude and not a rate.
• Of the 510, 112 failed or were cancelled outright, and cancellations are usually a new push superseding an old one rather than an escape.
• Deleting every one of those 112 takes the multiplier from 5.1 to 4.0, not to 1.
• The rest is ordinary iteration, some fraction of it rebasing onto a main that moved underneath, which is the same problem in ordinary clothes.
• The primitive is therefore worth about a quarter of the excess, and only if every one of those failures turns out to be a composition escape, which I cannot show. A serial queue stays slow either way.

Databases avoid all of this for one reason. A transaction declares its read/write set. The system knows what each transaction touched, so it can detect conflicts, order what must be ordered, and let everything else proceed.

That question (which operations may run without coordination) has an exact answer in the literature. Bailis and colleagues named it invariant confluence : a set of operations is I-confluent with respect to an invariant precisely when merging any two invariant-preserving reachable states that share a common ancestor yields another invariant-preserving state.

This is a necessary and sufficient condition for running those operations coordination-free while maintaining the invariant, not a heuristic. If your workload is I-confluent you can run coordination-free and stay correct; if it is not, coordination is a requirement, and no merge function will save you.

The result does not transfer to code cleanly, and it’s worth saying where it breaks. I-confluence is defined over a merge operator on states, with reachability and a fixed invariant, whereas for code changes there is no defined merge over program states, and the invariant here is the test suite, which every merge rewrites. What I’m borrowing is the shape of the question, not the theorem.

Source code carries a partial declaration at best. Build systems get closest: Bazel and Nix require every target to declare its inputs and outputs (a real read/write set at file granularity), and that is how affected-target computation works across enormous monorepos.

However, file granularity is the wrong granularity here. My two colliding pull requests touch entirely disjoint files. A build graph would call them independent, and a build graph would be right, and they would still destroy each other, because the thing they share is a row in a database that neither one names.

At the granularity where my conflicts live, nothing declares those effects. So there is no conflict detection there, and without it you can’t tell which regime a pair of changes is in, and if you can’t tell, the only safe policy is to assume the worst and coordinate everything.

What is the runtime read/write set of a code change, and can it be computed precisely enough, and cheaply enough, to decide which changes are safe to integrate concurrently?

I’ve already built a bad version of this without recognizing it. My affected-spec selector maps a change to the tests it could affect, so that CI only runs what is relevant. That’s a read-set estimator, and its failure mode is the research problem in miniature: it traced a one-screen change through a shared UI component, concluded the change could affect anything, and scheduled all 115 test files.

An over-approximation collapsing to “conflicts with everything” is the same degenerate case as a lock covering the whole table, or as serializing every update to a dictionary even when the changes are to disjoint keys. Sound, and useless.

One objection here is that I am demanding a soundness nobody needs. Large organizations run test selection that is deliberately imprecise and absorb the escapes: changes that should have been tested, were not, and break trunk after they land.

It’s tempting to say I can’t absorb escapes because my blast radius is smaller, but that has it backwards: an escape into Google’s trunk blocks far more people than an escape into mine. What they have, and I don’t, is containment. Post-submit continuous builds, automated culprit-finding, and rollback tooling all bound what an escape costs.

I have none of that. What I have instead is a queue of agents that respond to a red trunk by independently rediscovering the failure, each one paying for its own full CI run, none of them asked to, and none of them aware that eleven others are doing exactly the same thing at exactly the same moment. Escapes are cheaper for Google not because fewer people are blocked, since more are, but because they built the machine that catches and bounds the damage. Without that machine, imprecise selection is a cost I can’t absorb.

Prior art exists, and none of it quite lands:

Requirements for Coordination-Free Development

Coordination-free multi-agent development, if it’s possible at all, needs solving at two layers, in order. Getting the second right while the first is broken buys nothing.

The environment has to permit it first. Separate databases, working directories, port ranges, and instances of every service, per agent, enforced rather than attempted. Writers in one checkout contend regardless of how elegant your merge semantics are, and I still do not know how many of those 84 were writers.

Large companies solved this a while ago, and the shape of the fix isn’t subtle: make every developer’s environment a faithful copy of the one that builds and ships the product. Either move development onto cloud machines provisioned from the same definition as CI, or reproduce that environment in miniature on the laptop, with a container per service, a database per test, and a toolchain pinned to a lockfile. Somebody owns that, full time. The cost of getting it wrong is paid once, by a platform team, instead of every day by everyone else.

I don’t have a platform team, and neither does anyone else building alone with agents. An independent developer can now reach a working prototype absurdly fast, faster than at any point in my career, and then hit the wall this whole post is describing: one environment, artifacts that assume one writer, and no organization to absorb the difference. Getting off the ground is close to solved. Staying up is not, and I don’t have a good answer for it.

That’s the same argument I made a few months ago about civil engineering, coming at it from the other side. When construction gets cheap, design is what fails, and the design here is the substrate: what runs where, what is isolated from what, and what has to be serialized. An agent isn’t going to do that for you, and it won’t show up in the prototype.

Then the artifacts have to stop requiring a coordinator. A total order needs somebody to assign it, and in the single-writer world the assigner was free: migration numbers went up because one person added one at a time. Put N agents on it and the assigner is gone. You can reintroduce one as a central coordinator handing out positions, but a central coordinator is coordination, and it caps throughput at one.

Two mainstream designs exist and always have: a total order by timestamp, as Rails has done since 2008, and a dependency graph, as Django and Alembic do. Under one writer the choice barely matters. Under many it matters enormously, because a total order needs an assigner and a graph doesn’t. My agent picked the total order.

Nobody ever weighed the two, and that’s the part worth sitting with, because it isn’t really about migrations. I ratified the timestamp scheme myself, back in March, and what I ratified was a fix for filename collisions : that was the problem in front of me, the fix solved it, and so I shipped it and wrote a post about it without once asking what the scheme would do under load.

An agent proposed it, an agent reviewed it, and I signed off on the part I understood. Review is one of those protocols that terminates in a person, and what the person is supposed to supply is an independent prior. Neither reviewer had one: not the agent, which reached for the most common pattern, and not me, because I was evaluating a collision fix and never saw an ordering decision. The better option was documented, shipped in two major frameworks, and one search away.

This is what vibe coding actually costs, and it’s not bad code. The code was ordinary and defensible. What is missing is the one question nobody asked, because asking it requires somebody who has been hurt by the answer before, and I had removed all of those people from the loop on purpose.

Django and Alembic don’t solve it either, and the reason is the granularity problem again. A Django dependency is an edge to another migration : this one runs after 0042_add_venue, which says nothing about what either one reads or writes. Two migrations with no declared edge are unordered, so if one adds a column and the other’s UPDATE reads it, that pair conflicts, no edge exists, and nothing detects it. The DAG removed the global sequence and left the effects undeclared.

And when two branches produce divergent heads, the resolution is a merge revision: a node with two parents that doesn’t verify the branches are compatible. It records a human’s assertion that they are, the same protocol terminating in the same person, one layer down.

So the proposal isn’t “invent a DAG.” The DAG exists. It is to make the edges effect-based instead of identity-based: know that this migration writes show_attendance and reads shows.date, and derive the ordering constraints and the genuine independence from that.

The objection is that this relocates the labor, and “somebody declares it” across 84 agents is not a plan. For general code that lands. For migrations it doesn’t, because the effects are already in the text: SQL names its objects out loud, so parsing ALTER TABLE show_attendance ADD COLUMN or UPDATE shows SET ... WHERE date > ... yields the read set and the write set with no human, no annotation, and no agent remembering. All 1,388 of mine are plain .sql, and the parsing is commodity, since sqlglot does column-level lineage today (which has not stopped dbt, in the very ecosystem where effects are most extractable, from building its graph out of ref() and source() macros written by hand).

Two caveats. Disjointness only rules out conflicts through the objects a migration names, so an invariant spanning two tables can still break when two migrations touch one table each: the name-and-length record, one level down. And my sixteen seal incidents have a two-line fix with no research content in it. The case rests on the class rather than on those sixteen, because any ordering constraint several writers must satisfy independently has this shape.

The hard case is the handler-through-ORM-through-SQL path, where the effects genuinely aren’t in the text. The easy case has been sitting in my repository the whole time, unbuilt.

None of that is the CRDT move. CRDTs design a data type so that concurrent operations commute by construction; this is optimistic concurrency control, which extracts read/write sets and coordinates only the pairs that overlap. Designing for commutativity means defining the data type, and for a software system in general there is nothing to define.

For migrations there is, and it may be the better path. Restrict the schema to additive, idempotent DDL: add columns nullable, never rename in place, never drop anything, and guard every statement so that applying it twice is the same as applying it once. Under that discipline the schema is a grow-only set and every migration is an inflation, so any two of them commute and the order stops mattering. No assigner, no extraction, no DAG. The bill arrives at contraction, when you do finally want the column gone, and that step commutes with nothing, so it gets batched and performed deliberately by somebody who is paying attention. Which means the two proposals aren’t really competing: commutativity by construction covers the additive majority and needs no extraction at all, and extracting read/write sets earns its keep on the contraction step and on everything that isn’t a migration.

The discipline isn’t exotic. It’s what microservices already force on you, because there you cannot control rollout order either: two versions of a service run at the same time, so every change has to be tolerable to the version that hasn’t shipped yet and to the one already deployed. Expand, migrate, contract. It is well understood, it is genuinely expensive, and it is one of the things practitioners are worst at, because it requires holding in your head a version of the system you are not looking at.

Agents are worse at it than we are. An agent told to rename a column renames the column, because that is the direct expression of the intent and it is correct in every example it has ever seen, all of which were written in a world with one writer and one deploy. The three-step version is more code, it looks redundant at the moment it’s written, and its entire justification is a concurrency property that appears nowhere in the diff. Which is this post again: the discipline exists, it works, and nothing left in the loop has the prior that would make anybody reach for it.

The same argument applies to source code and lands somewhere less comfortable. Git’s answer to two agents editing one file is to linearize, rebase the second onto the first, and resolve the conflicts: a coordination protocol with a human standing at the conflict point. It works because textual conflicts are rare enough and humans read them well. Both halves are under pressure now: agents collide constantly, and the human who was supposed to adjudicate is increasingly not reading the diff.

Which raises the question I keep arriving at and don’t love. Text files, line-based diffs, and sequential human review are a human interface. A structural representation, where merge is defined over the program’s structure instead of over lines, where a change declares what it reads and writes, and where invariants are attached to the artifact and checked mechanically, is a far better substrate for many concurrent writers.

People have proposed exactly that for decades and it mostly didn’t take, not merely because developers like text, though they do, but because text is the interchange format that every other tool in the ecosystem already agrees on (diff, blame, bisect, grep, review, CI logs, and every static analyzer ever written), so that replacing it means replacing all of them, per language, before you get any benefit at all.

The exceptions are the argument rather than an exception to it. Darcs and then Pijul built patch algebras where independent changes provably commute (the CRDT move applied directly to source control), and Unison stores a content-addressed syntax tree with no text merge at all. They exist, they work, and almost nobody uses them.

The tempting move is to say that agents change that calculus, because the reader has left the room and the ergonomics objection leaves with them. I do not think that is safe yet.

Language models are the most text-dependent writers we have ever built: trained on text diffs and line-oriented source, and currently worse at manipulating syntax trees than at manipulating the text I would be proposing to abandon. The party being served may not be human, but it learned to code by reading us. Whether that is a durable property of these systems or an artifact of what we happened to train them on is the load-bearing question, and it’s not one I can answer from a migration log.

Takeaways

So here is what I don’t know, gathered in one place rather than dropped along the way.

Whether a runtime read/write set can be computed precisely enough to be worth having. Whether text is a contingent fact about today’s models or a durable one. And, most uncomfortably, what the specification is once humans stop reading code: if the test suite becomes the only surviving statement of what the system should do, and the suite is also written by agents, then attaching invariants to the artifact and checking them mechanically relocates the trust problem rather than solving it.

That last one is a different complaint from the rest of this post. I’ve been arguing that the suite gets run against the wrong artifact. That’s the worry that it is the wrong artifact.

The short-term fixes are unglamorous and I’m doing them anyway: a merge queue with a concurrency limit of one, a worktree with its own port range and database per agent, a cap on how many subagents a session may spawn, and one incident per root cause instead of one per symptom. That is admission control and resource isolation, plus better bookkeeping. None of it is research, and none of it touches the verification multiplier, the part that needs something nobody has.

Codex could describe all of this afterward, lucidly, and could answer none of it beforehand. It spawned 84 workers anyway, which is less a mistake than a preview.

My setup is deliberately extreme, and a team that reviews its diffs and never let an agent design its schema layer will see a fraction of what I saw. They will still see some of it. The substrate underneath is the same one, and it still assumes there is one of you.

If you are working on conflict detection, effect extraction, or merge queues that don’t cost a fortune, or if you have hit this same wall from the other direction inside a large organization, please reach out. I would very much like to be told that some part of this is already solved.

Three open questions is a worse ending than one answer. It’s the honest count.

---

This is part of an ongoing series about buildingZabriskie with AI agents. Previously: The Test Suite Was the Incident, The Tax on the Happy Path, and Multi-Agent Systems Have a Distributed Systems Problem.