五个月：把 Bug 当患者，把编码 Agent 当医疗团队

4 月的一个周三傍晚到周五下午之间，MOLT——Cockroach Labs 为将数据库迁移至 CockroachDB 而提供的工具——获得了从 IBM Db2 数据源进行迁移的能力。

Db2 是最早实现商业化的关系型数据库之一，拥有丰富的 SQL 方言、复杂的类型系统和原生线缆协议。为 MOLT 提供 Db2 支持，意味着需要新的 schema 转换器、用于拉取行数据并将其写入 CockroachDB 的新 Fetch 路径、用于在数据加载后进行比对的新 Verify 路径、一个供应商引入的 ANTLR 语法、用于 CI 的 Docker 镜像，以及超过一万行测试夹具。2024 年我们为 MOLT 工具增加 Oracle 支持时，对等工作耗时 9 个月，工程时间成本约为 16 万美元。

Db2 支持耗时不到两天，而且没有任何代码由人类编写。Token 账单为 4172 美元：速度快 164 倍，成本低 38 倍。

一切始于一条描述需求的 GitHub Issue：

随后，一个规划 Agent 阅读了该 Issue，判断其规模过大、不能整体处理，并将其拆分为 15 个子 Issue，同时建立了明确的依赖图：先是基础设施，然后是类型系统、行迭代器、Fetch、Verify、Convert、CI，最后是测试数据。其中两个子 Issue 在各自的工作分析中也被判定过大，因此又被进一步拆分。在此过程中，Agent 还针对自身工作提交了十余条 Issue，包括测试夹具缺口、一个类型映射 Bug，以及隔离级别修复。到父 Issue 关闭时，其下共创建了 32 个子 Issue，合并了 27 个 Pull Request，审查者将工作退回 55 次，9 个 Issue 被升级处理，其中 2 个升级给了人类。以 PostgreSQL——我们测试最充分的方言——为基准衡量，测试覆盖率达到同等水平。

但这一切究竟是如何发生的？背后的驱动力是什么？更重要的是，什么确保了最终产物具有高质量？在为 Db2 支持创建 Issue 之前，我们先构建了一所编码医院，称之为 MOLT Sinai。MOLT 是CockroachDB 的迁移工具集名称；一旦决定以医院为模型设计这条流水线，MOLT Sinai 这个双关名称就好得令人无法拒绝。这是因为 Mount Sinai 是多伦多和纽约都颇具影响力的教学医院，而这两座城市正是我们居住的地方。这个名字沿用了下来，对应的词汇体系也同样保留：Issue 是患者，合并是出院，负责的人类则是医疗主任（Chiefs of Medicine）。

Agent 快速编写大量代码并不是这个故事独有的现象；软件工厂正在整个行业中不断出现，例如 OpenAI 的这一项目。我们案例的独特之处在于：这些 Agent 与 main 分支之间设置了什么、我们为何将其构建成医院的样子，以及让它处理真实工作五个月后发生了什么。

为什么是教学医院？

编码 Agent 的问题在于：尽管它们完全有能力编写代码，但人们很难相信它们能产出可用于生产环境、经过充分测试、易于维护且符合预期范围的代码。运行 Agent 的成本相对较低，但当它们的代码将交付到客户手中时，这一点并不能带来多少安慰。与其在向客户迁移至 CockroachDB 的过程中写入错误数据，我们宁愿让 Agent 花更长时间运行——即使是 10 倍时间。速度固然重要，但质量重要得多。

过去一年中，已经出现了多种尝试自动化 Agent 编码的方案，其中许多针对吞吐量进行了优化。例如，Gas Town 会让许多 Agent 并行处理一个代码库，并通过合并处理层解决冲突。当瓶颈在于代码编写速度时，这是一种不错的设计。Cursor 还有一个重建 SQLite 的实验，其速度之快，以至于他们需要自行编写版本控制系统来处理大量变更。这是一个有趣的实验，但显然并不是为向用户交付高质量数据库代码而设计的。在绝大多数早期 Agent 自动化实验中，速度是主要驱动力。

我们的动机不同。我们构建的是分布式 SQL 数据库，以及一套将工作负载迁移到该数据库的工具。一处迁移工具中的细微 Bug，可能会在数据进入一个理应将正确性置于一切之上的数据库时损坏用户数据。因此，我们关心的问题不是“每小时能完成多少个 PR？”，而是“其中有多少 PR 如果由我们亲自合并，会令我们感到尴尬？”经过这样的重新定义，我们显然不能采用一个以吞吐量为优化目标的 Agent 群体模型。

因此，我们寻找了一种已经长期管理高风险工作的模型：它处理由能力参差不齐的人员完成的工作，要求强制交接、强制第二意见，并且有关于交接疏忽会导致何种问题的研究文献。我们最终选择了教学医院。

这个比喻赋予了我们如下角色：

在这个核心周围，还配备了你在医院中可能见到的其他参与者——或者如果你看过《The Pitt》，也会觉得熟悉：每 30 分钟巡查一次、寻找卡住患者的护士长（Charge Nurse）；当 main 分支出问题时可以封锁整栋医院全部工作流的感染控制 Agent（Infection Control）；每周撰写流程审查报告并组织事故后复盘会议的安全部门（Safety Department）；以及提出新工作的研究部门（Research Department）。

我们并不声称这种模式比 Agent 群体更便宜，或比单一 Agent 更简单。但它确实能产出比两者都更高质量的产品。当你身处数据库行业，就会愿意为这种质量付费。正如下文所示，它也带来教学医院中一些令人遗憾的方面——官僚流程、等待时间和成本——但我们相信，这些都是值得接受的权衡。

如何构建？

医院中的一切都运行在 GitHub Actions 上。每位患者的状态保存在其 Issue 的标签中，也保存在医院为跟踪每位患者进度而创建的一些临时文件中。

每个阶段都是一个 GitHub workflow，当其“label in”被添加到 Issue 时启动。Agent 完成工作后会写入一个“label out”，而这个标签便是下一阶段的 label in。虚线箭头将工作送回前序环节：被拒绝的计划退回工作分析阶段，CI 运行失败或出现修改请求时退回治疗阶段，而卡住的阶段则升级给医疗主任。

图中未展示的是：四个部门按各自的时间表在此流程之外运行。护士长会重新运行停滞的阶段；当 main 分支出现问题时，感染控制部门会封锁医院；研究和安全部门则会创建新的 Issue 和政策 PR，它们和其他患者一样，经由医疗主任进入流程。

添加标签会触发一个 GitHub workflow，该工作流加载特定角色的 Prompt，使 Agent 以医院员工的身份开展工作。完成工作后，Agent 会向包含该 Issue 病历的文件添加一条结构化记录，并添加流程中的下一个标签。记录中带有诸如 <!-- SINAI:TREATMENT_PLAN --> 的哨兵标记，以便后续 Agent 可以找到并解析它。

编码在技能文件中的几条基础医院规则，提供了大部分安全保障：

没有计划就不能写代码，没有审查就不能执行计划。 在 Fellow 修复任何问题之前，必须先复现问题，进行鉴别诊断——借用医院术语——执行能够缩小问题范围的定向调查，并提交治疗计划：诊断、需要修改的文件、需要新增的测试、风险和待解决问题。审查主治医生（Review Attending）会由另一个 Agent 实例担任；其 Prompt 围绕发现问题而非实施修复构建。该 Agent 会阅读计划，并予以批准或拒绝。只有这样，治疗才能开始。我们加入这一关卡，是因为一个自信地实施错误修复的 Agent，浪费的时间和 Token 远多于停下来检查计划的 Agent。

不要即兴发挥。 如果治疗期间范围发生变化，Fellow 必须停止，提交修订后的计划，并重新进入计划审查。技能文件中的指令只有两个词："Don't improvise."（不要即兴发挥。）

不走捷径，尤其不能在测试上走捷径。 Agent 不得为了让测试通过而禁用、跳过、削弱或修改测试。如果测试失败，在被证明有误之前，测试就是正确的。如果 Agent 确实认为测试有问题，对测试的修改必须单独成为一个 commit，基于其自身理由进行论证，并与修复工作分开审查。同样，一个没有实际覆盖变更内容的通过测试，比没有测试更糟糕。所有代码审查都从禁用该变更并确保所有新增测试失败开始。如果新增测试没有失败，那么它们要么超出范围，要么就是不正确的。

绝不盲目挣扎。 当 Fellow 卡住时，它不会只是更努力地尝试。它会编写一份 I-PASS（Illness severity、Patient summary、Action list、Situation awareness、Synthesis，即病情严重程度、患者摘要、行动清单、情境认知、综合确认）交接记录。这是一种直接借鉴自教学医院的结构化格式，随后再升级给 Attending。接手的 Agent 必须在开始工作前写下自己对交接内容的理解。医院中关于 I-PASS 的研究显示，如果医生和护士以标准化方式记录患者信息，交接班期间的不良事件可减少 50%。我们的 Agent 遵循同样的协议。

出院护士审计审查者。 合并前的最后一道关卡不会再次审查代码，而是核实审查是否被正确执行：是否存在批准记录、审查模板是否填写完成、是否存在未解决的讨论线程、CI 是否通过、commit 历史是否干净。它会发布一份清单，准确说明自己核实了哪些内容；如果有任何项目无法勾选，出院就会失败，未勾选项会被保留并附带说明。之所以制定这一规则，是因为当你把控制权交给 Agent 时，需要确保它们确实在遵从指令，而有时它们并没有。

在打扰医疗主任前，先查阅先例。 人类医疗主任作出的、可推广适用的每一个决策，都会记录在一个只能追加的先例日志中。希望升级问题的 Agent 必须先阅读日志；如果存在适用先例，它就应遵循并引用该先例，而不是升级。只有人类可以创建先例。五个月以来，这一机制减少了 Agent 向医疗主任升级问题的频率。

它交付了什么？

开始开发 MOLT Sinai 时，我们创建了构建 MOLT 工具的代码仓库镜像。这个镜像随后成为试验场，用于测试医院模型的效果。以下是在该仓库中，从 4 月 21 日至 9 月 11 日期间发生的情况：

在不到 5 个月时间里，医院流水线合并了超过 100 万行代码，并将变更拆分为足够小的单元，以便 Agent 可以有把握地进行审查。

你可能注意到了上表中一个耐人寻味的数字：1299。仓库中提交的 Issue 里，近一半是医院为医院自己创建的：拆分出的子 Issue、Fellow 从计划中划出范围的后续事项、研究部门提出的建议，以及安全部门制定的纠正措施。流水线会生成自己的待办事项列表。人类决定接收哪些工作，但从第二个月起，人类就不再是工作的主要来源。

虽然 Db2 支持是我们首个通过这所医院处理的大型功能，但那只是一次实验。在自主运行第一周后，我们开启了“人工批准模式”（human approval mode），要求每次合并都必须经过人工审阅。只有在这一模式下，我们才能确信代码质量足以交付给客户。在医院具备这层额外安全保障后，我们开始着手真正为医院而创建的工作：构建Migration Assistant。这是一个基于 AI 的工具，可引导用户完成从 Postgres 到 CockroachDB 的完整迁移，包括 schema 转换、数据加载与验证，以及例行转换。Migration Assistant 目前已针对 Postgres 迁移进入预览阶段，且几乎完全由医院中的 Agent 员工编写。

它花费了多少？

自 4 月开始运行 MOLT Sinai 以来，它已消耗略高于 13.5 万美元的 Claude Token，平均每位患者——即每个 GitHub Issue——约为 84 美元。Issue 通常会在医院中停留一到两天，时间往往主要消耗在等待人工审查者上。就 LLM 模型而言，医院也随着模型进步而不断演进。

最初，流水线的每个阶段都运行 Opus 模型；但随着 Fable 的推出，现在它会使用 Fable 进行规划，并由 Fable 生成的计划决定实现阶段应使用的模型。如果实现在 Sonnet 或 Opus 模型上遇到困难，则可以升级使用 Fable 模型。这有助于控制成本，同时仍可在必要时使用更昂贵、能力也更强的模型。

如上图所示，每次切换到后续模型时，都需要进行一些工作来调整 Agent 技能，以优化成本。我们每隔几周就进行连续的优化轮次，这也是图表走势呈现波动的原因之一。还有其他原因，例如某一时期的工作组合——UI 任务成本更高，因为 Agent 必须处理截图和其他视觉数据——以及 CI 环境中的一些特殊情况；但这些细节不适合在一篇入门博客文章中展开。

医院自己构建了自己！

创建 MOLT Sinai 时，我们将最初的一组 workflow、技能和 Prompt 直接构建到镜像仓库中。不久之后，我们意识到了所构建内容的价值……

[原文至此截断]

Between a Wednesday evening and a Friday afternoon in April, MOLT, the tool Cockroach Labs ships for migrating databases to CockroachDB, gained the ability to migrate from IBM Db2 sources.

Db2, one of the first commercially available relational databases, has a rich SQL dialect, a complex type system, and a native wire protocol. Supporting it in MOLT meant a new schema converter, a new Fetch path for pulling rows out and getting them into CockroachDB, a new Verify path for comparing them once they were loaded, a vendored ANTLR grammar, a Docker image for CI, and more than ten thousand lines of test fixtures. When we added Oracle support to the MOLT tools in 2024, the equivalent work took 9 months, and cost around $160K in engineering time.

Db2 took less than two days, and no human wrote any of the code. The token bill was $4172: 164x faster, and 38x cheaper.

It all started with a single GitHub issue describing the ask:

A planning agent then read it, decided it was too large to treat all at once, and decomposed it into fifteen sub-issues with an explicit dependency graph: first the foundation, then the type system, then the row iterator, then Fetch, then Verify, then Convert, then CI, then test data. Two of the sub-issues were judged as too large in their own workups and were decomposed again. Along the way the agents filed another dozen issues against their own work: fixture gaps, a type-mapping bug, and an isolation-level fix. By the time the parent issue closed, thirty-two sub-issues had been opened under it, twenty-seven pull requests had merged, reviewers had sent work back fifty-five times, and nine issues had been escalated, two of them to a human. Test coverage, as measured against PostgreSQL (our best-tested dialect) was equivalent.

But how did all of this happen? What was driving this, and more importantly, what was ensuring that what came out the other side was high quality? Before we opened the issue for Db2 support, we first built a coding hospital which we call MOLT Sinai. MOLT is the name of CockroachDB’s migration tool set, and once we had decided to model the pipeline on a hospital, MOLT Sinai was too good of a portmanteau to resist. That’s because Mount Sinai is a prominent teaching hospital in both Toronto and New York - the cities in which we live. The name stuck, and so did the vocabulary. Issues are patients. Merging is discharge. The humans in charge are the Chiefs of Medicine.

Agents writing a lot of code quickly is not unique to this story, as software factories are popping up all over the industry (like this one at OpenAI). What’s unique to our case is what stood between those agents and main, why we built it to look like a hospital, and what happened when we ran it on real work for five months.

Why a teaching hospital

The problem with coding agents is that although they’re more than capable of writing code, it’s hard to trust them to produce production-grade code that is well-tested, maintainable, and within the desired scope. Agents are relatively cheap to run, but that's little comfort when their code is going to land in customers' hands. We'd much rather have the agents take longer to run (even 10x longer) than land the wrong data in a customer migration to CockroachDB. Speed matters, but quality matters much much more.

Several attempts to automate agent coding have appeared over the last year, many of which have been optimized for throughput. Gas Town, for example, runs many agents in parallel against a codebase with a merge-handling layer to sort out collisions. That is a good design when the bottleneck is how fast code gets written. And there’s this experiment from Cursor in which they rebuilt SQLite so quickly that they needed to write their own version control system to handle the change volume. It’s an interesting experiment, but clearly it’s not designed to ship quality database code to users. In the vast majority of early agent automation experiments, speed was the main motivator.

Our motivation was different. We build a distributed SQL database, and a suite of tools to migrate workloads to that database. A subtle bug in a migration tool could corrupt someone's data on the way into a database that is supposed to maintain correctness above all else. As a result, the question we cared about was not "how many PRs per hour?" but rather, "how many of those PRs would we have been embarrassed to have merged ourselves?" With that reframing it was clear that a throughput optimized swarm of agents wasn’t a good model.

So we looked for a model with long experience managing high-stakes work done by staff of uneven ability, with mandatory handoffs, mandatory second opinions, and literature on what goes wrong when the handoffs are sloppy. And we landed on a teaching hospital.

The metaphor gave us these roles:

Around that core sit the other actors you would expect to see if you’ve ever been inside a hospital (or have watched The Pitt): a Charge Nurse who rounds every thirty minutes looking for stuck patients, an Infection Control agent that can lock down every workflow in the building if main breaks, a Safety Department that writes weekly process reviews and runs post-incident conferences, and a Research Department that proposes new work.

We don’t claim that this is cheaper than a swarm or simpler than a single agent. It does however produce a higher-quality product than either. When you’re in the database business, you’re prepared to pay for that quality. As you’ll see below, it also brings some of the regrettable aspects of teaching hospitals (bureaucracy, wait times, and cost) but we believe that those are worthwhile tradeoffs.

How it’s built

Everything in the hospital runs on GitHub Actions. The state of every patient lives in the labels on its issue, as well as in some scratch files that the hospital creates to keep track of each patient’s progress.

Each stage is a GitHub workflow that starts when its “ label in” lands on the issue. When the agent finishes, it writes a “ label out” , and that label is the next stage's “label in”. Dashed arrows send work back: a rejected plan to workup, a red CI run or requested changes to treatment, and a stuck stage up to the Chief.

Not shown: four departments run outside this flow on their own schedules. The Charge Nurse re-runs stalled stages, Infection Control locks the hospital down when main breaks, and the Research and Safety departments file new issues and policy PRs, which enter through the Chief like any other patient.

Applying a label fires a GitHub workflow which loads a role-specific prompt so that the agent acts as an employee in the hospital. Once it’s completed its work, the agent adds a structured note to the file containing that issue’s chart, and applies the next label in the flow. The note carries a sentinel like <!-- SINAI:TREATMENT_PLAN --> so that later agents can find and parse it.

A few basic hospital rules, encoded in the skills, provide most of the safety:

No code before a plan, no plan without review. Before the Fellow fixes anything, it has to reproduce the problem, perform a differential diagnosis (to use hospital terminology), run whatever targeted investigation narrows it, and post a treatment plan: diagnosis, files to change, tests to add, risks, open questions. A Review Attending, a different agent instance running a prompt built around finding fault rather than fixing, reads the plan and approves, or rejects it. Only then does treatment start. We added this gate because an agent that confidently implements the wrong fix wastes far more time and tokens than one that stops to check its plan.

Don't improvise. If scope changes during treatment, the Fellow stops, posts a revised plan, and goes back to plan review. The instruction in the skill file is two words: "Don't improvise."

No shortcuts, especially around testing. An agent may not disable, skip, weaken, or modify a test to make it pass. If a test fails, the test is correct until proven otherwise. If the agent genuinely believes the test is wrong, the change to the test has to be its own commit, justified on its own merits and reviewed separately from the fix. Similarly, a passing test that does not actually exercise the change is worse than no test at all. All code reviews start by disabling the change and ensuring that all new tests fail. If they don’t they’re either out of scope, or incorrect.

Never flail. When a Fellow is stuck, it does not try harder. It writes an I-PASS (Illness severity, Patient summary, Action list, Situation awareness, Synthesis) handoff, which is a structured format borrowed directly from teaching hospitals, and then escalates to an Attending. The receiving agent has to write back what it understood of the handoff before it starts working. Studies of I-PASS in hospitals report a 50% reduction in adverse events during shift changes if physicians and nurses write down patient information in a standardized way. Our agents follow the same protocol.

The Discharge Nurse audits the reviewer. The last gate before merge does not re-review the code. It verifies that the review happened properly: an approval exists, the review template was filled out, no threads are unresolved, CI is green, the commit history is clean. It posts a checklist showing exactly what it verified, and if any box cannot be checked, discharge fails with that box left unchecked and annotated. The rule exists because when you’re turning over control to agents, you want to ensure that they’re doing what they’re told, and sometimes they aren’t.

Check the precedents before bothering the Chief. Every decision the human Chief makes that generalizes, is recorded in an append-only precedent log. An agent that wants to escalate has to read the log first, and if a precedent applies, it follows the precedent and cites it instead of escalating. Only a human can create precedents. Over five months, this has cut down how often agents escalate to the Chief.

What it shipped

When we started working on MOLT Sinai, we created a mirror of the repo in which the MOLT tools are built. This mirror then became a proving ground where we could test the efficacy of the hospital model. Here’s what transpired in that repo between April 21 and September 11:

In just under 5 months, the hospital pipeline landed well over a million lines of code, split into changes small enough that they could be confidently reviewed by agents.

You may have noticed one curious number in the table above: 1299. Nearly half the issues filed in the repository were created by the hospital for the hospital: decomposition children, follow-ups the Fellow scoped out of a plan, proposals from the Research Department, corrective actions from the Safety Department. The pipeline generates its own backlog. A human decides what gets admitted, but the humans stopped being the main source of work in the second month.

While Db2 support was the first large feature we put through the hospital, that was just an experiment. After its first week of autonomous operation, we turned on “human approval mode”, which requires every merge to have a set of human eyes pass over it. It was only in this mode that we could be certain that the code was high enough quality to ship to customers. With the hospital having this extra safety layer, we went to work on what we really created the hospital for: building the Migration Assistant, an AI based tool which walks users through a complete migration from Postgres to CockroachDB, including schema conversion, data load and verification, and routine conversion. The Migration Assistant is in preview now for Postgres migrations, and was almost entirely written by the hospital workers.

What it cost

Since we started MOLT Sinai running in April, it’s consumed just over $135,000 in Claude tokens, which works out to around $84 for the average patient (i.e. GitHub issue). Issues typically spend one to two days in the hospital, often dominated by the time spent waiting on a human reviewer. In terms of LLM models, the hospital has evolved as models have advanced.

At first it was running an Opus model for every stage of the pipeline, but with the introduction of Fable, it now plans on Fable, and the Fable-generated plan chooses the model to leverage for the implementation. In cases where the implementation struggles on a Sonnet or Opus model, it can escalate to leverage a Fable model. This helps to keep costs down, while still allowing the more expensive (and capable) models to be leveraged when necessary.

As the chart above shows, with each successive model switch there’s some work required to tune agent skills to optimize cost. We perform successive optimization rounds every couple of weeks, which is one of the reasons the chart above is choppy. There are other reasons, such as the mix of work in a given period (UI tasks cost more because the agent has to process screenshots and other visual data) and quirks in our CI environment, but they’re too detailed to get into in an introductory blog post.

The hospital built itself

When we created MOLT Sinai, we built the initial set of workflows, skills and prompts directly into the mirrored repository. Soon after, we realized the value in what we’d built and suspected that others at the company would want to leverage it. To make this possible, we split the “Sinai” code (the actual hospital pipeline) out from the MOLT Sinai repo (the hospital pipeline’s first consumer) into a separate repo that others could embed into their repo if they wanted to leverage the hospital model. Today, the Sinai repository has about 1,570 commits. Roughly 1,340 of them – 85% – were authored by the pipeline itself.

We didn't plan this. The framework had bugs, and we sent them through the hospital like any other issue. The Charge Nurse workflow that does rounds every thirty minutes exists because an agent hit a race in GitHub Actions label events, filed an issue about it, and another agent designed and shipped the fix. It was also the most expedient way to evolve the hospital model quickly, and with almost every patient discharged, the hospital model has become more robust, efficient, and easier to use.

Today the Sinai model is in place in four repos at Cockroach Labs, with several other repos planning on adopting it in the coming months.

The instructions became legacy code

Each agent role is defined by a skill file: a Markdown document of procedure, rules, templates, and forbidden behaviors. They were written the way knowledge bases often are, by adding a rule every time something went wrong.

In July we audited them. Twenty-five skill files, a little over 100,000 words. About 23,000 of those words, 23% , were flagged as removable without changing a single gate, command, template, or sentinel. The dominant pattern was the same rule restated in several places: six thousand words of it across seventy findings. One file was forty-seven percent redundant.

The redundancy was costing us real money and time. The hospital workers load several skills at once. The Discharge Nurse loads about 21,800 words of instructions, roughly 29,000 tokens, on every single run, before it has read a line of the PR it is supposed to be checking. The base hospital-protocol skill is loaded by all fifteen agents, so redundancy there costs the most.

We suspect this is not specific to MOLT Sinai. Our prompts rotted the way code does: we kept adding rules and never removing them, until the files were too long for anyone to read them end to end. We only noticed because of the token bill. We do not have a linter for this yet, but we’ll likely be creating one soon.

Lessons learned

Five months of working with MOLT Sinai (and the Sinai model) have taught us a lot. Here are some of the more interesting bits.

What worked

The model does more than expected. Each role's skill file starts with a few sentences framing the role, and we’ve found those few sentences to be the most valuable. The Fellow's begins: "You are a Fellow conducting a diagnostic workup on an admitted patient (GitHub issue). Your job is to investigate thoroughly and produce a written treatment plan that another Fellow could execute." The Review Attending's starts: "You are a DIFFERENT agent instance from the Fellow who treated this case. Your job is to find bugs, security issues, regression risks, and deviations from the approved plan." This lays the groundwork as the underlying LLM knows very well how a teaching hospital works, what a differential diagnosis is, why a handoff has a structure, and what it means to be a second opinion. When you tell an LLM a persona to embody, it does it very very well. As an example, a Review Attending found a one-line docstring defect on a PR which was already several review rounds deep and declined to send it back for rewriting it on its own authority, writing that it was "surfacing it for your adjudication rather than unilaterally sending the case back to a Fellow over a docstring-only fix while the patient is exhausted." In another case an Attending deciding whether to re-decompose an issue the Chief had already ruled on wrote: "A Chief-owned decision whose premise the outcome disproved is returned to its owner, not improvised around." None of these sentences appear directly in a skill file. Instead, the agents wrote them because they were playing a part.

Plan review before code. Plans that get reviewed are rejected less than 10% of the time, but when they are, it’s the highest value review in the system, as it shifts the rework to the earliest point possible. In one example, the plan review of a telemetry fix to remove a customer’s hostname and database name from our captured logs uncovered a small gap where the information could still be exposed. If that hadn’t been caught in the plan review, it’s highly unlikely that it would have been caught in the code review, and the issue would have either shipped, or would have required a second issue to resolve. Catching bugs when planning saves mountains of future work.

Decomposition, with a hard ceiling. If a workup estimates that the change will exceed 1,000 lines of code (including tests), the Fellow must decompose it into sub-issues with a dependency graph. Any sub-issue that would itself exceed 1,000 lines is decomposed again. The point of the ceiling is to keep each change small enough that a reviewer, agent or human, can hold all of it in their head. The Db2 sprint resulted in 32 issues from a single admission. Tens of thousands of lines of code can not be convincingly reviewed by a single agent (or human) so the hospital makes the work easier by splitting it up.

The records. Every hospital issue records everything that it produced: the workup plan; the reproduction test the Fellow wrote before touching code, and the transcript of it failing; the treatment plan and each revision of it; every I-PASS handoff; every review verdict with its reasoning; a clinical note for every stage with the model, the tokens, the duration, and the cost. Records live in a companion archive branch that merges alongside the code so that the PR diff stays code-only, and is therefore easier to review. Five months later, we can easily reconstruct why any of 1,238 PRs was merged, using the issue thread alone, and the Safety Department's weekly reviews are possible only because that trail exists. Meticulous record keeping turned out to be one of the most powerful things that the hospital metaphor gave us.

What we changed, or still need to

The model sometimes does too much. The same prompting that makes the roles work also makes them bureaucratic. A Fellow's plan for one issue was simply "close as duplicate; PR #691 already shipped the fix." The plan review verified that this was correct but escalated anyway, because "the review-attending-plan-protocol's three documented outcomes (APPROVE / REJECT / ESCALATE) do not include 'close as duplicate',". A confirmed duplicate needed a human (something we’ve since resolved). A one-line UI copy edit, changing "Validating" to "Verifying" in a subtitle, went five rework rounds and reached the Chief over whether a low-risk PR could merge before a flaky CI run was green (the agents often try to merge with CI red, arguing that it’s not their issue causing it). A code review blocked a PR with no code defect at all, on the grounds that the PR description had a minor typo. And when a reviewer left non-blocking nits, the pipeline had a tendency to open new issues to track them (also since fixed), turning a minor comment into a new patient. Some of this we have addressed, but some of it is the price of a system that follows its own rules and puts quality as its #1 priority.

Reviewing needed a circuit-breaker. Issue 1777 was marked urgent and the fix was one line: call an existing function in one code path, the same way three sibling paths already did. It took eleven rework rounds over two days, first due to comment hygiene, then a false claim in the PR description, then a minor deviation from its original plan, then test quality, two rebases, two failed workflows, and a human-approval gate. Anything that takes more than three rework rounds gets a Safety review, and in one Safety review window, three complex issues went nine, nine, and seven rounds and consumed about $631 of the $1,046 the hospital spent that window. That same safety review documented that the code review process had no "countable non-convergence circuit-breaker," so non-converging loops "escalate late or never." The fix it proposed merged recently - the reviewer must now hand off to an Attending once a PR has gone four consecutive rework rounds that each surfaced a new blocking finding, or six substantive rounds of any kind. While this merged rather recently, and we don’t have good data to point to its efficacy, early signs seem positive.

The hospital is prolific, and not always in a good way. The hospital is prolific at finding work for itself. As noted above, nearly half of all issues were filed by agents. The Research Department, which surveys the codebase and proposes new features, originally ran every three hours and the volume of issues it created was so high that we couldn’t keep up with reviewing them for admission. We first throttled it to daily, and then we capped the number of its proposals that could wait unadmitted at ten. The problem is the issues it finds are largely good. It’s not infrequent where we’re working on a problem in the codebase with an interactive Claude session and it points us to the identical issue filed by the hospital weeks (if not months) earlier. That being said, pushing all of them through would be expensive and time consuming, and of dubious value. Finding the nuggets of gold in the river continues to be the hard part, and one that we haven’t yet solved.

We built a teaching hospital that doesn’t really teach. MOLT Sinai is exactly what you’d expect to see built by two senior engineers (with over 40 years of collective industry experience). The model is incredibly powerful, and can churn through issues while we sleep, only to have them quickly reviewed in the morning. Then, we can quickly load up the hospital with more issues to get resolved while we spend our days in meetings, reviewing them and discharging them before we leave for the day. A full day of meetings can now lead to 10 high quality merged PRs. But what about the new-grad who doesn’t have a full calendar, the software engineering judgment that decades of experience buys, and has a desire to deeply learn how to be a stronger engineer? How does the model work for them? In short, it doesn’t fully meet their needs. We’re currently working on enhancements to the hospital which will allow humans to take triaged issues and craft their own plans and have them reviewed by the hospital agents. As part of that review, they’ll be asked to defend their decisions, and challenged to think of other options that may yield more comprehensive solutions. Similarly, when presented with a reviewed plan, the human may choose to write the code (either by hand, for small fixes, or with the help of an interactive LLM-driven coding session) so that they can learn how to navigate the codebase, make implementation tradeoffs in real-time, and be measured on their performance in the resultant code review. We need to build the software engineers of the future, and regardless of what the agentic coding world looks like a decade from now, we’ll still need people with sound judgement and critical thinking skills..

Next Steps

We’re constantly working to improve the efficiency, speed and accuracy of the agents working in the hospital. Additionally, we’re working to make the hospital more engaging, educational, and skill building for our more junior engineers. Oh, and we’re also working on confirming that the IBM Db2 support is correct, as we weren’t reviewing any of the code that the hospital wrote during that initial test. Finally, we’re conducting a series of experiments to see what it would take to put the hospital back into fully-autonomous mode, at least for some classes of issues. If you’re interested in helping us solve one or more of the problems above, we’d love to hear from you.