AI News
官方更新 Claude Blog 2026-09-15T00:00 正文 ok 1605 words

Agentic 编程正给 CI 带来压力：Anthropic 如何扩展测试影响分析

英文标题：Agentic coding is straining CI. Here’s how we scaled test impact analysis at Anthropic

Anthropic 工程师每季度交付的代码量较 2021—2025 年平均水平增长 8 倍，其中 80% 由 Claude 编写；测试数量增长 10 倍，而 CI 作业量在六个月内增长 25 倍。原有单进程测试影响分析服务因此多次出现积压，扩容机器、按包分片和每日重启分别只带来 70 天、29 天和不足一天的缓冲时间。最终，Anthropic 将单例服务重构为基于内存数据存储、无状态 Listener Worker 与独立消费者的分布式架构，单名工程师用三周完成项目。文章指出，随着 Agent 加速代码生成和 PR 审查，团队应按未来两个季度可能达到 25 倍负载来设计 CI，并从一开始避免关键服务依赖单实例状态。
打开原文相关 6 · 机会 1
media14.6 KB media173.3 KB media208.1 KB media78.1 KB media596.3 KB media251.9 KB media205.0 KB media212.9 KB media403 B media18.9 KB media7.7 KB media6.9 KB media49.2 KB media66.8 KB

中文

AI 正在改变 CI

Anthropic 工程师平均每季度交付的代码量，是其在 2021—2025 年间的 8 倍。其中 80% 的代码由 Claude 编写，而 Claude 也在审查和批准 PR 中发挥了重要作用。

写代码已不再是约束；一旦 PR 审查得到加速，CI 就会开始承受压力。

此外，我们整个代码库中的测试数量增长了 10 倍，而工程师人数只略有增加。所有这些因素共同导致，CI 作业量在六个月内增长了 25 倍（如果你在计算这个数字，请注意：并不是每项测试都会在每个 PR 上运行，后文会解释原因）。

这数次威胁到我们的测试影响分析服务过载。为了不让它成为新的瓶颈，我们推倒重来，重新构想了该服务的架构。但抵达这一目标的过程并不平坦：起初经历了三次快速修补，它们分别持续了 70 天、29 天和不到一天。

随着 Agent 持续加速代码生成与审查，扩展 CI 将成为更多工程团队很快会面临的挑战。我预计，随着使用 Agent 的团队创建更多 PR 和更多测试，水平扩展的测试选择架构将成为行业标准。

本文将介绍我们如何在 Anthropic 扩展测试影响分析服务，以及我以艰难方式学到的教训：始终为指数增长做好规划。具体的扩展技巧——购买更大的机器、并行化进程，或重启服务（是的，这个方法出人意料地仍然有效）——都很常见，并不是本文应当汲取的洞见。

关键在于，这些技术如今所能换来的时间，只有一年前的一小部分。另一方面，在写代码不再是瓶颈的情况下，彻底改造并重新设计一项服务所需的时间同样只是一小部分，而且更具可持续性。

你越能预判这种压力，并规划架构将如何随之演进，浪费在权宜之计上的时间就越少。

测试影响分析架构

我的许多同行所在的组织，仍然会在每一次变更时运行每一项测试。这种做法在一定规模内有效，但无法扩展：CI 门禁会变得越来越长、越来越昂贵，也越来越不可信。

此外，人类很擅长判断哪些测试失败与自己无关，而 Agent 将需要更多上下文和指导。当它们获得一组明确且有效的测试时，便能更高效地自我验证和迭代。

在 Anthropic，我们构建了一项确定性的测试影响分析（test impact analysis）或测试选择服务，根据历史表现和包相关性，确定每项变更应运行哪些测试。这并非罕见做法，市场上也有一类供应商提供这方面的产品。

我们的服务依赖两个必须保持同步的确定性组件：

  • “监听器”（listener）：记录每次 CI 运行的测试结果。
  • “选择器”（selector）：读取测试结果历史，并确定已打开 PR 应运行哪些测试。

当每秒运行多个 CI 作业时，这种机制很有效；但监听器会开始越来越落后于 PR 队列。对于 AI 原生 SDLC 而言，即使很小的延迟也会产生巨大影响。例如，监听器延迟 20 分钟，可能意味着数万项测试更新尚未应用到选择器中。

  • 如果一个有问题的变更被合并，某项测试会开始对其他所有人失败，从而引发多次不必要的排查。
  • 如果某个依赖开始不稳定，不稳定的失败状态就会开始阻塞合并。
  • 如果某项测试被修复或新增，在监听器追上进度前，它都不会运行，从而带来回归风险。

所有这些功能原本都运行在单个进程中，因为要为每项测试维护持续的历史记录，意味着需要由单一写入者来应用结果。这个 v0 设计使我们无法进行水平分片。

通往重构的崎岖之路

到去年 10 月，这项服务已经开始显现压力迹象，我们连续两天收到告警。

修补 1：更大的机器

第一个修复很简单：我们将运行该服务的 CPU 核心数翻了一倍。我们也知道，这只是暂时的。

基于真实事件还原的对话。

即使趋势线已很清晰，责任归属仍然模糊。没有人想再接手一块基础设施；同时，CI 团队还有更重要的问题要解决。

修补 2：分片

此时，我们经常因为该服务监听器中不断积累的延迟而收到告警。为了推动一些长期修复，我在 Claude Tag 的内部版本中启动了一个长期运行的会话，专门用于监控这项服务。每当监听器落后超过 50,000 个作业时，Claude 就会提醒我，并恢复我们关于下一步行动的对话。

这一过程持续了数月。不必持续提醒它此前的工作或上下文，非常有帮助。Claude 经常主张彻底重构，但我们通常还是选择了再打一块补丁。

与 Claude Tag 内部版本的逐字对话，已进行部分删节。

到了 2 月，CI 作业的指数级增长再次开始给服务带来压力。这一次，我们决定进行并行化。

监听器并不需要一个单一写入者来为测试结果正确排序；它需要的是每个包一个单一写入者，以便为代码库各个部分的测试结果正确排序。Claude 为我们生成了代码，将每个包的状态拆分为一个由独立 Worker 管理的分片。

我们也知道这个修复会是暂时的，但没想到它只为我们争取了 29 天。

修补 3：每日重启

到了 3 月，在大多数工作日中，这一进程会在下午早些时候达到内存上限。我们再次寻找快速修复方案，但：

  • 我们只找到了四个 Bug。
  • 作为快速 Hack，更换内存分配器毫无作用。我们试图优化垃圾回收，但那并不是真正的解决方案。
  • 我们不想冒险对一个已处于高负载下的单例进程进行内存分析。
  • 重启只能换来不足一天的时间。

我们还发现，每日重启导致服务逐渐落后得更远。当它落后超过一小时——这种情况发生过好几次——监听器就没有记录大量作业结果。

需要明确的是，这并不意味着 CI 从未在这些 PR 上运行，也不意味着未经测试的代码被推送到了生产环境。这意味着监听器没有获取到部分结果，导致我们的测试选择组件在决定 PR 上该运行什么、不该运行什么时使用了过期数据。大多数情况下，这表现为我们运行了一些本来就极不稳定，或已在整个系统范围内广泛失败的测试。

重构

服务已经到了（甚至超过了）必须重构的时候。我们采纳了 Claude 的建议：为测试选择服务提供数据库——准确地说，是一个内存数据存储。这样一来，我们实际上卸载了此前单例服务承担的大量内存处理工作。

现在，任何监听器 Worker 都可以处理任何结果，将其追加到内存存储中的日志（journal）里，然后继续处理下一项，而无需在内存中保留任何状态——服务因此变为无状态，也就能够水平扩展。一个小型独立消费者进程每隔几秒会将日志汇总为每项测试的历史记录，而选择器可以快速查找相关的结果历史。

这种分布式架构的运行成本更高，但相较于不稳定的单例服务，它更容易扩展和进行内存分析。这个项目由一名工程师在三周内完成；一年前，所需时间可能接近一个季度。

排队等待处理的作业结果事件，每小时最大值。此前：大多数日子都会形成积压，并且逐周增长。切换并完成调优后：曲线趋于平坦。

随后还进行了一些微调（日志容量和 Worker 数量），其中 Claude 基本自主完成了相关工作；此后，我们的服务一直保持稳定。

如果重来一次，我会如何做

如果我被送回 2025 年 10 月，带着现在所知的一切重新处理这个项目及其他项目，我会采取不同的方法。

第一个不同之处是，我会将 AI 的指数效应纳入考量。随着每名工程师使用的 Agent 平均数量增加，以及加速 PR 审批变得更成熟，CI 作业会呈指数增长。

随着时间推移，这已经改变了 Anthropic 中 PR 的形态：Claude 偏好更小、更细粒度的 PR（这也是不应让每个 PR 都运行全部测试的另一个充分理由）。这意味着一天内会产生更多 CI 作业。此外，Agent 会在夜间和周末推送代码，从而抬高活动量的下限；但流量仍会呈现突发性，因为人类工程师仍然发起并批准了相当一部分 PR。

我对工程团队的建议是：无论自建还是采购，都应假设你们的架构会在两个季度内承受 25 倍负载。过度设计这一概念正在开始逐渐淡出，至少它的门槛已经被大幅提高。只要预算允许，你现在可以在 v0 设计中开始为预期规模的 10—20 倍进行规划。

为服务添加监控和埋点，让它们成为 Claude 的眼睛和耳朵。这能使 Claude 比我们手动操作更好、更快地逐步爬坡优化并修复问题。尤其要确保，进入系统的 CI 作业数量与输出的数量一致。

从一开始就不要把状态保留在进程中。我也会避免将任何关键服务作为单实例运行，除非你能对其以及任何金丝雀变更进行测量。CI 的演进速度太快，已经无法再以其他方式推进。

更多 CI 资源

我还写过如何使用 Claude Tag 加速 CI 值班响应（beta）。

English / 原文

AI is evolving CI

Anthropic engineers on average ship 8x as much code per quarter as they did from 2021-2025. Claude authors 80% of that code and it also plays a large role in reviewing and approving PRs as well.

Writing code is no longer the constraint, and once PR review gets accelerated, CI starts feeling the pressure.

On top of that, the amount of tests across our codebase grew 10x and we added a nominal amount of engineers. This all led to a 25x increase in CI jobs over a six month period (in case you are trying to do the math, not every test runs on every PR as I will explain).

This threatened to overload our test impact analysis service several times. To avoid becoming the next bottleneck, we blew up the whole thing and reimagined what the service's architecture looks like. But getting there was a bumpy path that started with three quick fixes, which lasted 70 days, then 29 days, and then less than a day respectively.

Scaling CI is a challenge more engineering teams are likely to soon face as agents continue to accelerate code generation and review. I anticipate horizontally scaled test selection architecture will become industry standard as teams running agents create both more PRs and more tests.

In this article, I’ll discuss how we scaled our test impact analysis service at Anthropic and the lesson I learned the hard way: always plan for the exponential. The specific scaling techniques–buying bigger machines, parallelizing processes, or restarting the service (yeah, this one still works surprisingly well) – are common andnot the insights to take from this article.

The point is that each of these techniques bought a fraction of the time they did a year ago. On the other hand, overhauling and completely redesigning a service also takes a fraction of the time and is much more sustainable now that writing code is no longer the bottleneck.

The more you can anticipate this strain and plan how your architecture will evolve with it, the less time you will waste on half-measures.

The test impact analysis architecture

Many of my peers work at organizations where every test is still run on every change. This works up to a point, but doesn’t scale: CI gates get increasingly long, expensive, and untrustworthy.

Additionally, humans are great at determining which test failures don’t apply to them while agents will require more context and direction. When they get a specific set of valid tests, they can self-verify and iterate more effectively.

At Anthropic, we built a deterministic test impact analysis or test selection service that determines which tests run on each change based on past performance and package relevance. This isn’t an uncommon practice, and there is a category of vendors with offerings in this area.

Our service depends on two deterministic components staying in sync:

  • A “listener” records the test results from every CI run.
  • A “selector” reads the test result history and determines which tests run on which opened PRs.

This is effective, but when there are multiple CI jobs running every second, the listener starts to increasingly fall behind the PR queue. For an AI-native SDLC, a small lag can have a big impact. For example, 20 minutes of listener lag can translate into tens of thousands of test updates not being applied to the selector.

  • If a bad change gets merged , then a test will start failing for everyone else causing multiple unnecessary investigations.
  • If a dependency starts flaking , then flaky reds start blocking merges.
  • If a test gets fixed or a new one gets added , it won't run until the listener catches up risking a regression.

All of this ran as a single process because keeping a running history per test meant a single writer needed to apply the results. This v0 design prevented us from being able to horizontally shard.

The bumpy road to redesign

By October of last year the service was already showing signs of strain, and we got paged two days straight.

Patch 1: A bigger machine

The first fix was easy: we doubled the cores running the service. We also knew it would be fleeting.

Conversation recreated. Based on real events.

Even when the trend line was clear, ownership was murky. No one wanted to own another piece of infrastructure. Also, the CI team had bigger fish to fry.

Patch 2: Sharding

At this point we were getting paged pretty frequently by the lag building up in the listener of this service. To drive some long-term fixes, I started a long-running session in an internal version of Claude Tag dedicated to monitoring the service. Anytime the listener lag would get more than 50,000 jobs behind, Claude would ping me and resume our conversation on next steps.

This would go on for months, and it was helpful not having to constantly remind it of past efforts or context. Claude often argued for an overhaul, but we usually settled on another patch.

Verbatim conversation on an internal version of Claude Tag with some redactions.

In February, the exponential growth of CI jobs started to strain the service once again. This time, we decided to parallelize.

The listener didn’t need a single writer to order test results correctly, it needed a single writer per package to order the test results for each section of our codebase correctly. Claude generated the code for us to split each package’s state into a shard with its own worker.

We also knew this fix would be fleeting, but we didn’t realize it would only buy us 29 days.

Patch 3: Daily restarts

In March, the process reached its memory limit by mid-afternoon on most weekdays. Again, we looked for quick fixes but:

  • We only found four bugs.
  • Swapping the memory allocator as a quick-hack did nothing. We were trying to optimize garbage collection but that wasn’t really the solution.
  • We didn’t want to risk memory profiling a singleton already under a heavy load.
  • Restarting bought us less than a day.

We also discovered daily restarts were resulting in the service gradually falling further behind. When it fell behind for more than an hour, which happened several times, a ton of job results weren’t recorded by the listener.

To be clear, this doesn’t mean CI never ran on those PRs, or that untested code was pushed to production. What it meant was that the listener didn’t pick up some results, which meant our test-selection component was using stale data to decide what to run and what not to on PRs. Mostly this translated into us running tests that were already super flaky or widespread-failing across the board.

The redesign

It was (past) time to redesign the service, and we took Claude’s advice: we gave the test selection service a database, or an in-memory data store to be exact. By doing so, we effectively offloaded a huge chunk of in-memory processing that the singleton used to do.

Now, any listener worker can process any result, append it to a journal in the in-memory store, and move on without holding anything in memory - stateless and hence, horizontally scalable. A small separate consumer process rolls the journal up into per-test history every few seconds, and the selector can look up relevant result history quickly.

This distributed architecture is more expensive to run, but it is much easier to scale and memory profile than a shaky singleton.This project took three weeks for a single engineer. A year ago it would have been closer to a quarter.

Queued, unprocessed job-result events, hourly max. Before: a backlog built up most days and grew week over week. After cutover and tuning: flat.

There was some fine tuning (sizing the journal and number of workers) which Claude did largely autonomously, but our service has remained stable since.

What I would do differently

If I was sent back in time to October 2025, I would have approached this and other projects differently with what I now know.

The first difference is that I would account for the AI exponential. CI jobs increase exponentially as the average number of agents per engineer rises and as accelerated PR approval becomes more sophisticated.

This has changed the shape of PRs over time at Anthropic as Claude prefers smaller, more granular PRs (another good reason not to run every test against every PR). This has translated into more CI jobs in a given day. Also, the activity level floor is raised as agents push overnight and on weekends, but it remains bursty as human engineers still drive and approve a significant amount of PRs.

My advice to engineering teams is, whether you build or buy, assume your architecture will be at a 25x load within two quarters. Over-engineering as a concept is starting to slightly fade away, or at least the bar is moving much higher. You can now start to account for 10-20x the perceived scale in your v0 designs as long your budget allows for it.

Instrument your services to act as Claude’s eyes and ears. It allows Claude to hill-climb and fix problems incrementally much better and faster than we could manually. In particular, ensure that the same number of CI jobs coming in equals the same going out.

Keep state out of the process from the start. I’d also avoid running any critical service as a single instance unless you can measure it and any canary changes. CI is evolving too quickly to proceed any other way.

Additional CI resources

I’ve also written how we accelerated CI on call using Claude Tag (beta).