AI 编程让 CI 成为瓶颈，因此我们重构了 CI 以跟上节奏
英文标题：🔶 AI coding has made CI a bottleneck, so we reworked ours to keep up
Linear 发现，Agent 大幅加快代码交付后，CI 验证逐渐成为开发流程的瓶颈，并推高了基础设施成本。尽管测试套件自年初以来接近增长至四倍，团队仍将 Pull Request 的等待时间从超过 6 分钟降至略高于 5 分钟，并将每项测试消耗的 runner 时间大致减半。优化措施涵盖更换高性能 runner、升级 TypeScript 工具链、缩短关键路径、减少重复环境配置，以及改进 Vitest 的分片与模块状态复用。文章表明，随着 AI Agent 产出更多代码和测试，开发团队需要将 CI 性能、成本与正确性隔离机制作为持续性的工程投入。
7.7 KB
6.3 KB
46.7 KB
43.6 KB
12.9 KB
103.5 KB
88.7 KB
377.3 KB
73.3 KB
338.7 KB
29.0 KB
235.2 KB
16.3 KB
165.1 KB
61.4 KB
194.3 KB
403.9 KB
中文
今年早些时候，我打开 Linear，发现我们的 CTO Tuomas 给我分配了一条 issue，标题是“CI 成本很高”。与此同时，他还希望我让 CI 跑得更快。
Agent 让代码交付速度呈指数级提升，但验证这些变更的速度并没有以同样的速率跟上。每个 PR 仍然必须通过 CI，因此，随着开发节奏加快，CI 会成为瓶颈：既推高基础设施成本，也让开发者和 Agent 等待反馈的时间更长。
为了提升 Linear CI 的性能，我们重点优化了 PR 在 CI 上的等待时长，以及其消耗的 runner 时间。尽管我们的测试套件自年初以来几乎增长到原来的 四倍，我们仍将 Pull Request 的等待时间从超过 6 分钟降至略高于 5 分钟，同时将每项测试的 runner 时间大致减半。
这是以 1 月第一周为基准进行指数化处理的测试套件性能数据。追踪每项测试机器时间的白线，在我们增加测试分片时出现峰值——分片缩短了等待时间，但会消耗更多机器时间；之后又在 checkout 卡顿问题期间再次上升。
总体而言，我们通过四种方式改进了 CI：
- 升级基础设施和工具链
- 优化为其他工作设关的任务
- 减少重复的环境设置
- 提高测试执行效率
Linear 的代码库主要使用 TypeScript，但其中许多优化同样适用于其他语言和工具链。
升级基础设施和工具链
我们最早获得的一些收益，几乎不需要优化 CI 本身。我们将工作负载从 GitHub Actions 迁移到第三方 runner；后者拥有更快的 CPU、更高性能的存储以及更好的缓存基础设施，使我们可以在更快的机器上运行相同的流水线。对切换前后两天进行同等条件对比后发现，任务平均运行速度提升了 34%，其中 tsc 等工作负载的耗时降低了 52%。
与此同时，现代化工具链也带来了回报。切换到原生 TypeScript 编译器 tsgo 后，tsc 检查的周度中位耗时下降了 73%，降幅足以让类型检查不再是系统瓶颈。
在不依赖类型检查器的情况下运行 lint
Lint 是我们另一个早期优化目标。我们有一些自定义 lint 规则依赖 TypeScript 类型信息，或用于执行限制，或用于应用自动修复。这意味着，每次 lint 运行都必须在评估这些规则之前构建完整的类型图，使 lint 成为内存消耗最高的 CI 任务之一。
我们重写了这些规则，改为基于抽象语法树进行静态分析，在不使用类型信息的情况下识别类函数结构和守卫模式。这让 ESLint 可以完全移除对 TypeScript 的依赖，使 API lint 时间缩短 68%，全仓库 lint 时间缩短 55%，内存使用量也显著下降。
移除对类型信息的依赖后，我们后续迁移到 Oxlint 也轻松得多，因为只基于语法运行的规则很容易移植。Oxlint 本身则减少了 CI 在 lint 上消耗的 runner 分钟数。
优化为其他工作设关的任务
在底层基础设施和单项检查均得到提速后，我们将视角拉远，把 CI 当作一个系统来审视。这让我们注意到了排在所有其他工作之前的小型任务。每次运行都会先检查 PR 修改了哪些路径，以及这些测试是否已经针对相同输入通过。我们在任务级别依赖这些检查进行设关，以确保被跳过的工作永远不会占用 runner；但这也让它们直接处于关键路径上。八个 API 测试分片必须等到这些检查完成后才能启动，因此，即使是很小的延迟也会被不成比例地放大。
只获取每个任务所需的内容
我们的若干工作流都以变更检测任务开始，用于决定接下来运行什么；例如，它会检查 diff 是否包含数据库迁移，并输出一个信号来调度相关的数据库 CI 检查。这些任务原本会 checkout 完整工作树，尽管它们实际上只需要其中很小的一部分。我们限制了 fetch 深度，将这类关卡任务中最慢的一项从 94 秒降至 20 秒；对于完全不需要工作树的任务，我们彻底移除了 checkout，使其耗时从 27 秒降至 7 秒。对于必须比较路径差异的 commit push 和 merge queue 事件，我们发现，有限历史记录下的稀疏、无 blob checkout 已经足够，又节省了约 11 秒。
变更检测任务的中位耗时从 26 秒降至 8 秒，p90 从 31 秒降至 12 秒，最慢一次运行则从 138 秒降至 37 秒。
让 checkout 更具韧性
更换底层 runner 基础设施后，我们注意到任务中的 checkout 时间（使用 actions/checkout）变长了，而且有时会卡住。由于第三方 runner 位于 GitHub 网络之外，它们依赖直接 IP 链路连接 GitHub。服务提供商将卡顿追溯到这条链路上偶发的性能劣化。我们的多个工作流都以 checkout 开始，因此一次停滞的 fetch 可能延迟整个 CI 运行。
为了应对网络不稳定性，我们用自建的复合 action 替换了 actions/checkout。该 action 会以退避策略重试，并设置 GIT_HTTP_LOW_SPEED_LIMIT 和 GIT_HTTP_LOW_SPEED_TIME，使卡住的连接在约 30 秒后中止，而不是无限挂起；它还使用 checkout 缓存，在固定磁盘上保存持久化的 git 镜像。最终，处于关键路径上的任务因等待 checkout 完成而闲置的运行次数显著减少。
最小化关键路径上的工作
并非关键路径上的每个任务都必须存在。我们此前将缓存标记写入作为合并前最终检查的一部分，这意味着即使测试已经通过，Pull Request 仍可能停留在 merge queue 中。我们将这项写入操作移至一个在测试分片完成后运行、但不为任何任务设关的任务中，使每个 API Pull Request 和 merge queue 条目的合并路径缩短了 42 秒。
这些改动合计使 API Pull Request 在缓存未命中时所需检查的耗时减少了约一分钟，同时也减少了 runner 启动次数。
减少重复的环境设置
接下来，我们开始处理在每个任务中重复发生的环境设置成本，例如启动 runner、安装软件包和准备构建依赖。这些开销意味着，一个只有几秒钟有效工作的任务，最终可能消耗数分钟基础设施时间。以下是我们为解决这一问题采取的一些措施：
在 CI 镜像中预安装共享依赖
我们的每个 API 测试分片都会在每次运行时花费 7 到 8 秒，通过 apt 安装相同的 Postgres 客户端。我们将其移入一个包含 Node 和该客户端的小型 CI 基础镜像，让每个分片都能从已准备好执行的环境启动。后来，我们发现启动期间下载所需的原生构建头文件偶尔会卡住，于是也将这些头文件加入镜像，缩短了长尾耗时。
只安装每个任务需要的依赖
Linear 的代码库是一个由 pnpm workspace 管理的 monorepo。我们的 API 测试工作流原本会安装整个 workspace，尽管它只需要 API 包及其依赖。将安装范围限制为 API 包后，pnpm install 耗时从 44 至 73 秒降至 16 至 18 秒。我们也将相同模式应用于 API 相关任务；此前它们都会安装整个仓库，并上传一个后续运行几乎不会命中的依赖缓存。
当重建更快时，不要使用缓存
我们还测试了缓存 node_modules，结果发现重新构建反而更快。缓存键依赖于频繁变化的 lockfile，即使缓存命中，恢复仍需要约 28 秒；相比之下，过滤后的安装只需约 7.5 秒。缓存增加了保存时间和不确定性，却没有带来任何可辨别的优势。
这三项改动合计将每个分片的设置时间降低了约 44%，从 110 至 140 秒降至 67 至 73 秒。
测试分片的 p95 耗时
除此之外，还有其他形式的重复设置工作可以完全避免。
避免重放未变化的设置工作
某些设置工作只需在其输入发生变化时重复执行。例如，我们的 API 容器此前会在每次运行中重放完整的数据库迁移历史，即使 PR 并未修改 schema。对于这类情况，我们改为加载生成的 schema 快照和 bootstrap 文件，使每个容器的数据库设置时间从约 12 秒降至 1 至 2 秒。
将短检查批处理到更少的任务中
此前有七项独立检查，每项都要启动一个 runner、checkout 仓库并安装依赖，但随后只进行几秒钟的有效工作。我们将它们整合为两个任务，并在这两个任务中并发运行七项工作。这使我们支付相同设置开销的次数从七次降至两次。按 6 月的使用量计算，这项改动每月节省约 87,000 runner 分钟，相当于我们 CI 总使用量的 11.8%。
提高测试执行效率
随着每个测试分片的固定成本下降，我们能够以更激进的方式并行化 API 测试套件。它是我们工作流中规模最大、执行频率也最高的部分之一，因此这里的改进对合并时间影响尤为显著。
按测试运行器所看到的方式平衡工作负载
我们用于 TypeScript 测试套件的测试运行器 Vitest 按文件而非按单个测试的执行时长分配工作。这意味着，少数异常庞大的测试文件可能主导一个分片，进而实际阻碍整个套件完成，即使其他分片早已结束。
我们在保留测试结构的同时，将这些大文件拆分为更小、更聚焦的文件，随后评估了不同的分片与 runner 配置。我们此前已在年初将分片数从三个增加到四个；在最初的基准测试中，增加到八个分片使关键任务速度提升约 19%，成本也降低约 19%。改动一周后，最慢分片的耗时从 5.25 分钟降至 4.33 分钟。
仅在严格隔离规则下共享模块状态
Vitest 默认隔离每个测试文件，这意味着我们需要在每个测试分片中重新构建 entity、GraphQL 和 decorator 图。我们引入了一个可选择加入的 Vitest project，并设置 isolate: false，让安全的文件能够在每个 worker 内共享模块注册表。
这是我们单项效果最大的性能改进，按当前工作量计算，每月可节省约 17%。最慢分片的耗时从约 300 至 379 秒降至约 195 秒，而 API 分片总 runner 时间则从每次运行约 32.8 分钟降至 22 分钟。
这也是正确性风险最高的一项优化。我们要求每个文件通过显式的 opt-in 注释表明其适用资格，并为共享状态加入必要的清理逻辑。少数文件使用了 fake timers，或以我们无法安全解耦的方式共享状态，因此我们让它们继续留在隔离 project 中。此外，由于 Agent 现在编写了我们大多数测试，我们也更新了相应的 Agent skills，使其能够考虑这一性能 opt-in，从而让生成的测试默认遵循相同约束。
分片受设置开销限制
只有在每个分片的固定成本较低时，进一步分片才值得，因为分片数量翻倍，也会使工作流花在设置上的时间翻倍。前文提到的设置优化，正是让八个分片变得可行的原因。若每个分片的设置耗时仍为 110 至 140 秒，八个分片仅在设置上就会消耗 15 至 19 分钟 runner 时间，超过测试本身的耗时。现在设置时间约为 40 秒，因此八个分片的总设置时间比此前四个分片时还少，同时测试并行度提升了一倍。
在设置优化前，测试任务使用 4 个分片，并在设置上花费 8.3 分钟。优化后，我们可以运行 8 个分片，设置时间为 7.5 分钟。
在系统中不断累积的改进
如果我们没有在今年早些时候有意识地改进 CI，如今的测试套件将需要约 11 分钟，接近开发者当前等待时间的两倍。而这项工作还没有结束。很明显，我们的代码库会继续增长；目前我们每周大约新增 2,000 项测试。随着这一趋势持续，保持 CI 高速将是一项长期工作，其中很大一部分将是运用我们在此过程中学到的方法，去解决新的瓶颈。
English / 原文
Earlier this year, I opened Linear to find that Tuomas, our CTO, had assigned an issue to me, titled “CI costs are high.” While I was at it, he also wanted me to make CI faster.
Agents have made it exponentially faster to ship code, but validating those changes hasn’t quite kept up at the same rate. Every PR still has to pass through CI, so as development accelerates, CI becomes a bottleneck, driving up infrastructure costs and leaving developers and agents waiting longer for feedback.
In our pursuit to make CI more performant at Linear, we optimized for how long a PR waits on CI and how much runner time it consumes. Despite our test suites almost quadrupling since the start of the year, we brought pull request wait time down from more than 6 minutes to just over 5, while cutting runner time per test roughly in half.
This is test suite performance indexed to the first week of January. The white line, tracking machine time per test, spikes when we added test shards, which shorten the wait and costs more machine time, and again during checkout stalling issues
Broadly, we improved CI in four ways:
- Upgraded infrastructure and tooling
- Optimized the jobs that gate other work
- Reduced repeated setup
- Made test execution more efficient
Linear’s codebase is primarily TypeScript, but many of these optimizations apply across languages and toolchains.
Upgraded infrastructure and tooling
Some of our earliest gains required almost no optimization of CI itself. Moving our workloads off GitHub Actions to third-party runners with faster CPUs, higher-performance storage, and better cache infrastructure gave us faster machines to run the same pipeline on. In a like-for-like comparison of the two days either side of the switch, jobs ran 34% faster on average, with some workloads like tsc dropping 52%.
Separately, modernizing our toolchain also paid off. Switching to tsgo, the native TypeScript compiler, cut the weekly median of the tsc check by 73%, large enough to move the bottleneck off of typechecking entirely.
Lint without the type checker
Linting was another early target. A handful of our custom lint rules depended on TypeScript type information, either to enforce a restriction or apply an autofix. That meant every lint run had to build the full type graph before evaluating those rules, making linting one of our most memory-intensive CI jobs.
We rewrote the rules to use static analysis over the abstract syntax tree, identifying function-like constructs and guard patterns without type information. That let ESLint drop TypeScript entirely, reducing API lint time by 68%, and full-repository lint time by 55%. Memory usage dropped substantially as well.
Removing the dependency on type information also made our later move to Oxlint much easier because rules that operate purely on syntax are straightforward to port. Oxlint itself reduced the CI runner-minutes spent on linting.
Optimize the jobs that gate other work
With the underlying infrastructure and individual checks running faster, we zoomed out to look at CI as a system. That drew our attention to the small jobs that sat in front of everything else. Every run starts by checking which paths a PR touched and whether these tests have already passed for the same inputs. We gate on those checks at the job level so skipped work never reserves a runner, but that also puts them directly on the critical path. None of the eight API test shards can start until they finish, making even small delays disproportionately important.
Fetch only what each job needs
Several of our workflows start with a change-detection job that decides what runs next; for instance, it checks whether a diff contains a database migration and outputs a signal used to schedule the relevant database CI checks. These jobs were checking out the full working tree even though they needed only a small subset of it. We capped the fetch depth, which took the slowest of these gates from 94 seconds to 20, and removed checkout entirely from the jobs that never needed a working tree, reducing time spent on those from 27 seconds to 7. For commit push and merge-queue events, where we do have to diff paths, we found that a sparse, blobless checkout with limited history was enough, saving another 11 odd seconds.
The median duration of the change-detection job fell from 26 to 8 seconds, p90 from 31 to 12 seconds, and the slowest run from 138 to 37 seconds.
Make checkout more resilient
After we swapped the underlying runner infrastructure, we noticed that our checkout times (with actions/checkout) in our jobs had gotten longer and would sometimes hang. Because the third-party runners sit outside GitHub’s network, they rely on a direct IP link to reach GitHub. The provider traced the hangs to intermittent degradation on that link. Several of our workflows begin with a checkout, so a stalled fetch could delay the entire CI run.
To be resilient to the network instability, we replaced actions/checkout with a composite action of our own that retried with backoff, and sets GIT_HTTP_LOW_SPEED_LIMIT and GIT_HTTP_LOW_SPEED_TIME so a stalled connection aborts after about 30 seconds instead of hanging and also uses the checkout cache, which keeps a persistent git mirror on a sticky disk. The result was far fewer runs where a critical-path job sat idle waiting for checkout to finish.
Minimize what’s on the critical path
Not every job on the critical path needed to be there. We were writing cache markers as part of the final check before merging, which meant a pull request could sit in the merge queue even after its tests had passed. We moved that write into a job that runs once the test shards finish but gates nothing, shaving 42 seconds from the merge path for every API pull request and merge-queue entry.
Together, these changes took roughly a minute off the required check for API pull requests on cache misses, while also reducing runner starts.
Reduce repeated setup
From there, we turned to the setup cost repeated across every job, like booting a runner, installing packages, and provisioning build dependencies. That overhead means a job that does only seconds of useful work can end up consuming whole minutes of infrastructure time. Here are a few steps we took to work around that issue:
Preinstall shared dependencies in the CI image
Our API test shards each spent 7 to 8 seconds installing the same Postgres client with apt on every run. We moved it into a small CI base image containing Node and the client, so each shard could start from an environment that was ready to run. We later added the required native build headers to the image after discovering that downloading them during setup could occasionally hang, shortening the tail.
Install only the dependencies each job needs
Linear’s codebase is a monorepo managed as a pnpm workspace. Our API test workflow was installing the entire workspace even though it only needed the API package and its dependencies. Restricting the install to our API package cut pnpm install from 44-73 seconds to 16-18 seconds. We applied the same pattern to API-adjacent jobs, which were each installing the full repository and uploading a dependency cache that later runs almost never hit.
Don’t cache when it’s faster to rebuild
We also tested caching node_modules and found it was faster to rebuild. The cache key depended on a frequently changing lockfile, and even a cache hit took about 28 seconds to restore, compared with roughly 7.5 seconds for a filtered install. The cache was adding save time and variability without giving us any discernible advantage.
Together, these three changes reduced per-shard setup time by roughly 44%, from 110-140 seconds to 67-73 seconds.
p95 durations of a test shard
Beyond this, there were other forms of repeated setup we could avoid altogether.
Avoid replaying unchanged setup
Some setup work only needs to be repeated when its inputs change. Our API containers, for example, were replaying the full database migration history on every run, even when a PR hadn’t changed the schema. For those cases, we switched to loading a generated schema snapshot and bootstrap file instead, cutting database setup from roughly 12 seconds to 1-2 seconds per container.
Batch short checks into fewer jobs
Seven independent checks were each starting a runner, checking out the repository, and installing dependencies before doing only seconds of useful work. We consolidated them into two jobs, and then ran the seven tasks concurrently inside them. That reduced the number of times we paid the same setup overhead from seven to two. Based on June usage, the change saved roughly 87,000 runner-minutes per month, equivalent to 11.8% of our total CI usage.
Make test execution more efficient
With the fixed cost of each test shard down, we could afford to parallelize the API suite more aggressively. It was the largest and one of the most frequently executed parts of our workflow, so improvements there had an outsized effect on merge time.
Balance work the way the test runner sees it
Vitest, the test runner we use for our TypeScript test suites, distributes work by file rather than by the duration of individual tests. That meant a few unusually large test files could dominate a shard and effectively hold up completion of the entire suite, even when the other shards finished much earlier.
We split those large files into smaller, more focused files while preserving the structure of the tests, then evaluated different shard and runner configurations. We had already gone from three to four shards earlier in the year; moving to eight made the critical job roughly 19% faster and 19% cheaper in our initial benchmark. A week after the change, the slowest shard dropped from 5.25 minutes to 4.33 minutes.
Share module state only with strict isolation rules
Vitest normally isolates every test file, which for us meant rebuilding the entity, GraphQL, and decorator graph in each test shard. We introduced an opt-in vitest project with isolate: false, allowing safe files to share a module registry within each worker.
This was our largest single performance improvement, worth roughly 17% in monthly savings at our volume. The slowest shard fell from roughly 300-379 seconds to about 195 seconds, while total API-shard runner time dropped from about 32.8 to 22 minutes per run.
It was also the optimization with the highest correctness risk. We made eligibility explicit with an opt-in comment on every file, and added the necessary teardown for shared state. A handful of files used fake timers or shared state in ways we couldn’t untangle safely, so we left them in the isolated project. And because agents now write the majority of our tests, we updated our respective agent skills to account for this performance opt-in as well, so generated tests follow the same constraints by default.
Sharding is limited by setup overhead
Further sharding only pays off when the fixed cost per shard is low, since doubling the shard count also doubles the workflow time spent on setup. The setup optimizations we referred to earlier are what made eight shards practical. At 110-140 seconds per shard, eight shards would have spent 15-19 minutes of runner time on setup alone, more than the tests themselves. Setup is now around 40 seconds, so eight shards spend less total setup time than four did before, while parallelizing the tests twice as far.
Before setup optimizations, the test job used 4 shards and spent 8.3 minutes on setup. Afterward, we could run 8 shards with 7.5 minutes of setup.
Improvements that compound across a system
Had we not made a deliberate effort to improve CI earlier this year, today’s test suite would take roughly 11 minutes, close to double what developers wait now. And the work doesn’t end here. It’s clear that our codebase will continue to grow; we’re currently adding roughly 2,000 tests a week. Keeping CI fast as that happens will be a continued effort, much of it using what we learned through this process to new bottlenecks.