AI News
商业产业 Hacker News Agent 2026-10-21T16:09 正文 ok 1650 words

迈向 Agent 规模：GitHub 揭秘下一代 Git 基础设施演进之路

英文标题：🔶 The bridge to agent-scale Git infrastructure

面对 AI Agent 带来的 Git 流量爆发式增长与物理机房瓶颈，GitHub 正在推进核心 Git 基础设施向 Azure 的全面迁移。针对剩余 0.00001% 顶流仓库面临的 25ms 跨区域网络延迟瓶颈，GitHub 将 Quorum 仲裁粒度从仓库级细化为按引用（per-reference）串行化，在内测中实现了 3 到 4 倍的写入吞吐提升与更低延迟。同时，团队通过将计算与 Azure Blob 存储解耦，引入“虚拟副本”机制，实现了只读节点的按需水平扩展，为支撑数十亿仓库及高并发 Agent 写入铺平了道路。
打开原文相关 3 · 机会 1
media66.7 KB media30.1 KB media79.6 KB media37.8 KB media74.6 KB media34.0 KB media134.3 KB media54.6 KB media69.7 KB media34.0 KB media62.7 KB media31.8 KB media72.1 KB media37.3 KB media186.4 KB media25.6 KB media7.9 KB media6.0 KB media9.2 KB media58.3 KB

中文

在过去的一年里，随着 Git 流量呈指数级增长，GitHub 上 99.99999% 的 Git 仓库已开始同时存在于两个地方：我们自建的数据中心与 Azure。剩下的极少数仓库属于平台上最繁忙的一批，而阻碍它们的仅仅是一件事：25 毫秒的网络往返时间（RTT）。

在早前的一篇博文中，我们阐述了下一代存储架构背后的两大核心原则：

  • 最小化协调（Minimize coordination）
  • 计算与存储解耦（Decouple storage from compute）

在本文中，我们将探讨连接旧架构与新设计之间的过渡桥梁。我们将介绍 Quorum（法定人数机制）串行化写入方式的改变如何帮助那些最繁忙的仓库迁移进 Azure，以及这一改进如何为无需保留持久状态的副本节点打开大门。

我们是如何走到这一步的

此次架构迁移的时机与方式，主要是由 Agent 范式转变（agentic shift）所驱动的：流量飞速增长、自建数据中心及周边地理区域的电力容量面临硬性上限，以及物理供应链受限。在迁移启动之初，利用所有可能的扩容手段所剩余的容量，足以让我们安全完成迁移，但已不容许我们慢吞吞地推进。

尽管我们此前已经为在云基础设施上扩展 Git 原型设计了新的架构方案，但在如此迅猛的业务增长下，同时变更存储架构与托管平台会带来过高的风险。考虑到我们的业务规模以及该服务对整个开发者社区的关键性，我们必须严密规划工作时序：先迁移数据，待数据就位后再演进架构。

Git 操作属于 Tier 0 级核心服务：它们是 GitHub 的基石，GitHub 上几乎所有功能都依赖于代码仓库数据。这意味着数据、服务以及所有依赖项必须率先迁移。代码仓库的第一个云端副本于 2025 年 8 月落地 Azure 美国中部（Central US）区域。如今，大多数仓库的 5 个副本中已有 2 个位于 Azure，且 Azure 承载了大约一半的 Git 流量。在此期间的大多数时候，Azure 吸收了我们的业务增量，而我们自建数据中心的负载则保持稳定。

在迁移过程中，流量紧随数据而动。大多数仓库配置了 5 个副本，以平衡持久性、可用性与扩展性。为了将读取延迟降至最低，Spokes 采用优先路由机制，将流量导向与请求处于同一区域和可用区的副本。随着副本逐步迁入 Azure，Azure 所能承载的流量比例也随之递增，直到最后一个副本迁移完毕，旧区域彻底停止接收请求。每个仓库都会经历这一过程：

一个仓库都不能少

那么，剩下的 0.00001% 仓库该怎么办？

在我们现有的架构中，跨多个区域托管的仓库其写入吞吐量上限相对较低。Spokes 的 Quorum 虽然只需要多数派同意即可提交写入，但它在继续处理之前会等待每一个健康的副本做出响应（直至超时），因此慢速副本会拖慢每次写入，而不是被允许掉队。在实践中，这意味着最慢的健康副本决定了每次写入的延迟；又因为每次写入在提交前都会持有整个仓库范围的全局锁，它同时也决定了吞吐量的上限。一旦某个副本位于目标 Azure 区域，它与我们自建数据中心的距离大约有 1,000 英里，单次网络往返延迟约为 25 毫秒，而一次写入通常需要多次网络往返。对于一个本就接近吞吐量上限的繁忙仓库而言，每次串行写入根本无法承受这种开销，否则每次 push 都会让它进一步落后于请求积压。

这正是我们的第一条原则——最小化协调——大显身手之处。具体到此场景就是：不要对无需串行化的内容进行串行化。Git 的一致性要求是针对每个引用（reference）独立适用的。一旦服务器接受了对某个引用的更新，此后任何客户端都不应再看到旧值，只能看到该值或更新的值。无论该更新是 fast-forward、force push 还是分支删除，这一原则均成立。但在绝大多数情况下，对不同引用的更新彼此互不依赖，完全可以独立应用。

因此，我们将 Quorum 的作用范围从整个仓库缩小到了单个引用级别。

此前，在整个引用更新期间（包括网络往返）持有仓库级全局锁，实际上把 Quorum 变成了一个单通道、无序的请求队列，同一时刻只能有一个写入处于处理中（in flight）。队列中的每个写入所耗费的时间均由最慢的健康副本决定。随着写入速率增长，队列也随之拉长，每个写入排队等待的时间都长于前一个。

而在按引用串行化（per-reference serialization）模式下，只有竞争同一个共享引用的写入才需要相互等待。在实际操作中，仓库中的大多数写入针对的都是不同的引用——例如 Agent 和人类开发者向各自独立的分支推送代码——因此它们不再需要彼此阻塞等待。尽管每次写入仍需承担跨区域的网络延迟，但针对不同引用的写入可以同时并行处理。这彻底消除了仓库层面的单一瓶颈。在类似负载下对比两种方案，可以直观地看到队列积压与超时的改善情况：

这些改进使得仓库的写入吞吐量上限提升幅度大致与并发写入的引用数量成正比，从而顺利将最后这批遗留仓库迁入 Azure。在基于真实流量的内部测试中，针对不同引用的写入操作稳定展现出了 3 到 4 倍的吞吐量提升，同时延迟更低。

独立扩展读取能力

大多数仓库都呈现严重的读多写少特性，读写比通常在 100:1 到 1,000:1 之间。如今，读写请求访问的是相同的副本，导致它们竞争相同的系统资源，无法做到独立扩展。

将存储与计算解耦改变了这一局面。一旦仓库数据能够持久化存储在 Azure Blob Storage 中并随每次传入写入即时更新，我们便可以开始将读流量从写 Quorum 中剥离出来，针对两者各自所需的计算资源进行独立水平扩容。例如，读操作如果能获得比目前多得多的内存缓存将大有裨益，但在现有架构下这些内存还要预留给压缩优化（compaction）操作。一旦二者分离，各自就能拥有专属的配置。

第一步是将对象存储纳入 Quorum 的成员之一，但作为一种特殊的节点，我们称之为“虚拟副本（virtual replica）”。代表该虚拟副本的计算节点是我们首个完全无状态的临时节点（ephemeral node）。它作为连接对象存储的桥梁，负责流式传输传入的写入数据，且不承担任何读服务。正因为它是 Quorum 的正式成员，写入只有在对象存储中确保持久化后才会返回确认。

这虽然会为每次写入增加少许延迟，但就像应对跨区域网络 RTT 一样，按引用串行化可以防止其演变成吞吐量瓶颈。此外，虚拟写入副本还会将针对不相交引用的事务合并为发向对象存储的单批次网络 I/O，因此繁忙的仓库只需付出一次网络往返代价，即可完成众多并发写入，而非每次写入单独往返一次。目前，虚拟写入副本还承担着压缩优化的职责，只读副本可直接继承压缩后的状态。

引入虚拟写入副本后，我们便能够根据仓库的实际需求，按需水平扩展任意数量的“虚拟只读副本（virtual read replicas）”。每个只读副本从 Azure Blob Storage 预热其缓存，在无需加入 Quorum 的情况下直接开始承接读流量，它们将对象存储视作唯一信任源（source of truth），并在后台与对象存储进行本地状态对齐。

这一步释放了巨大的只读扩展潜力。在内部测试中，我们的分布式负载测试工具此前在当前架构下经常触及系统瓶颈；而在面对虚拟只读副本时，测试工具自身反而先达到了性能极限。当然，这也会带来一些后续步骤中需要解决的挑战：

  1. 虚拟只读副本初期是在后台与对象存储对齐状态，这非常适用于某些读负载，但并不能通吃所有场景。要支撑所有类型的读请求，必须将这种状态对齐转为内联机制（in-line），仅拉取自副本上次同步以来发生变更的引用状态与数据。
  2. 尽管我们通过转向按引用粒度模型大幅提升了写入吞吐量，但 Quorum 机制本身仍然是进一步提升吞吐量的限制因素。

我们该如何将写入吞吐量推向更高层次？

下一步是什么

今天的 GitHub 建立在 Spokes Quorum 之上。多年来，Spokes 的职责已远不止是一个 Quorum 协调器：它还是一个智能路由与负载均衡层，是一套供内部客户端访问和处理仓库数据、日调用量远超 1000 亿次操作的统一 API，并且包含了一组被称为“sweeper”的后台系统，负责在各个存储节点之间进行仓库布局、修复和再平衡。

Spokes 服务体系所包含的绝大部分能力，在新架构下依然能够像在现有架构中一样发挥作用。但我们也清醒地认识到：带领 GitHub 支撑起十亿个代码仓库的提交协议（commit protocol），已无法再带我们跨越至下一个十亿及更远的未来。

我们的演进之路尚未结束。在这些步骤就位后，下一个天花板就是写入吞吐量。目前每次写入依然需要等待各自持有仓库完整副本的节点确认，然而实际上写入操作中真正需要严格排序的内容少之又少。Git 对象是不可变的，并且以其内容哈希命名，因此它们无需经过共识协商即可直接落盘存储。真正剩下需要排序的数据非常精简，而难点在于如何在不将瓶颈转移到别处的前提下，尽可能快地完成这部分数据的定序。在白板上画出架构草图是一回事，在数以亿计、被开发者和 Agent 全天候不间断写入的真实仓库底层付诸实施则是另一回事。实现这一目标倒逼我们重新审视其中的二阶效应（second-order effects），例如持久化引用状态究竟该存放在何处，以及压缩和垃圾回收机制如何扩展。

Agent 带来的流量增长势头丝毫没有放缓，而上述每一个步骤在落地时，都能在无需等待最终形态完成的前提下，为代码仓库争取到更大的承载余量。

下周预告： 在即将到来的 GitHub Universe 大会以及本系列的最后一篇博文中，我们将带大家全面了解该架构的最终形态、它为写入吞吐量带来的变革、我们在此过程中必须解决的二阶问题，以及为何在技术栈的深处依然保留着一个（轻量级）共识集群。

感谢 Brian Celenza 对本文的贡献。

English / 原文

Over the past year, while Git traffic grew exponentially, 99.99999% of the Git repositories on GitHub began living in two places at once: our own data centers and Azure. The rest are among the busiest on the platform, and they’re held back by just one thing: a 25ms round trip.

In an earlier blog post, we laid out the two principles behind our next-generation storage architecture:

  • Minimize coordination
  • Decouple storage from compute

In this post, we’ll look at the immediate bridge between our old design and our new one. We’ll talk about how a change to the way our quorum serializes writes gets the busiest repositories into Azure, and how that same step opens the door to replicas that keep no persistent state of their own.

How we got here

The timing and shape of the move were driven by the agentic shift: rapidly growing traffic, a hard ceiling on the power capacity of our data centers and the surrounding geographical region, and physical supply chain constraints. At the beginning of our migration, the remaining capacity with all possible expansion gave us enough time to migrate safely, but not enough to do it slowly.

While we had prototyped new architectural approaches for scaling Git on cloud infrastructure, changing the storage architecture and the hosting platform at the same time, under that growth, carried too much risk. Given our scale and how critical the service is to the community, we needed to sequence the work carefully: move the data first, then evolve the architecture once it arrived.

Git operations are a Tier 0 service: they’re a core function of GitHub, and nearly everything in GitHub depends on the repository data. That meant the data, services, and every dependency had to move first. The first copy of a repository landed in Azure’s Central US region in August 2025. Today, most repositories have two of their five replicas in Azure, and Azure serves about half of Git traffic. For most of that time, Azure absorbed our growth while our own data centers held steady.

During the migration, traffic follows the data. Most repositories have five replicas to balance durability, availability, and scale. To minimize read latency, Spokes uses preferential routing to direct traffic to a replica in the same region and availability zone as the request. As replicas move to Azure, the share of the traffic that can be served from Azure grows with each one, until the last replica arrives and the old region stops receiving requests entirely. Every repository goes through this process:

No repository left behind

So, what about the remaining 0.00001% of repositories?

In our current architecture, a repository hosted in more than one region has a lower write throughput ceiling. Spokes’ quorum only needs a majority to commit a write, but it waits for every healthy replica to respond (up to a timeout) before moving on, so slow replicas delay every write rather than falling behind. In practice, that means the slowest healthy replica sets the latency of every write, and because each write holds a repository-wide lock until it commits, it also sets the throughput ceiling. Once a replica lives in the target Azure region, it’s about 1,000 miles from our data centers, making each network round trip around 25ms, and a write requires multiple round trips. A repository already near its throughput ceiling can’t absorb that on every serialized write; it would fall further behind with every push.

This is where our first principle, minimizing coordination, comes into play. Specifically for this case: don’t serialize what doesn’t need to be. Git’s consistency requirements apply to each reference individually. Once the server accepts an update to a reference, no client should see an older value afterward, only that value or a newer one. That holds whether the update is a fast-forward, a force push, or a delete. But in most cases, updates to different references don’t depend on each other and can be applied independently.

So, we’re changing the scope of the quorum from the whole repository to individual references.

Holding a repository-wide lock for the entire reference update, round trips included, turns the quorum into a single, unordered request queue with one write in flight at a time. Each write in that queue takes as long as the slowest healthy replica. As the write rate grows, so does the queue, and each write waits longer than the one before it.

With per-reference serialization, only writes that contend on a shared reference must wait for each other. In practice, most writes to a repository touch different references—for example, agents and developers pushing to their own branches—so they no longer wait on each other. Each write still experiences the cross-region latency but writes to different references can be in flight at the same time. That removes the single bottleneck on the repository. We can visualize how queueing and timeouts build up by comparing the two approaches side by side under similar load:

These changes raise a repository’s write throughput ceiling roughly in proportion to how many references are being written at once, and get the last remaining repositories into Azure. In internal testing with realistic traffic, writes to different references consistently saw 3–4x higher throughput at lower latency.

Scaling reads on their own

Most repositories are heavily read-skewed, with somewhere between 100 and 1,000 reads for every write. Today, reads and writes hit the same replicas, so they compete for the same resources and can’t be scaled independently.

Decoupling storage from compute changes that. Once repository data is durably stored in Azure Blob Storage and updated with each incoming write, we can begin to decouple read traffic from the write quorum, and independently scale the compute needed for each. For example, reads would benefit from far more in-memory cache than we give them today, but that memory is also needed for compaction. Once they’re separated, each gets its own configuration.

The first step is making object storage a member of the quorum, but a special one that we call a virtual replica. The compute that represents it is our first fully ephemeral node. It serves as a bridge to object storage, streaming incoming writes and serving no reads. Because it’s a member of the quorum, a write isn’t acknowledged until it’s durable in object storage.

That adds a little latency to each write, but just like the cross-region round trip, per-reference serialization keeps it from becoming a throughput bottleneck. The virtual write replica also coalesces transactions on disjoint references into a single set of network I/Os to object storage, so a busy repository pays for one round trip across many concurrent writes instead of one per write. For now, the virtual write replica also handles compaction, and read replicas inherit the compacted state.

With the virtual write replica in place, we can scale out as many virtual read replicas as a repository needs, on demand. Each read replica warms its cache from Azure Blob Storage and starts taking read traffic without joining the quorum, using object storage as the source of truth and reconciling its local state against it in the background.

This step unlocks significant read scale. In internal testing, our distributed load testing harness routinely found ceilings in the current architecture. Against virtual read replicas, the harness hit its own limits first. It also comes with a few challenges that are solved in subsequent steps:

  1. Virtual read replicas start out reconciling with object storage in the background, which makes them a good fit for some read workloads but not all of them. Serving every kind of read requires moving that reconciliation in-line, fetching only the reference state and data that changed since the replica last caught up.
  2. While we’ve incrementally increased write throughput by shifting to a per-reference model, the quorum remains the bounding factor for increasing further.

How can we push write throughput further?

What comes next

The GitHub of today was built on the Spokes quorum, and over the years Spokes has become much more than a quorum coordinator. It’s also an intelligent routing and load balancing layer, a unified API that internal clients use to access and transact repository data at well over 100 billion operations per day, and a collection of background systems we call “sweepers” that handle repository placement, repair, and rebalancing between storage nodes.

The vast majority of what lives under the Spokes service umbrella applies just as well to the new architecture as it does to the existing one. But we also recognize that the commit protocol that has taken GitHub to a billion repositories won’t take it to the next billion and beyond.

We’re not done yet. With these steps in place, the next limit is write throughput. Every write still waits on replicas that each hold a full copy of the repository, yet very little of a write actually needs to be ordered. Git objects are immutable and named by their contents, so they can be stored without anyone agreeing on them. What’s left is small, and the hard part is ordering it as fast as possible without moving the bottleneck somewhere else. Designing for that on a whiteboard is one thing. Doing it underneath a billion repositories that developers and agents are writing to around the clock is another. Getting there forced us to rethink the second-order effects too, like where durable reference state lives and how compaction and garbage collection scale.

Agent traffic isn’t slowing down, and each of these steps gives repositories more headroom as it lands, without waiting for the end state.

Coming next week: At GitHub Universe and in the final post in this series, we’ll take a tour through that end state, what it does for write throughput, the second-order problems we had to solve to get there, and why there’s still a (small) consensus group somewhere in the stack.

Thanks to Brian Celenza for their contributions to this post.