🔶 Agent 正在涌向 Web，但 Web 尚未准备好

Konstantin Ryabitsev 的文章 Creepy crawlies 用数据量化了一个所有运营公开 Git 托管服务的人都已感受到一段时间的现象：Agent 正日益成为公开信息托管者的沉重负担。因此，越来越多的人派出越来越多的 Agent，进行越来越长的无效搜寻，流量正以惊人的速度增长。

流量增长并不完全是一个新问题。我们以前见过。大约在世纪之交时，计算机速度很慢，渲染 HTML 的成本很高。为缓解这一问题，Akamai 推出了 Edge Side Includes：一种允许缓存缓存页面 片段 的标记语言。假设你想在每个页面上复用计算成本高昂的天气报告，ESI 可以通过将该功能包裹在 ESI 标签中来实现这一点。

kernel.org 的 Git 是 ESI 缓存的有力候选，因为每个提交都是不可变的，并且拥有全局唯一 ID。正如 Konstantin 在文章中清楚说明的那样，git.kernel.org 的数据集很小，问题在于每一份数据都以 922 种不同方式呈现。这里有 150 万个不同的提交。这是一类特殊的缓存问题：键空间远大于内容空间。Edge Side Includes 正是为此而构建，因此我花了几天时间，看看能把网站性能提升到什么程度。

简而言之：linux.git 中每一个提交，在渲染并 gzip 压缩后，总计约为 22 GB 的 gzip 内容。整套内容可以放进一台机器的内存；一旦做到这一点，fork 数量就不再重要。

一个提交的多种变体

对于带有 id= 参数的任何页面，cgit 已经设置了十年的 Expires。问题在于，同一提交的每个变体都以不同方式为提交套上页面框架。这些副本之间的差异仅限于页面框架：

• 文档头部的标题、样式表、克隆 URL
• 含有仓库名称、描述、所有者和分支下拉菜单的页头表格
• ……

因此，计划是：在 diff 处切分页面，让成本低廉的框架按请求渲染，并将 diff 制作为一个片段；无论是哪个 fork 或分支请求它，其字节内容都完全相同。

cgit 端

在我的 fork 中，新增了一个 cgitrc 选项 enable-esi=1，以及一个新页面 esi-diff。

启用 ESI 后，提交页和 diff 页会与此前完全一样地渲染 head、页头、标签页、提交信息和 diff 控件。在原本 diff 开始的位置，cgit 会写入：

Varnish 端

VCL 位于 contrib/varnish/cgit-esi.vcl。其中有趣的部分如下：

片段 URL 仍会指定一个仓库，因为 cgit 需要通过它打开对象存储。Varnish 会忽略该仓库。它能够这样做，是基于这样一个假设：由 SHA 标识的 Git 提交，在每个仓库中都是相同的。第一个渲染某个提交的 fork，会为所有其他 fork 渲染它。

演示环境

演示运行在一台服役多年的 Linux 机器上：Intel i7-7700、四核、64 GB 内存。它可通过 kernel.varnish.org 公开访问。由于 cgit 是 CGI、每个请求都要进行一次 fork/exec，我使用了一个 lighttpd 实例来实际执行 CGI。Varnish 本身不能运行 CGI。lighttpd 前面部署着 varnishd。请注意其内置的 TLS 支持。

active.vcl 只有三行：版本、一个后端，以及对 contrib VCL 的 include。我增加了一个 vcl_deliver，为每个响应标记 X-Cache: hit 或 miss，方便查看。

我第一次哪里错了

我的第一次基准测试选取了 git.git 中一个 1.6 万行的提交，测量页面框架相对于完整页面的表现，并报告节省了 84%。太好了！但不幸的是，这是一个离群值。典型提交的成本低得多。因此，我改为随机抽样 150 个 git.git 提交。中位变更量：28 行。一般般。

本节及之后两节的数据在我的桌面机 Ryzen 9800X3D 上测得，而不是那台旧的 Intel 服务器。

| 中位数 | ---|--- 完整页面，ESI 关闭 | 6.61 ms 仅 ESI 框架 | 5.61 ms 仅 ESI 片段 | 1.42 ms

框架成本是片段成本的四倍。跨全部 922 个 fork 缓存片段可节省 24%，而一次冷缓存未命中反而会比完全不拆分差 7%。令人失望。对提交路径进行性能分析后，立即找到了成本所在：

框架耗时的 90% 来自一次调用，而且它与 diff 毫无关系。Decorations 是提交主题旁边的 tag: v2.54.0 标签。为了生成这些标签，Git 会读取仓库中的每一个 ref。我最初试图通过删除该调用来修复，结果几乎没有节省任何时间，因为如果没有其他调用者先执行，log-tree.c 中的 get_name_decoration 会自行调用 load_ref_decorations。它慢的实际原因更有意思。以下是它的每 ref 回调：

对于每个附注标签，它会解压标签对象，以找出该标签指向哪个提交。git.git 有 1008 个这样的标签。与此同时，packed-refs 已经在每个标签旁的 ^ 行记录了答案，而 ref 迭代器会通过 ref->peeled_oid 将其交给回调。Decoration 代码却忽略了它！改为使用它只需大约 20 行代码，并且能为每个提交页和日志页节省 4 ms。

同一个错误，犯了两次

peel 修复阻止了 Git 解压每一个标签对象，但没有阻止它向对象存储询问每个标签 是什么。每个 ref 仍有两个问题：ref 目标是什么类型，以及它 peel 后得到的对象是什么类型。每一个问题都需要在 pack 索引中进行一次二分查找。

在 git.git 中，该索引为 164 MB。在 linux.git 中，它为 389 MB，覆盖 1180 万个对象；一个提交页会遍历 946 个 ref，因此每次请求都要支付大约 1900 次这样的查找成本。在一个新进程中，该索引的页表每次也都要从头建立。框架耗时的 78%，只是为了在主题行旁绘制 tag: v6.18。这正是 CGI 开始显得笨重的地方。拥有一个可缓存这些内容的持久化 Web 应用会很有用。

这两个问题都不需要再问。packed-refs 只会为指向标签对象的 ref 写入 peeled 行，因此如果 ref 后端交给我们一个 peeled OID，其类型就是 OBJ_TAG，我们已经知道了。而 peeled 对象只会被用作 decoration 表中的键，没有任何地方会读取它的类型，因此 lookup_unknown_object() 给出的 struct 与之后由 lookup_commit() 填充的 struct 相同。一次对照测试让节省幅度一目了然：通过一个只有 1 个 ref、而非 946 个 ref 的仓库提供相同对象，框架成本仅为 1.7 ms。

git.git 从 6.5 ms 降至 2.2 ms。

让 cgit 飞起来

随着框架不再由 decorations 主导，大型 diff 再次变得突出。渲染一个 1.7 MB 的 diff 需要进行 10 万次 write，平均每次仅 18 字节。html_raw 会为每个 HTML 片段调用一次 write(2)，而 cgit 生成 HTML 的片段非常小。使用 64 KB 缓冲区可将其降至 27 次写入。减少 10 万次系统调用确实很有帮助。

结果

针对 git.git、在 CGI 环境下将输出写入管道时，每个请求的中位耗时：

原版	+peel	+buffer
提交页	7.67 ms	2.03 ms
1.6 万行 diff	95.87 ms	95.90 ms
1.6 万行并排 diff	181.19 ms	181.89 ms

这两个补丁是互补的，作用于不同页面。peel 修复针对的是拥有大量标签的仓库中每请求的固定成本。buffer 针对输出量，因此树页面无法从第一个补丁中获益，却能从第二个补丁中完全受益。

归根结底，性能如何？

如果页面整体都在缓存中——框架和片段都是如此——这台演示机器可提供约 每秒 21,000 个页面，即 3 至 5 Gbit/s，具体取决于提交内容有多大。在四个老旧核心上完成 TLS 终止、ESI 组装和在线 gzip。网卡是千兆网卡，因此在这一工作负载下，网络会远早于 Varnish 达到极限。

不过，爬虫从不会两次请求同一个 URL。片段已被缓存并可随时提供；框架则没有。因此，一次爬取的成本是每页一次 cgit 渲染。而 cgit 仍然很慢，大约比 Varnish 慢 20 倍。这才是有用的结论，因为 cgit 是可以修复的。Decoration 优化让性能从每秒 200 个框架提升到每秒 1,000 个。

片段是成本更高的一半，且其成本不会变低。Decoration 修复对它没有作用，因为片段没有需要 decoration 的主题行。它变得更便宜的方式是：所有 922 个 fork 只渲染一次，而不是渲染 922 次。框架降至 2 ms 后，解析 cgitrc 已不再免费：924 个仓库 stanza 共 137 KB，每次 CGI 调用都要重新读取，耗时 1.3 ms，几乎与渲染页面一样久。如果我见过低垂的果实，那这就是了。

更大的机器能做什么

框架渲染极易并行化。每次请求都 fork 和 exec，没有共享状态。因此它会随核心数扩展，直到内存带宽或进程创建速率达到极限。这里的测量结果是：修复后每个物理核心每秒 250 个框架，修复前为 50 个。如果假设 2026 年的一个核心能完成与 2017 年核心大致相同的工作量，那么工作量应当或多或少随核心数量扩展。因此，一台 64 核系统应能提供约每秒 1.6 万次渲染。

这没有解决什么，以及下一步

我学到了一件令人意外的事：如今 Linux 上的 fork 比过去便宜得多。Fork 真的、真的很快。Exec cgit 的成本约为 0.33 ms。当一个框架需要 15 ms 时，这只是噪声。现在框架只需 2 ms，它占到了 15%，常驻 cgit 看起来便很有吸引力。因为持久化进程无需在每个请求中重新为 389 MB pack 索引触发缺页，它可以保留查找表、各种缓存值等。

patch 和 rawdiff 页面完全不需要 ESI。它们是没有链接的纯文本，因此已经与仓库无关；根据对象 ID 对它们进行哈希，只需编写 VCL，不涉及 cgit。

这里还有大量可以做的事。但我相当确信，如果你真的想做，可以让这个系统跑满一条 10Gbps 链路。这应该能为团队争取大约 6 个月，直到下一次它再次崩溃。

自行运行

使用子模块构建 cgit，在 cgitrc 中设置 enable-esi=1 和 virtual-root，在 VCL 中 include contrib/varnish/cgit-esi.vcl，并将 Varnish 部署在前端。手册页中有一个名为 EDGE SIDE INCLUDES 的章节介绍详细信息，tests/t0112-esi.sh 则准确展示了框架和片段的样子。contrib/perf/ 中的三个性能补丁独立于上述所有内容；无论是否运行缓存，它们都会带来帮助。

可在此查看基于 git.kernel.org 仓库列表构建的 Varnish ESI 演示：kernel.varnish.org

该配置为 Varnish Cache 制作和测试。它很可能无需修改就能运行在 Vinyl Cache 上，你只需要以某种方式配置 TLS 支持。

想了解更多？

如果你有兴趣进一步讨论这一话题，可通过 per.buer @ varnish-software.com 联系我。或者，你也可以联系 Varnish 团队。

Konstantin Ryabitsev's post Creepy crawlies puts numbers on something anyone running a public git host has felt for a while: agents are an increasingly heavy burden on anyone publicly hosting information. So an ever increasing number of people are sending out more and more agents on longer and longer goose-chases and the traffic is increasing at an alarming rate.

Increasing traffic is not an entirely new problem. We’ve seen it before. Back around the turn of the century, computers were slow. Rendering HTML was expensive. To alleviate this Akamai introduced Edge Side Includes ; a markup language which allows a cache to cache fragments of a page. Say you want to reuse the computationally expensive weather report on every page. ESI can help you do this by wrapping the feature in ESI tags.

The kernel.org git is a strong candidate for ESI caching because each commit is immutable and has a globally unique ID. As Konstantin clearly expresses in his post, git.kernel.org’s dataset is small, the problem is that every piece of data is presented in 922 different ways. There are 1.5 million distinct commits. That is a caching problem of a particular kind: the key space is much bigger than the content space. Edge Side Includes are built for exactly this, so I spent a few days to see how far I can get making the website go faster.

TL:DR: every commit in linux.git, rendered and gzipped, is about 22 GB of gzipped content. You can hold the entire thing in RAM on one machine, and once you do, the fork count stops mattering.

Variations of a commit

cgit already sets a ten year Expires on any page that has an id= parameter. The problem is that each variant of the same commit frames the commit in a different way. What differs between those copies is confined to the frame of the page:

• the title, stylesheet, clone URLs in the document head
• the header table with repository name, description, owner and the branch dropdown
• …

So the plan was: cut the page at the diff, let the cheap frame be rendered per request, and make the diff a fragment whose bytes are identical no matter which fork or branch asked for it.

The cgit side

In my fork there is one new cgitrc option, enable-esi=1, and one new page, esi-diff.

With ESI on, the commit and diff pages render the head, the header, the tabs, the commit info and the diff controls exactly as before. Where the diff would start, cgit writes

The Varnish side

The VCL lives in contrib/varnish/cgit-esi.vcl. The interesting bits:

The fragment URL still names a repository, because cgit needs one to open the object store. Varnish ignores the repository. It can do that based on the assumption that a git commit, identified by the SHA, is identical in every repo. The first fork to render a certain commit renders it for all of them.

The demo

The demo runs on one well worn Linux box: an Intel i7-7700, four cores, 64 GB of memory. It’s publicly available as kernel.varnish.org. Since cgit is CGI, with a fork/exec for each request, I have a lighttpd instance to do the actual CGI. Varnish can’t do CGI on its own. In front of lighttpd sits varnishd. Note the built-in TLS support.

active.vcl is three lines: the version, a backend, and an include of the contrib VCL. I added a vcl_deliver that stamps X-Cache: hit or miss on every response for your viewing pleasure.

Where I was wrong the first time

My first benchmark picked a 16 thousand line commit in git.git, measured the frame against the whole page, and reported an 84% saving. Whohoo! This was unfortunately an outliner. The cost of the typical commit is much, much lower. So I sampled 150 random git.git commits instead. Median change: 28 lines. Meh.

This section and the two after it are measured on my desktop, a Ryzen 9800X3D, not the old Intel-based server

| median
---|---
whole page, ESI off | 6.61 ms
ESI frame only | 5.61 ms
ESI fragment only | 1.42 ms

The frame cost four times what the fragment cost. Caching the fragment across all 922 forks would save 24%, and a cold miss would be 7% worse than not splitting at all. Disappointing. Profiling the commit path found the cost immediately:

Ninety percent of the frame is one call, and it has nothing to do with the diff. Decorations are the "tag: v2.54.0" labels next to a commit subject. To produce them, git reads every ref in the repository. My first attempt to fix this was to delete the call, which saved ~nothing, because get_name_decoration in log-tree.c calls load_ref_decorations itself if nobody else has. The actual reason it is slow is better. Here is its per-ref callback:

For every annotated tag, it inflates the tag object to find out which commit it points at. git.git has 1008 of them. Meanwhile packed-refs already records the answer on a ^ line next to each tag, and the ref iterator hands it to the callback in ref->peeled_oid. The decoration code ignores it! Using it instead is about twenty lines, and shaves off 4 ms on every commit and log page.

The same mistake, twice

The peel fix stopped git inflating every tag object. It did not stop it asking the object store what every tag was. Two questions survive per ref: what type is the ref target, and what type is the thing it peels to. Each one is a binary search through the pack index.

On git.git that index is 164 MB. On linux.git it is 389 MB covering 11.8 million objects, and a commit page walks 946 refs, so it pays about 1900 of those searches. Every request. In a fresh process, the page table for that index is built from scratch each time too. Seventy-eight percent of the frame, to draw "tag: v6.18" next to a subject line. This is where the CGI becomes cumbersome. Having a persistent web application that can cache this would have been useful.

Neither question needs asking. packed-refs only writes a peeled line for refs that point at a tag object, so if the ref backend handed us a peeled oid, the type is OBJ_TAG and we already know it. And the peeled object is only ever used as a key in the decoration table. Nothing ever reads its type, so lookup_unknown_object() gives you the same struct that a later lookup_commit() will fill in. A control run makes the size of it obvious: serve the same objects through a repository with one ref instead of 946 and the frame costs 1.7 ms.

git.git goes from 6.5 ms to 2.2 ms.

Making cgit go brrrr

With the frame no longer dominated by decorations, the large diffs stood out again. Rendering a 1.7 MB diff took 100k writes averaging 18 bytes each. html_raw calls write(2) once per fragment of HTML, and cgit emits HTML in very small fragments. A 64 KB buffer takes that to 27 writes. Shaving off 100k system calls really helps.

Results

Median per request, output to a pipe as under CGI, against git.git:

| stock | +peel | +buffer
---|---|---|---
commit page | 7.67 ms | 2.03 ms | 1.86 ms
16k-line diff | 95.87 ms | 95.90 ms | 77.97 ms
16k-line side-by-side diff | 181.19 ms | 181.89 ms | 93.18 ms

The two patches are complementary and land on different pages. The peel fix targets fixed per-request cost on repositories with many tags. The buffer targets output volume, which is why the tree page gains nothing from the first and everything from the second.

At the end of the day, how is performance?

If the page is entirely in cache, frame and fragment both, the demo box serves about 21,000 pages a second , which is 3 to 5 Gbit/s depending on how fat the commits are. TLS terminated, ESI assembled, gzip on the wire, on four ancient cores. The NIC is a gigabit, so on that workload the network runs out long before Varnish does.

A crawler never asks for the same URL twice, though. The fragment is cached and ready to go; the frame is not. So the cost of a crawl is one cgit render per page. And cgit is still slow. About 20x slower than Varnish. Which is the useful conclusion, because cgit is fixable. The decoration work took us from 200 to 1,000 frames a second.

The fragment is the expensive half and it does not get cheaper. The decoration fixes do nothing for it, because a fragment has no subject line to decorate. It gets cheaper by being rendered once for all 922 forks instead of 922 times. With the frame down at 2 ms, parsing cgitrc is no longer free: 924 repository stanzas is 137 KB re-read on every CGI invocation, and it costs 1.3 ms. Nearly as much as rendering the page. That is a low hanging fruit if I ever saw one.

What a bigger box would do

Frame rendering is embarrassingly parallel. A fork and exec per request, no shared state. So it scales with cores until memory bandwidth or the process rate gives out. Measured here: 250 frames/s per physical core after the fixes, 50 before. If we assume a 2026 core does about the same amount of work as a 2017 core, the amount of work should more or less scale with the number of cores. So a 64 core system should give us about 16k renders per second.

What this does not do, and what is next

One surprising thing I learned. Forking on Linux is a lot cheaper today than it used to be. Forking is really, really fast. Exec'ing cgit costs about 0.33 ms. When a frame took 15 ms that was noise. Now that a frame takes 2 ms it is fifteen percent, and a resident cgit starts to look very appealing. Because a persistent process would stop re-faulting the 389 MB pack index on every request, it could keep lookup-tables various caches values, etc.

The patch and rawdiff pages need no ESI at all. They are plain text with no links, so they are already repository neutral, and hashing them on the object id is a VCL that didn’t involve cgit.

There are tons of things that can be done here. But I’m pretty certain one can make this system saturate a 10Gbps link, if you really want to. That should buy the team about 6 months until the next time it breaks.

Running it yourself

Build cgit with the submodule, set enable-esi=1 and a virtual-root in cgitrc, include contrib/varnish/cgit-esi.vcl in your VCL, and put Varnish in front. The man page has a section called EDGE SIDE INCLUDES with the details, and tests/t0112-esi.sh shows exactly what the frames and fragments look like. The three performance patches in contrib/perf/ are independent of all of that and help whether or not you run a cache.

See the Varnish ESI demo built from git.kernel.org's repository list here: kernel.varnish.org

This setup was made and tested for Varnish Cache. It will likely run without modification on Vinyl Cache, you just need to set up TLS support somehow.

Want to learn more?

I'm at per.buer @ varnish-software.com if you're interested in discussing this topic further. Alternatively you can reach out to Varnish in general.