更小、更快、更安全：大规模运行 Kimi 和 GLM
英文标题：🔶 Smaller, faster, safer: running Kimi and GLM at scale
Cloudflare Workers AI 通过 KV 缓存 FP8 量化、模型权重 INT4 压缩和共享 KV 缓存完整性检查，提升了 Moonshot Kimi K 系列及 Z.ai GLM 模型的大规模推理效率。Kimi K2.6 使用 FP8 KV 缓存后，可容纳的上下文从约 68.6 万 token 提升至约 137 万 token，并在高并发下实现更高吞吐和更低单 token 成本，准确率基本不变。GLM 5.2 的权重从 FP8 压缩为 INT4 后，模型检查点缩小约 40%，低并发解码吞吐最高提升 55%，同时评测质量与 FP8 基本一致。为避免更多请求共享缓存带来的数据完整性风险，Cloudflare 还部署了缓存页校验机制，其吞吐和 p95 延迟开销均低于 1%。
650 B
1.6 KB
976 B
2.3 KB
1.1 KB
3.0 KB
41.9 KB
41.9 KB
1.1 KB
66.4 KB
中文
Workers AI 在靠近用户的 Cloudflare 数据中心 GPU 上，为全球一些最优秀的开放模型运行推理服务。其中能力最强、需求也最高的两个模型，是 Moonshot 的 Kimi K 系列和 Z.ai 的 GLM。它们都是大型、长上下文、混合专家模型，使用体验出色；但由于内存限制，也极难以高效率提供服务。
我们此前曾介绍过如何在 Workers AI 上部署大型模型，以及如何将推理的预填充和解码阶段分离，以充分发挥每张 GPU 的性能。本文将介绍在这些基础之上叠加使用的三项技术，用于将这些模型装入内存并维持高速运行：量化 KV 缓存、压缩模型权重，以及由于前两项技术会让更多请求共享硬件资源，保护这些请求共享的缓存。这些优化使我们能够在不改变模型准确率的情况下，以更低成本服务更多客户。
我们的全部实验和生产流量均使用开源推理服务框架 SGLang 运行和测试。我们发现 SGLang 提供了市场上最佳的性能，并且与 SGLang 团队密切合作，将补丁和新功能上游贡献出去，让开源社区也能使用我们的成果。
量化 KV 缓存
模型生成文本时，会将已处理的每个 token 的注意力键（K）和值（V）存储在一个称为 KV 缓存的结构中。KV 缓存使模型能够延续长对话，而不必在生成每个新 token 时重新读取整个上下文。对于长上下文模型而言，缓存会迅速增长，而且通常最先耗尽 GPU 内存的是 KV 缓存，而不是模型权重。
默认情况下，缓存以 16 位精度（BF16）存储。我们改用 8 位浮点数（FP8，e4m3）存储，这使其大小减半。对于 Kimi K2.6，这将我们能够在内存中容纳的上下文从约 68.6 万 token 提升至约 137 万 token，增加了一倍。
有必要准确说明收益来自哪里，因为它并非原始速度的提升。量化缓存会为每个 token 增加少量工作，因为 FP8 注意力内核在读取数值时必须进行转换。它改变的是可同时驻留在内存中的请求数量。以下测量针对在解耦式 H200 部署上运行 Kimi K2.6 解码的结果，直接比较注意力内核：
| 并发请求数 | BF16 KV 缓存（tok/s） | FP8 KV 缓存（tok/s） |
|---|---|---|
| 1 | 137 | 125 |
| 8 | 731 | 689 |
| 16 | 1,106 | 1,028 |
| 32 | 1,558 | 1,489 |
| 64 | 内存不足 | 2,192 |
在任一单独的并发级别下，BF16 的每 token 速度会快几个百分点。但 BF16 在 32 个并发请求时就耗尽了缓存，无法接纳第 33 个请求；FP8 则可继续扩展到 64 个请求，并达到每秒 2,192 个 token，较 BF16 的峰值高约 41%，每 token 成本则低约 30%。由于我们将预填充和解码运行在独立资源池中，因此可以将这项技术应用在收益最大的场景：预填充受计算能力而非内存限制，因此我们在该阶段保留 BF16 缓存，以维持其略高的吞吐量。
如果这会改变模型答案，那么上述优化就没有意义，因此我们进行了验证。在整个评测套件中，FP8 和 BF16 缓存的表现难以区分：
| 基准测试 | BF16 KV | FP8 KV |
|---|---|---|
| GSM8K | 94.24 | 94.09 |
| ARC-Easy | 89.06 | 89.14 |
| ARC-Challenge | 66.72 | 67.49 |
| MMLU | 89.11 | 89.04 |
| MMLU-Pro | 80.29 | 79.29 |
| mcxams（内部基准） | 61 / 63 | 61 / 63 |
| 工具调用有效性 | 92.2% | 92.6% |
压缩模型权重
KV 缓存是 GPU 内存的一项需求，模型权重则是另一项。对于 GLM 5.2，我们将权重从 8 位浮点数压缩为 4 位整数（INT4），且没有准确率损失。检查点大小从 705 GB 缩小至 421 GB，约减少 40%；在 8 路张量并行部署中，每张 GPU 的内存占用从约 88 GB 降至 52 GB，从而能在同一硬件上为约 118 万 token 的 KV 缓存留出空间。
在整个评测套件中，INT4 和 FP8 权重的表现难以区分：
| 基准测试 / 能力 | 指标 | FP8 | INT4 |
|---|---|---|---|
| GSM8K | 精确匹配 | 94.39% | 93.56% |
| GSM8K | 弹性匹配 | 94.24% | 93.48% |
| ARC-Easy | 准确率 | 86.62% | 86.15% |
| ARC-Easy | 归一化准确率 | 84.51% | 85.19% |
| ARC-Challenge | 准确率 | 64.93% | 64.85% |
| ARC-Challenge | 归一化准确率 | 67.24% | 66.64% |
| MMLU | 平均值 | 86.60% | 86.54% |
| MMLU-Pro | 精确匹配 | 80.80% | 80.47% |
| mcxams（内部基准） | 通过 | 62 / 63 | 62 / 63 |
更小的权重会使解码阶段更快，原因很明确：生成每个 token 都意味着从 GPU 内存中流式读取模型权重，因此解码速度受内存带宽限制。传输的数据更少，每个 token 就能更快产生。这种效果在低并发下最明显，因为此时单请求延迟最重要：
| 并发请求数 | GLM FP8（tok/s） | GLM INT4（tok/s） | INT4 提升幅度 |
|---|---|---|---|
| 1 | 60 | 92 | +55% |
| 8 | 425 | 513 | +21% |
| 16 | 683 | 825 | +21% |
| 32 | 994 | 1,267 | +27% |
| 64 | 1,672 | 1,933 | +16% |
预填充的表现有所不同。它受计算能力限制，而 INT4 权重必须先展开才能供模型进行矩阵乘法，因此这一额外步骤会让预填充变慢，而不是变快：GLM 在 FP8 下的预填充吞吐约为每秒 10,160 个 token，在 INT4 下则为 8,660 个 token。与 KV 缓存的情况一样，解耦式设计使其成为一种选择而非妥协：我们在 INT4 占优的解码阶段使用 INT4，在 FP8 占优的预填充阶段使用 FP8。我们运行的每项基准测试中，模型准确率与 FP8 模型的差距均在 0.8 个百分点以内，因此质量表现难以区分。
保护共享 KV 缓存
上述两项技术具有同样的效果：它们使更多请求能够同时共享一张 GPU 的内存。这种效率正是核心目标，但也意味着数百个请求正在读取和写入同一个物理 KV 缓存的页面。使这一过程高效的机制——分页注意力、连续批处理、缓存复用——都依赖于绝对正确的簿记管理；而在我们的请求量级下，即使十亿分之一的错误也会频繁出现。
因此，我们构建了 KV 缓存完整性检查，作为一道防御层。基本思路很直接：每个物理缓存页面都会获得一个标签，并在页面被重新分配时变更；服务器则记录每个请求预计使用的页面及其标签。在受支持的解码操作从缓存中读取之前，系统会检查这些映射关系。如果任何内容不匹配，受影响的请求将被中止，而不会被允许返回来自错误页面的数据。
决定一项安全检查能否上线的问题，在于它的成本。我们在一个中等规模的生产模型上对其进行了测量，配置为两个预填充实例和两个解码实例，输入长度为 8,192 个 token，输出长度为 1,000 个 token：
| 并发数 | 吞吐变化 | p95 延迟变化 |
|---|---|---|
| 1 | −0.53% | +0.42% |
| 2 | −0.38% | +0.54% |
| 4 | −0.79% | +0.63% |
| 8 | −0.43% | +0.80% |
吞吐量和尾延迟的成本均低于 1%，即使 95% 置信区间的上限也保持在约 1%。我们通过将验证作为独立的批量检查运行，而不是将其融合进注意力内核，来维持较低的计算成本；后者会在 GPU 线程组之间引入竞争条件。该功能按部署启用，默认路径使用没有可测量开销的空操作追踪器，因此不需要该功能的部署无需承担任何成本。
下一步
高效服务前沿模型是一个不断变化的目标，这正是我们持续开展的工作。我们正在将 FP8 KV 缓存扩展到更多集群，在 Blackwell（NVIDIA 的 GPU 架构）上验证 NVFP4 权重，并致力于以几乎可以忽略的成本，在所有地方默认启用完整性检查。这些优化将使我们能够以更低成本、在保持相同准确率的前提下，继续服务更多客户。
如果你喜欢将最优秀的开放模型压缩进 GPU，并将其服务提供给数百万开发者这一类问题，欢迎加入我们。
English / 原文
Workers AI runs inference for some of the best open models in the world on GPUs in Cloudflare data centers close to your users. Two of the most capable, and most demanding, are Moonshot's Kimi K-series and Z.ai's GLM. They are large, long-context, mixture-of-experts models, and they are wonderful to use. They are also very hard to serve efficiently because of memory constraints.
We've written before about how we serve large models on Workers AI and about separating the prefill and decode phases of inference to get more out of each GPU. This post looks at three techniques we layer on top of that to fit these models into memory and keep them fast: quantizing the KV cache, compressing the model weights, and, because both of those pack more requests onto shared hardware, protecting the cache those requests share. These optimizations enable us to support more customers at lower costs, with no change in model accuracy.
All our experiments and production traffic are running and benchmarked with SGLang, an open-source inference serving framework. We found that SGLang offers the best performance in the market, and we work closely with the SGLang team to upstream patches and new features to make our work available to the open-source community.
Quantizing the KV cache
As a model generates text, it stores the attention keys (K) and values (V) for every token it has already processed in a structure called the KV cache. The cache is what lets the model extend a long conversation without re-reading the entire context on every new token. For a long-context model, it grows quickly, and it is usually the KV cache, not the model's weights, that fills up GPU memory first.
By default, the cache is stored in 16-bit precision (BF16). We store it in 8-bit floating point instead (FP8, e4m3), which halves its size. On Kimi K2.6, that raises the amount of context we can hold in memory from roughly 686,000 tokens to about 1.37 million, twice as much.
It's worth being precise about where the benefit comes from, because it isn't raw speed. Quantizing the cache adds a small amount of work per token, since the FP8 attention kernel has to convert values as it reads them. What it changes is how many requests we can keep resident at once. The following measurements are for Kimi K2.6 decoding on a disaggregated H200 deployment, comparing the attention kernels directly:
| Concurrent requests | BF16 KV cache (tok/s) | FP8 KV cache (tok/s) |
|---|---|---|
| 1 | 137 | 125 |
| 8 | 731 | 689 |
| 16 | 1,106 | 1,028 |
| 32 | 1,558 | 1,489 |
| 64 | Out of memory | 2,192 |
At any single concurrency level, BF16 is a few percent faster per token. But BF16 runs out of cache at 32 concurrent requests and can't admit a 33rd, while FP8 keeps going to 64 and reaches 2,192 tokens per second, about 41% higher than BF16's peak, for roughly 30% less cost per token. Because we run prefill and decode as separate pools, we can apply this where it helps most: prefill is compute-bound rather than memory-bound, so there we leave the cache in BF16 and keep its slightly higher throughput.
None of this would matter if it changed the model's answers, so we checked. Across our evaluation suite, FP8 and BF16 caches are indistinguishable:
| Benchmark | BF16 KV | FP8 KV |
|---|---|---|
| GSM8K | 94.24 | 94.09 |
| ARC-Easy | 89.06 | 89.14 |
| ARC-Challenge | 66.72 | 67.49 |
| MMLU | 89.11 | 89.04 |
| MMLU-Pro | 80.29 | 79.29 |
| mcxams (internal benchmark) | 61 / 63 | 61 / 63 |
| Tool-call validity | 92.2% | 92.6% |
Compressing the model weights
The KV cache is one demand on GPU memory; the model's weights are the other. For GLM 5.2, we compress the weights from 8-bit floating point down to 4-bit integers (INT4) with no loss in accuracy. The checkpoint shrinks from 705 GB to 421 GB, about 40%, and per-GPU memory across an 8-way tensor-parallel deployment drops from roughly 88 GB to 52 GB, which leaves room for around 1.18 million tokens of KV cache on the same hardware.
Across our evaluation suite, INT4 and FP8 weights are indistinguishable:
| Benchmark / Capability | Metric | FP8 | INT4 |
|---|---|---|---|
| GSM8K | Exact match | 94.39% | 93.56% |
| GSM8K | Flexible | 94.24% | 93.48% |
| ARC-Easy | Accuracy | 86.62% | 86.15% |
| ARC-Easy | Acc (norm) | 84.51% | 85.19% |
| ARC-Challenge | Accuracy | 64.93% | 64.85% |
| ARC-Challenge | Acc (norm) | 67.24% | 66.64% |
| MMLU | Average | 86.60% | 86.54% |
| MMLU-Pro | Exact | 80.80% | 80.47% |
| mcxams (internal benchmark) | Passed | 62 / 63 | 62 / 63 |
Smaller weights make the decode phase faster, and for a clear reason: generating each token means streaming the model's weights out of GPU memory, so decode speed is limited by memory bandwidth. Move less data and every token arrives sooner. The effect is largest at low concurrency, where per-request latency matters most:
| Concurrent requests | GLM FP8 (tok/s) | GLM INT4 (tok/s) | INT4 gain |
|---|---|---|---|
| 1 | 60 | 92 | +55% |
| 8 | 425 | 513 | +21% |
| 16 | 683 | 825 | +21% |
| 32 | 994 | 1,267 | +27% |
| 64 | 1,672 | 1,933 | +16% |
Prefill behaves differently. It is compute-bound, and INT4 weights have to be expanded back out before the model can multiply with them, so that extra step makes prefill slower rather than faster, GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4. As with the KV cache, the disaggregated design turns this into a choice rather than a compromise: we run INT4 for decode, where it wins, and FP8 for prefill, where it wins. Model accuracy stays within 0.8 points of the FP8 model across every benchmark we run, making its quality indistinguishable.
Protecting a shared KV cache
Both techniques above have the same effect: they let many more requests share one GPU's memory at the same time. That efficiency is the whole point, but it also means hundreds of requests are reading and writing pages of the same physical KV cache. The mechanisms that make this fast, paged attention, continuous batching, cache reuse, all rely on getting the bookkeeping exactly right, and at our request volumes, even a one-in-a-billion mistake would show up regularly.
So we built KV cache integrity checking as a layer of defense. The idea is straightforward: every physical cache page gets a tag that changes whenever the page is reallocated, and the server records which pages and tags each request expects to use. Before supported decode operations read from the cache, those mappings are checked. If anything doesn't match, the affected request is aborted rather than allowed to return data from the wrong page.
The question that decides whether a safety check ships is what it costs. We measured it on a mid-sized production model in a two-prefill, two-decode configuration, with 8,192-token inputs and 1,000-token outputs:
| Concurrency | Throughput change | p95 latency change |
|---|---|---|
| 1 | â0.53% | +0.42% |
| 2 | â0.38% | +0.54% |
| 4 | â0.79% | +0.63% |
| 8 | â0.43% | +0.80% |
The cost is under 1% on both throughput and tail latency, and even the upper bound of the 95% confidence interval stays near 1%. We kept it computationally cheap by running the validation as a separate batch check rather than fusing it into the attention kernel, which would have introduced a race between GPU thread groups. It's enabled per deployment, and the default path uses a no-op tracker with no measurable overhead, so deployments that don't need it pay nothing.
What's next
Serving frontier models efficiently is a moving target, and this is the ongoing work behind it. We're expanding FP8 KV caches across more of the fleet, validating NVFP4 weights on Blackwell (NVIDIAâs GPU architecture), and working toward making integrity checks something we can leave on everywhere at negligible cost. These optimizations will allow us to continue to support more customers at a lower cost and at the same accuracy.
If squeezing the best open models onto GPUs and serving them to millions of developers sounds like your kind of problem, come work with us.