极致优化 vLLM 中的 Kimi K3：借助 Coding Agent 实现解码吞吐与交互性能倍增

NVIDIA GB300 上的 Kimi K3 解码性能。

Coding Agent 现在已经能够协助完成模型优化中最艰难的环节：编写复杂的 GPU 内核（kernel）、执行跨算子融合（cross-operator fusions）、重叠计算与通信，以及在不同批大小（batch size）和 GPU 配置下进行调优。针对 Kimi K3，我们通过 Agent 辅助开发在 NVIDIA GB300 上构建了一条统一的 CuTe DSL 解码路径，覆盖了 8 卡或 16 卡 GPU 上的注意力机制（attention）、MoE 以及跨卡通信。这些优化在 16 卡 NVIDIA GB300 上配合 DSpark（合成 AL=6）、Batch 1 条件下实现了超过 840 tokens/s 的性能；在多个测试配置下带来了 1.5–2 倍 的解码吞吐提升；并在 AgentX 工作负载上实现了最高达 1.8 倍 的 P90 交互性能提升。

由 Agent 优化的推理代码正逐渐超出我们的代码审查与维护流程承载能力。逐行理解每个内核、排查潜在隐患并测试每种受支持配置的人工审查方式极其耗时。但另一方面，如果略过这一过程，又可能面临交付带有缺陷并在生产环境中发生故障的代码的风险。

要兼顾这两个目标，有着一条清晰可行的路线：我们需要转变组织性能优化工作的方式。在采用 Agentic 优化的同时，我们必须设计能够自动化验证可靠性和性能的系统，并在架构设计上从源头强制实现代码隔离与风险最小化。

所提议的快速通道模型优化 RFC为这种方法奠定了基础。我们的指导原则是在有效管控风险的同时降低代码审查负担。为此，我们采取以下措施：

1. 隔离变更。 将优化限定在模型专属代码之内。
2. 明确边界，保留核心特性。 明确定义受支持的硬件和工作负载，但同时完整保留生产级 LLM 部署所必需的核心功能。
3. 系统化验证正确性。 运行现成的性能与正确性基准测试套件，以确保代码完整性与可靠性，并保证在所有受支持的配置下均具有严格更优的性能表现。
4. 扩展验证能力。 允许贡献者在自己的 GPU 上运行标准评测，并在请求维护者审查前分享可复现的结果。

Kimi K3 是我们遵循该方法的首个贡献案例。本文将介绍我们如何优化 Kimi K3、如何建立交付与维护这些优化所需的信心，以及具体的性能表现。

目录

第一部分：优化 Kimi K3

本次贡献为 Kimi K3 引入了一条全新的解码路径，该路径使用针对特定子问题专门调优的 CuTe DSL 定制内核融合并替换了几乎所有算子。它面向 8 卡或 16 卡的 NVIDIA Blackwell GPU，适配 1 到 64 的解码批大小，并支持针对长上下文的可选解码上下文并行（decode context parallelism，DCP）。这些 GPU 共享一个支持多播（multicast）的 NVLink 域。更多详情请参阅该 PR。

Kimi K3 内核时间线：原生 vLLM 每层耗时 130 微秒并使用 23 个内核；专用解码耗时 69 微秒并使用 13 个内核。

1. 针对工作负载调优形状（Shapes）

不同的批大小需要采用不同的内核实现策略。对于注意力输入投影（attention input projections），我们在批大小为 1–2 时使用专用的微小矩阵内核，在更大批大小时使用 split-K 内核（将矩阵乘法拆分到更多 GPU 计算单元上）。所有此类投影均经过了精细调优，能够启动合适规模的内核，从而充分利用 GPU 硬件资源，并最大程度地重叠内存读取/通信与计算任务。

KDA 同样使用了一个经过优化的 CuTe DSL 解码内核，其任务调度针对每张 GPU 上的批大小和注意力头数量进行了定制调优。

针对长上下文场景，解码上下文并行（DCP）将 KV Cache 切分到多张 GPU 上。每张 GPU 计算其本地部分的注意力，并通过调优后的融合内核将局部结果与 v-proj 及门控（gating）相结合。

2. 重叠计算与通信

我们将注意力输出投影（output projection）直接与其全规约（all-reduce）以及随后的注意力残差计算进行了深度融合。GPU 内核将局部结果写入支持 NVLink 访问的缓冲区中，接收内核随即合并这些结果并继续执行后续计算。在某些小批次配置下，生产者和消费者甚至被融合到同一个内核中。数据交换通过无内存屏障（fence-less）的方式进行，开销极小。

在更大批次下，该实现会切换为两阶段规约（two-stage reduction）：各 GPU 先合并分配到的部分输出，然后再分发结果。这显著减少了 GPU 之间的数据搬运量。

3. 改进 MoE 调度与重叠

尽管专家层（experts）已经采用了张量并行（tensor parallelism），但基线实现在每张 GPU 上都会计算完整的路由器评分（router scores）和潜在输入投影（latent input projection）。我们将这些计算切分到各个 TP 秩（rank）上，并将每个 rank 的计算份额合并为单次矩阵乘法。GPU 之间通过 NVLink 交换结果。这减少了专家层执行开始前的重复计算、权重读取与内核启动开销。

共享专家（shared expert）运行在独立的 CUDA 流（stream）上。它的计算可以与路由计算或已路由专家的计算相互重叠（具体取决于配置）。调度逻辑精确控制每个操作占用的 GPU 资源比例，并经过严格调优以确保在没有资源争用的情况下实现负载均衡的并行计算。详情请参阅 experts_runtime。

4. 将后续权重预取至缓存

在小批次推理场景下，性能瓶颈通常受限于显存带宽（memory-bound）：即受限于从显存读取权重的速度。某些执行阶段无法完全打满可用的显存带宽，例如通信阶段：当通信进行时，我们可以在侧流（side stream）上运行专用内核进行权重预取，以确保通信一旦完成，下一个算子操作所需的权重在缓存中能获得高得多的命中率。

第二部分：快速且负责任地推进

Kimi K3 只是首批案例之一。我们相信社区可以协同共建，推动更多模型、GPU 和工作负载逼近性能物理极限（speed of light）。

不同的贡献者或 Agent 会发现不同的优化切入点：更优的通信调度，或是利好特定部署场景的算子融合。但如果每次贡献都建立在不同的假设前提、不完整的测试以及难以复现的基准测试之上，验证与维护工作很快就会成为整体瓶颈，甚至带来引入缺陷代码的风险。我们需要一种标准化的通用方式，将这些构想转化为用户可以在长期生产场景中信赖的稳定性改进。

这改变了我们对 Agent 交付成果的要求。我们应当对 Agent 产出的代码在测试、验证和规范层面设立明确预期。Agent 需要负责构建测试用例、执行全面的端到端评测与基准测试、排查异常故障，并清晰记录优化生效的先决条件。人类代码审查员依然把关顶层设计与验证依据，但可以从高层抽象入手，而不再需要陷入逐行审阅内核代码的泥潭。

依托所提议的快速通道 RFC，我们将这项工作围绕四大原则展开：

1. 隔离变更

将模型特有的优化严格限制在模型自身的代码范围内。这使得受影响的代码路径一目了然，并极大地缩小了审查员需要评估的影响半径（blast-radius）。

运行时隔离同样至关重要。审查员应当能够清晰追踪优化路径何时被激活、修改了哪些内部状态，以及执行流如何平滑回退至标准路径。

这一约束确实排除了一部分优化方案：输入预处理、采样（sampling）以及 KV Cache 管理往往会引入全局开销，而强隔离机制会将它们排除在外。这是架构设计上的必然取舍，并且在很大程度上是刻意为之的：这些组件在大部分 vLLM 部署中都是通用的，若为了特定工作负载随意改动它们，很可能会破坏数十种其他稳定的 vLLM 配置。针对这些核心组件的工作极其重要，理应由核心维护者深入参与，而不应通过快速通道推进。

2. 明确边界，但保留用户功能

一项优化并不需要覆盖所有部署场景才算有价值。它可以专门面向特定的 GPU 型号、并行配置或批大小区间。明确的边界划分不仅使调优工作更加聚焦，也为验证提供了清晰确定的测试集。

对于 K3，初始的硬件验证覆盖了搭载 TP8 和 TP16 的 GB300、最高 64 的解码批大小、搭配不同草稿长度（draft lengths）的推测解码，以及单个 NVLink 域内的解码上下文并行（DCP）配置。

这些边界划定了优化的适用场景，但绝不能以此为由削减“功能覆盖完整性”。生产级部署必须保留赋予服务其实用价值的底层功能。例如全上下文服务（full-context serving）、温度采样（temperature sampling）和结构化输出（structured output）等功能不可被“优化掉”或“选择性放弃”；它们与内核矩阵形状及数值精度检查一样，都是一等公民级别的核心特性，必须完整纳入验证计划之中。

3. 验证正确性

内核级单测与模型级评测关注的是不同维度的问题。内核测试验证不同形状和输入取值下的数值结果精度；模型评测则检验整条端到端执行路径是否完整保证了模型输出内容的正确性。二者不可偏废。

测试用例必须能够真实触发修改后的代码。针对小批次的优化必须在小批次场景下测试；注意力内核的变更必须覆盖长上下文。静默回退到原生路径的“伪测试”根本无法验证优化代码的有效性，这也是开发中常见的陷阱。

K3 的研发过程充分证明了这一点的重要性。我们在审查中发现，编译器曾一度将激活值读取指令移动到了同步等待指令之前。最终的修复方案是将此类内存加载限制为只针对静态权重生效。此类并发错误通常能够侥幸通过简单的功能测试，必须通过针对性审查与验证才能排查出来。

此外，诸如 MRCR 和 AIME25 等长上下文与长文本生成评测，暴露出了注意力和通信内核中敏感的数值精度偏差，而这些问题在使用模拟数据或固定时序的独立内核单测中是完全无法复现的。

4. 广泛检查性能

宣称的单点加速比仅仅是一个起点。我们更需要全面掌握变更在其影响的各类工作负载下的行为表现。这意味着需要对批大小、上下文长度以及所有受支持的并行配置进行全维度扫描。对 MoE 的性能测试必须包含真实的 Prompt 输入，因为专家路由的选择会直接影响权重的读取与计算复用比例。此外，纯解码阶段的基准数据也必须与混合 Prefill 和 Decode 的复合负载严格区分开来。

严谨可靠的验证必然会消耗大量 GPU 机时。相关的 自带 GPU 评测 RFC（Bring-Your-Own-GPU evaluation RFC） 提出了标准化评测镜像方案，允许贡献者在自己的硬件设备上运行基准测试。与 GitHub PR 挂钩的可复现测试报告包和提交记录，将极大增强验证结果的可信度。通过设置不同的评测梯度以及利用社区共享算力，有助于在控制 GPU 资源消耗并减轻官方 CI 基础设施压力的同时，大幅拓宽测试覆盖面。

第三部分：性能深度剖析

DSpark 下的解码性能

配合 DSpark，在受控的 AL=6 基准测试下，Kimi K3 在 16 张 NVIDIA GB300 GPU 上达到了 842 tokens/s 的解码速率。在相同的推测解码配置下，这比原生 vLLM 快 1.73 倍。完整结果表格请参阅该 PR。

Kimi K3 在 Batch 1、TP16、DSpark DL7 以及合成 AL=6 下的表现：解码吞吐为 842 tokens/s，而原生 vLLM 为 488，实现了 1.73 倍加速。

测试配置：TP16，DSpark 包含 7 个草稿 token，合成接受长度（AL）为 6，DL=7，ISL=1，OSL=1k，批大小为 1。TPS 衡量聚合解码吞吐量，不包含 prefill。

无推测解码下的解码性能

性能提升覆盖了所测试的整个批大小区间。TP16 在 批大小 16–64 时保持了 1.62–1.71 倍的性能增益，而 TP8 在相同区间内也实现了 1.31–1.43 倍 的加速。

批大小	TP8 tokens/s	TP8 加速比	TP16 tokens/s	TP16 加速比
1	193.3	1.58×	256.4	1.88×
2	317.8	1.50×	436.7	1.92×
4	540.6	1.55×	702.5	1.81×
8	820.1	1.45×	1,157.9	1.79×
16	1,274.8	1.43×	1,758.3	1.71×
32	1,702.6	1.33×	2,550.9	1.64×
64	2,375.9	1.31×	3,606.1	1.62×

未开启推测解码。吞吐量为整个 TP 组的总输出。

AgentX 上的性能提升

在 AgentX 基准测试中，优化后的 Kimi K3 实现还在并发度为 1 时实现了 1.8 倍的 P90 交互性能提升：达到 403.2 tokens/s/user（对比原生 vLLM 的 223.7 tokens/s/user）。该测试对比了在 16 卡 GB300 上的优化实现与此前在 8 卡 GB300 上的原生 vLLM 运行数据。

AgentX 回放了源自真实 Coding Agent 会话的流量模式，涵盖长上下文、多轮对话、工具调用停顿、并行子 Agent 以及 KV Cache 复用。它在精准还原这些复杂工作负载特征的同时，使用合成文本替换了隐私数据，从而能够在极其贴近真实生产环境的 Agent 流量下严苛测试推理服务系统的承载能力。[AgentX 评测方法说明](https://inferencex.semianalysis.c

Kimi K3 decode performance on NVIDIA GB300.

Coding agents can now help with some of the hardest parts of model optimization: writing complex GPU kernels, performing cross-operator fusions, overlapping computation with communication, and tuning execution across different batch sizes and GPU configurations. For Kimi K3, we used agent-assisted development to build a unified CuTe DSL decode path on the NVIDIA GB300, covering attention, MoE, and communication across 8 or 16 GPUs. The optimizations deliver over 840 tokens/s at batch 1 on 16 NVIDIA GB300 GPUs with DSpark (synthetic AL=6), 1.5–2× higher decode throughput in several tested configurations, and up to 1.8× higher P90 interactivity on the AgentX workload.

Agent-optimized inference code is outpacing our review and maintenance workflow. The manual review process of understanding every line of every kernel, mapping out potential issues, and testing every supported configuration, is very time-consuming work. On the other hand, if we omit this work then we risk shipping buggy software that could fail in production.

There is a clear line-of-sight on how to achieve both goals: we need to change how we organize performance work. Alongside agentic optimization, we need to design systems that allow automatic verification of reliability and performance, and we need to structure the software to enforce isolation and risk minimization by design.

The proposed fast-track model optimization RFC provides a foundation for this approach. Our guiding principle is to decrease review burden while managing risk. To accomplish this, we:

1. Isolate changes. Keep optimizations within the model’s code.
2. Bound the scope, preserve the core features. Define the supported hardware and workloads but preserve must-have features for production LLM use.
3. Validate correctness systematically. Run an established suite of performance and correctness benchmarks to ensure integrity and reliability, and strictly better performance on all supported configurations.
4. Scale validation. Let contributors run standard evaluations on their own GPUs and share reproducible results before requesting maintainer attention.

Kimi K3 is our first contribution following this approach. This post covers how we optimized Kimi K3, how we build the confidence needed to ship and maintain those optimizations, and the performance results.

Table of Contents

Part 1: Optimizing Kimi K3

This contribution introduces a new decode path for Kimi K3 which fuses and replaces almost all operations with customized CuTe DSL kernels, tuned for certain subproblems. It targets NVIDIA Blackwell GPUs with 8 or 16 GPUs, decode batches from 1 to 64, and optional decode context parallelism for long contexts. The GPUs share one NVLink domain with multicast support. For more details, please refer to the PR.

Kimi K3 kernel timeline: stock vLLM uses 130 microseconds and 23 kernels per layer; specialized decode uses 69 microseconds and 13 kernels.

1. Tune shapes for the workload

Different batch sizes need different kernel choices. For attention input projections, we use specialized small-matrix kernels at batch sizes 1–2 and split-K kernels at larger batches, which divide the matrix multiplication across more GPU workers. All such projections have been finely tuned to launch well-sized kernels that fully utilize GPU resources and overlap memory reads / communication and compute work as much as possible.

KDA also uses an optimized CuTe DSL decode kernel, with scheduling tuned for the batch size and number of attention heads on each GPU.

For long contexts, decode context parallelism (DCP) splits the KV cache across GPUs. Each GPU computes attention over its local portion, and a tuned fused kernel combines the partial results with v-proj and gating.

2. Overlap computation and communication

We fuse attention output projection directly to its all-reduce and the following attention-residual computation. GPU kernels publish partial results into NVLink-accessible buffers, and receiving kernels combine them and continue the computation. Some small-batch configurations fuse the producer and consumer into one kernel. Data exchange is done with a fence-less approach with minimal overhead.

At larger batches, the implementation switches to a two-stage reduction: GPUs first combine assigned portions of the output, then distribute the results. This reduces the data movement between GPUs.

3. Improve MoE scheduling and overlap

Although the experts already use tensor parallelism, the baseline computes the full router scores and latent input projection on every GPU. We split these calculations across the TP ranks and combine each rank’s share into one matrix multiplication. The GPUs exchange the results over NVLink. This reduces repeated computation, weight reads, and kernel launches before expert execution begins.

The shared expert runs on a separate stream. Its work overlaps with routing or routed-expert computation, depending on the configuration. The schedule controls how many GPU resources each operation uses, and is carefully tuned to ensure well-balanced parallelism without contention. Details are in experts_runtime.

4. Prefetching upcoming weights into cache

In small-batch inference, we are often memory-bound: limited by the speed with which we can load the weights from memory. Some phases cannot fully utilize the available memory bandwidth, such as communication: while this work happens we can run a dedicated kernel on a side stream to ensure that once the communication completes, the weights for the next operation will have a much higher cache hit rate.

Part 2: Moving Fast and Responsibly

Kimi K3 is just one of the first examples. We believe the community can build this together, to bring more models, GPUs, and workloads towards the speed of light.

Different contributors/agents will find different opportunities: a better communication schedule, or a fusion that helps a particular deployment. But if every contribution arrives with different assumptions, incomplete tests, and a benchmark that is hard to reproduce, the validation and maintenance work quickly becomes the bottleneck, or we risk shipping broken code. We need a common way to turn those ideas into improvements that users can rely on for their long-term production use-cases.

This changes what we ask agents to deliver. We should set clear expectations on testing, validations, and requirements for the code agents produce. Agents will construct tests, run comprehensive evaluations and benchmarks, investigate failures, and document the conditions under which the optimization works. Human reviewers still judge the design and the evidence, but they can start from a high-level abstraction, instead of a line-by-line code review.

Building on the proposed fast-track RFC, we organize this work around four principles.

1. Isolate changes

Keep model-specific optimizations within the model’s code. This makes the affected paths easier to identify and reduces the blast-radius a reviewer needs to consider.

Isolation also needs to hold at runtime. Reviewers should be able to trace when the optimized path is selected, what state it changes, and how execution returns to the standard path.

This constraint does exclude some optimizations: input preparation, sampling, and KV cache management can cause overhead that isolation excludes. This is a consequence of the design and is largely intentional: these components are all shared among most vLLM deployments, and changing them to optimize for a specific workload risks breaking dozens of other stable vLLM configurations. Work on these components is important, and should involve core maintainers much more closely than the fast-track would permit.

2. Bound the scope, but preserve the user’s features

An optimization does not need to improve every deployment to be useful. It can target a particular GPU, parallelism setting, or range of batch sizes. Clear boundaries make the work easier to tune and give us a defined set of configurations to validate.

For K3, the initial hardware validation covers GB300 with TP8 and TP16, decode batches up to 64, speculative decoding with various draft lengths, and decode context-parallel configurations within one NVLink domain.

These boundaries describe where the optimization applies, but not cutting ‘feature coverage’. Production deployment needs fundamental features that make the endpoint useful. Features like full-context serving, temperature sampling, and structured output cannot be “optimized away” or “opt-out”, they are first-class serving features and belong to the validation plan alongside kernel shapes and numerical checks.

3. Validate correctness

Kernel tests and model evaluations answer different questions. Kernel tests check numerical results across shapes and input values. Model evaluations check whether the complete execution path preserves the model’s answers. We need both.

The tests must actually exercise the changed code. A small-batch optimization needs evaluation at small batches. An attention kernel change needs long-context coverage. Tests that silently use the standard path provide little evidence about the optimization and are a common trap.

K3 development shows why this matters. Our audit discovered that the compiler moved activation reads ahead of a synchronization wait. The fix restricted that load type to static weights. Such errors can survive simple functional checks and require targeted investigation.

Additionally, long-context and long-generation evaluations such as MRCR and AIME25 surfaced sensitive numerical issues in attention and communication kernels that were not reproducible in standalone kernel tests using mock inputs and/or consistent launch timings.

4. Check performance broadly

A headline speedup is a starting point. We also need to know how the change behaves across the workloads it affects. That means sweeping batch sizes, context lengths, and supported parallelism configurations. MoE measurements should include real prompts because expert selection affects which weights are read and how much work can be reused. Pure decode results should also be distinguished from workloads that mix prefill and decode.

Reliable validation also takes GPU time. The related Bring-Your-Own-GPU evaluation RFC proposes standard evaluation images that contributors can run on their own hardware. Reproducible result bundles and submissions linked to GitHub PRs would make the evidence much more accessible. Different evaluation tiers and shared community compute would help expand coverage while managing GPU demand and reducing pressure on project-funded CI.

Part 3: Performance Deep Dive

Decode performance with DSpark

With DSpark, Kimi K3 reaches 842 tokens per second on 16 NVIDIA GB300 GPUs in the controlled AL=6 benchmark. That is 1.73× faster than stock vLLM with the same speculative decoding setup. Refer to the PR for a full results table.

Kimi K3 at batch 1 with TP16, DSpark DL7, and synthetic AL=6: 842 decode tokens/s versus 488 for stock vLLM, a 1.73x speedup.

Setup: TP16, DSpark with seven draft tokens, synthetic acceptance length of 6, DL=7, ISL=1, OSL=1k, batch size 1. TPS measures aggregate decode throughput and excludes prefill.

Decode performance without speculation

The improvements span across the tested batch range. TP16 retains a 1.62–1.71× gain at batches 16–64 , while TP8 delivers 1.31–1.43× over the same range.

Batch size	TP8 tokens/s	TP8 speedup	TP16 tokens/s	TP16 speedup
1	193.3	1.58×	256.4	1.88×
2	317.8	1.50×	436.7	1.92×
4	540.6	1.55×	702.5	1.81×
8	820.1	1.45×	1,157.9	1.79×
16	1,274.8	1.43×	1,758.3	1.71×
32	1,702.6	1.33×	2,550.9	1.64×
64	2,375.9	1.31×	3,606.1	1.62×

No speculative decoding. Throughput is for the entire TP group.

Improvement on AgentX

Our optimized Kimi K3 implementation also delivers 1.8× higher P90 interactivity at concurrency 1 on AgentX: 403.2 versus 223.7 tokens/s/user. This compares the optimized run on 16 GB300 GPUs with the earlier vLLM result on 8 GB300 GPUs.

AgentX replays traffic patterns from real coding-agent sessions, including long contexts, multiple conversation turns, tool pauses, parallel subagents, and KV-cache reuse. It preserves these workload patterns while replacing private content with synthetic text, testing the serving system under realistic agent traffic. AgentX methodology.

Kimi K3 AgentX results: P90 interactivity versus total throughput per GPU, comparing the earlier GB300 vLLM curve with specialized decode.

What's Next

For DeepSeek-V4.1-Flash, #61868 follows the K3 design patterns: behind the same opt-in decode_backend: latency_specialized switch, the implementation also supports up to 64 concurrent requests, seamless fallback for prefills and mixed batches, and speculative decoding. Instead of chasing a single hotspot, that executor cuts kernels and host work: it fuses all-reduce + mHC at every sublayer boundary, replaces the large-batch MoE and GEMMs with CuTe DSL kernels built for low-latency, deduplicates and shards the indexer’s work, and takes per-step work that stock vLLM runs eagerly on the CPU and replays it as a CUDA graph instead.

DeepSeek AgentX results on four GB300 GPUs: P90 interactivity versus total throughput per GPU with and without optimized decode.

Within two days of Meta’s Muse Spark Open release, our early results already exceeded 950 tokens per second per user , with seven draft tokens (K=7) and an acceptance length of six (AL=6). More optimizations are underway.

We are also releasing a similar mechanism to optimize the decode latency for GLM5.3 under a low-concurrency scenario too. We’ll share its optimization details and performance results in a follow-up.

For each contribution, we will continue to focus on concrete performance gains, production feature support, and reproducible correctness and performance checks.

Acknowledgements

We would like to thank Rohan Varma (NVIDIA) and Alex Steiner (NVIDIA) for their contributions to this work.

We would also like to thank Woosuk Kwon (Inferact), and Michael Goin and Lucas Wilkinson (Red Hat), for the model refactor and valuable feedback and support.