利用 vLLM 双向 KV 传输优化多轮 Agent 负载的首字延迟（TTFT）

多轮 Agent 工作负载依赖于随着 Agent 推理、调用工具并整合先前结果而不断积累的上下文。在每个新轮次中，客户端都会重新发送对话历史记录，包括系统指令、先前的提示词、工具调用结果以及模型回复，这导致不断增长的上下文被反复处理。在Prefill-Decode（P-D）分离架构中，每个请求被拆分到专用节点上：Prefill 节点处理提示词并构建其键值（KV）缓存，随后 Decode 节点获取这些 KV 块并生成响应。

然而，传统的解耦式服务架构仅在一个方向上传输 KV 缓存：即从 Prefill 节点到 Decode 节点。这种单向数据流给多轮工作负载带来了两方面的效率低下问题。首先，在生成响应后，Decode 节点同时持有提示词和新生成 Token 的 KV 缓存条目。而在处理下一轮时，Prefill 节点无法获取这些条目，因此必须为上一轮生成的响应 Token 重新计算 KV 投影。其次，Decode 节点维护着会话状态，而 Prefill 节点并不维护，这增加了活动会话在 Prefill 节点上发生缓存逐出（Cache Eviction）的概率。一旦发生完全缓存未命中，Prefill 节点必须为整个对话历史重新计算 KV 投影，包括系统指令、先前的用户请求、工具结果和模型生成的响应。随着对话内容的增长，这种冗余计算会消耗更多加速器算力，并显著增加首字延迟（TTFT）。

在构建对话式助手和 Agent 应用时，较长的交互历史可能会通过重复的 Prefill 计算增加延迟并消耗加速器算力。在本文中，我们将展示**双向 KV 缓存传输（bi-directional KV cache transfer）**如何让 Prefill 和 Decode 节点跨轮次复用先前计算的上下文，从而提高加速器利用率，并在对话增长时帮助应用保持敏捷的 TTFT 表现。

在我们使用 Amazon Elastic Compute Cloud (Amazon EC2) P5en 实例的 1P1D 部署的多轮基准测试中，双向 KV 传输将 Qwen3-32B 的 TTFT 降低了高达 1.6 倍，将 zai-org/GLM-5-FP8 的 TTFT 降低了高达 3.3 倍（参见性能数据部分）。测试结果会因工作负载和配置而异。这些实例通过 AWS Elastic Fabric Adapter（AWS EFA）互连，这是一种在 AWS 服务器间使用的低延迟远程直接内存访问（RDMA）网络。在受支持的实例类型上，AWS EFA 支持 GPUDirect RDMA 和 Neuron Direct RDMA，从而在 Prefill 和 Decode 节点的 GPU 或 AWS Trainium 设备内存之间直接移动 KV 块。

我们最初在通过 EFA 连接的 AWS Trainium 实例集群上构建并验证了该功能，并将其作为一项与加速器无关的通用能力贡献给了 vLLM。此处展示的结果来自于配备 EFA 的 Amazon EC2 P5en GPU 集群，表明这些优势同样适用于 GPU。本文将详细解释双向 KV 传输架构、路由器元数据流、KV 块保留与重计算控制机制、性能测试结果以及如何复现该基准测试。

双向 KV 传输

双向 KV 传输是一种机制：Decode 节点在后续的对话轮次中，将先前计算好的 KV 传回给 Prefill 节点。这样可以避免在 Prefill 节点上重新计算共享上下文，并通过可配置的 KV 重计算阈值进行控制。它缓解了前文所述的两种计算冗余场景。为了实现这一点，KV 必须在产生它的轮次结束后继续留存，并在感知 KV 缓存的路由器协调下沿反方向流动。

面向双向传输的 NIXL KV 连接器增强

我们扩展了 NVIDIA Inference Xfer Library (NIXL) KV 连接器，以允许 Prefill 节点消费远程 KV 块，类似于 Decode 节点消费 Prefill 节点提供的 KV 块。这不是一个微不足道的改动，因为这两种节点类型在架构上截然不同：Prefill 节点负责计算新提示词 Token 的 KV 表征，并能提供完整的提示词 KV 块；Decode 节点则缺乏生成完整 KV 表征的能力（或缺乏其计算效率），只能提供先前缓存的 KV 块。因此，Prefill 和 Decode 节点在计算远程与本地 KV 块的元数据方面仍然承担着各自的角色与逻辑。然而，底层的实际 KV 传输机制（即 NIXL READ）在 P->D 和 D->P 两个方向上是通用的。

下图首先以蓝色展示了现有的数据流：路由器将请求发送给 Prefill 节点，Prefill 节点计算提示词 KV 并返回其块元数据，随后路由器将请求和元数据转发给 Decode 节点，执行标准的 P→D NIXL READ。带有编号的红色路径展示了双向 KV 传输的新增流程：(1) Decode 节点在生成完成后保留符合条件的 KV 块，(2) 在流式响应中返回其元数据，(3) 路由器为下一次 Prefill 请求缓存该元数据。随后，Prefill 节点 (4) 在应用本地缓存匹配后检查远程可复用的 KV，(5) 当可复用 Token 数量达到阈值时排队发起异步接收，并在计算剩余后缀之前 (6) 执行 D→P NIXL READ。

图 1：标准的 prefill-to-decode 流程（蓝色）与双向 KV 传输新增流程（红色）

感知 KV 缓存的路由器（KV-cache-aware router）

vLLM 包含了连接器端对双向 KV 传输的支持，而跨对话轮次协调 KV 复用则需要一个感知 KV 缓存的路由器。vLLM GitHub 提供了一个示例代理实现，实现了本文中使用的流程，可供调整以适配您的环境。

感知 KV 缓存的路由器是位于客户端、Prefill 节点和 Decode 节点之间的一个轻量级有状态控制平面组件。它在内存中维护从客户端提供的 conversation_id 到 Decode 节点 kv_transfer_params 的映射关系。路由器仅存储块位置元数据，而不存储 KV 张量，并且该元数据是从响应流中直接获取的，无需单独向 Decode 节点发起查询。

下图展示了路由器如何跨对话轮次管理 KV 元数据。在第一轮中，缓存查询未命中，请求遵循标准的 P→D 流程。随后路由器从响应流中保存 Decode 节点的元数据。在后续轮次中，路由器利用 conversation_id 取回该元数据，以便 Prefill 节点复用 Decode 节点的 KV。在生成完成后，路由器会刷新元数据以备下一轮使用。

图 2：跨对话轮次的感知 KV 缓存路由器元数据流

块保留生存时间（Block retention TTL）

通常情况下，Decode 实例在生成结束时便立即释放请求对应的块。在双向 KV 传输模式下，它会将这些块保留一段可配置的生命周期（decoder_kv_blocks_ttl，默认 480 秒），并发布其位置与过期时间。这样，该对话的 KV（包括提示词前缀与生成的响应）在下一轮到达前都保持可用。对于在生存时间（TTL）过期前到达的新一轮交互，Prefill 节点将复用该 KV；若超出 TTL 到达，则降级为常规 Prefill。由于该 TTL 是固定的且不会自动续期，因此应根据轮次之间的预期时间间隔进行调优，以在 KV 复用率与加速器显存占用之间取得平衡。自然的下一步改进是将心跳续租机制扩展到 Decode 端，这样 Decode 实例就可以在对话变为空闲时立即回收块，而无需等待固定计时器结束。

KV 重计算阈值（KV recompute threshold）

KV 复用并不总是比本地重计算更快。对于较短的可复用前缀，互连与协调开销可能会超过重新计算这些 Token 所需的时间。kv_recompute_threshold 设置（默认为 64 个 Token）用于控制这一决策。当待计算的 Token 数量低于该阈值时，Prefill 节点会进行重计算；当等于或高于该阈值时，则从 Decode 节点拉取 KV。部署时可以根据模型计算时间、KV 大小以及网络互连性能对该值进行调优。

性能数据

在评估中，我们使用了通过 EFA 互连的 Amazon EC2 P5en 实例，在 1P1D 部署上运行 Qwen3-32B 和 GLM-5-FP8 模型。为了模拟 Prefill 节点已逐出本地缓存，或路由器选择了未持有该对话 KV 块的其他 Prefill 节点的场景，我们通过 --no-enable-prefix-caching 在 Prefill 节点上禁用了前缀缓存。

在这两款模型中，开启双向 KV 传输均减少了多轮对话中的 Prefill 计算量，且收益通常随着提示词长度的增加而提升。在该功能关闭（OFF）时，Prefill 实例在每一轮都必须重新计算不断增长的全部上下文，导致 TTFT 急剧攀升。Qwen3-32B 的 TTFT 从 2k 长度下的 158 ms 上升到 24k 下的 778 ms，而 GLM-5-FP8 则从 2k 下的 500 ms 升至 20k 下的 2,334 ms。而在开启（ON）状态下，Decode 实例保留的 KV 被复用而非重新计算，使 TTFT 曲线平缓得多。Qwen3-32B 在相同范围内仅从 122 ms 增至 460 ms（在最长提示词下约降低 1.6 倍），而 GLM-5-FP8 则从 358 ms 增至 702 ms，在长提示词下比纯重计算路径最高快 3.3 倍。Prefill 耗时的细分数据解释了背后的原因。

双向传输虽然增加了一小段 KV 传输时间（Qwen3-32B 约为 30 ms 至 130 ms；GLM-5-FP8 约为 29 ms 至 110 ms），但它省去的 Prefill 计算开销远大于这一 KV 传输时间。因此，开启传输时的总 Prefill 持续时间（Qwen3-32B：128 ms 至 414 ms；GLM-5-FP8：316 ms 至 541 ms）显著低于仅依靠重计算的路径（Qwen3-32B：153 ms 至 744 ms；GLM-5-FP8：500 ms 至 2,331 ms）。规模更大的 GLM-5-FP8 模型展现出了更大的绝对与相对差距，因为其更高的单 Token 重计算开销使得随着模型和上下文规模的扩大，KV 复用能够带来更高的回报。

Qwen3-32B

下图绘制了 Qwen3-32B 的 TTFT 随提示词长度变化的曲线，对比了双向 KV 传输开启（ON）与关闭（OFF）的情况。

图 3：开启和关闭双向 KV 传输时 Qwen3-32B TTFT 与提示词长度的关系

下一张图表将相同测试下的 Prefill 耗时拆解为重计算开销与 KV 传输开销两部分。

图 4：Qwen3-32B prefill 耗时拆分为重计算与 KV 传输两部分

两张图表共同表明：开启该功能时，TTFT 保持近乎平缓，而随着上下文增长，关闭该功能的路径则急剧上升，原因在于微小的 KV 传输时间所带来的收益远超其消除的大量重计算开销。

zai-org/GLM-5-FP8

以下两组视图展示了 GLM-5-FP8 的对应表现。第一张图绘制了开启与关闭该功能时 TTFT 随提示词长度的变化。

图 5：开启与关闭 KV 复用时 zai-org/GLM-5-FP8 的 TTFT 对比

第二张图将 Prefill 耗时拆解为重计算与 KV 传输开销。

图 6：zai-org/GLM-5-FP8 prefill 耗时拆解

GLM-5-FP8 展现了比 Qwen3-32B 显著得多的差距，在长提示词下 TTFT 降低高达约 3.3 倍，这是因为其更高的每 Token 重计算时间使得 KV 复用在上下文长度增长时能够释放出更大的性能收益。

如何运行基准测试

Multi-turn agentic workloads build on accumulated context as an agent reasons, invokes tools, and incorporates earlier results. With each new turn, the client resends the conversation history, including system instructions, prior prompts, tool results, and model responses, causing the growing context to be processed repeatedly. In Prefill-decode (P-D) disaggregation, each request is split across specialized nodes: the prefill node processes the prompt and builds its key-value (KV) cache, then the decode node retrieves those KV blocks and generates the response.

Conventional disaggregated serving, however, only transfers KV cache in one direction: from prefill to decode node. This one-directional flow creates two inefficiencies for multi-turn workloads. First, after generating a response, the decode node holds KV cache entries for both the prompt and the newly generated tokens. When processing the next turn, the prefill node cannot retrieve those entries, so it must recompute the KV projections for the previous response tokens. Second, the decode node maintains the session state while the prefill nodes don’t, increasing the probability of cache eviction for active sessions on the prefill nodes. On a full cache miss, the prefill node must recompute KV projections for the entire conversation history, including system instructions, prior user requests, tool results, and model-generated responses. As the conversation grows, this redundant work consumes more accelerator capacity and increases time to first token (TTFT).

When you build conversational assistants and agentic applications, longer interaction histories can increase latency and consume accelerator capacity through repeated prefill computation. In this post, we show how bi-directional KV cache transfer lets prefill and decode nodes reuse previously computed context across turns, which improves accelerator utilization and helps applications sustain responsive TTFT as conversations grow.

In our multi-turn benchmarks on a 1P1D deployment using Amazon Elastic Compute Cloud (Amazon EC2) P5en instances, bi-directional KV transfer reduced TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8 (see the Performance data section). Results vary with workload and configuration. The instances are connected with AWS Elastic Fabric Adapter (AWS EFA), a low-latency remote direct memory access (RDMA) network used across servers in AWS. On supported instance types, AWS EFA supports GPUDirect RDMA and Neuron Direct RDMA to move KV blocks directly between GPU or AWS Trainium device memory on the prefill and decode nodes.

We originally built and validated the feature on AWS Trainium instance clusters connected with EFA and contributed it to vLLM as an accelerator-agnostic capability. The results presented here are from an Amazon EC2 P5en GPU cluster with EFA, showing that the benefits also carry over to GPUs. This post explains the bi-directional KV transfer architecture, router metadata flow, KV-block retention and recompute controls, performance results, and how to reproduce the benchmark.

Bi-directional KV transfer

Bi-directional KV transfer is a mechanism in which a decode node returns previously computed KV to a prefill node on subsequent turns of a conversation. This avoids recomputing shared context on the prefill node and is governed by a configurable KV recompute threshold. It mitigates both computational-redundancy scenarios described earlier. To make it work, the KV must survive past the turn that produced it and travel in the reverse direction, coordinated by a KV-cache-aware router.

NIXL KV connector enhancements for bi-directional transfer

We extended the NVIDIA Inference Xfer Library (NIXL) KV connector to let prefill nodes consume remote KV blocks, similar to how decode nodes already consume KV blocks that the prefill node provides. This is not a trivial change, because the two node types are architecturally different: prefill nodes compute KV representations for new prompt tokens and can provide complete prompt KV blocks. Decode nodes lack the capability (or efficiency) to generate complete KV representations and can only serve previously cached KV blocks. As a result, prefill and decode nodes still carry their own roles and logic for computing the metadata of remote and local KV blocks. The actual KV transfer mechanism (a NIXL READ), however, is common to both the P->D and D->P directions.

The following diagram first shows the existing flow in blue. The router sends the request to prefill node, prefill node computes the prompt KV and returns its block metadata, and the router forwards the request and metadata to decode node for the standard P→D NIXL READ. The numbered red path illustrates the bi-directional KV transfer additions: (1) decode node retains eligible KV blocks after generation, (2) returns their metadata in the streaming response, and (3) the router caches that metadata for the next prefill request. Prefill node then (4) checks the remotely reusable KV after it applies local cache matches, (5) queues an asynchronous receive when the reusable token count meets the threshold, and (6) performs a D→P NIXL READ before computing the remaining suffix.

Figure 1: Standard prefill-to-decode flow (blue) and the bi-directional KV transfer additions (red)

KV-cache-aware router

vLLM includes connector-side support for bi-directional KV transfer, while coordinating KV reuse across conversation turns requires a KV-cache-aware router. vLLM GitHub provides an example proxy that implements the flow used in this post which can be adapted for your environment.

The KV-cache-aware router is a lightweight, stateful control-plane component between the client, prefill, and decode nodes. It maintains an in-memory mapping from the client-provided conversation_id to decode’s kv_transfer_params. The router stores only block-location metadata, not KV tensors, and obtains that metadata from the response stream rather than separately querying decode node.

The following diagram shows how the router manages KV metadata across conversation turns. On the first turn, the cache lookup misses and the request follows the standard P→D flow. The router then saves decode’s metadata from the response stream. On later turns, the router retrieves that metadata using the conversation_id so prefill can reuse KV from decode. It then refreshes the metadata after generation for the next turn.

Figure 2: KV-cache-aware router metadata flow across conversation turns

Block retention TTL

Normally a decode instance frees a request’s blocks as soon as generation finishes. Under bi-directional KV transfer mode, it holds them for a configurable lifetime (decoder_kv_blocks_ttl, default 480 seconds) and publishes their location and expiry. The conversation’s KV, both prompt prefix and generated response, then stays available for the next turn. For a turn that arrives before the time to live (TTL) expires, prefill node reuses the KV. One that arrives later falls back to normal prefill. Because this TTL is fixed and not renewed, tune it to the expected time between turns to balance KV reuse against accelerator-memory usage. A natural next improvement is to extend the heartbeat-renewed lease to the decode side. A decode instance could then reclaim blocks as soon as a conversation goes idle rather than at the end of a fixed timer.

KV recompute threshold

KV reuse isn’t always faster than local recomputation. For a short reusable prefix, interconnect and coordination overhead can exceed the time needed to recompute the tokens. The kv_recompute_threshold setting, which defaults to 64 tokens, controls this decision. Prefill node recomputes when the token count to compute is below the threshold and pulls KV from decode node when it is at or above the threshold. Deployments can tune this value based on model computation time, KV size, and interconnect performance.

Performance data

For evaluation, we used Qwen3-32B and GLM-5-FP8 models on a 1P1D deployment using Amazon EC2 P5en instances connected with EFA. To emulate scenarios in which the prefill node has evicted its local cache or the router selects a different prefill node that does not have the conversation’s KV blocks, we disabled prefix caching on prefill node with --no-enable-prefix-caching.

Across both the models, turning on bi-directional KV transfer reduces prefill computation on multi-turn conversations, and the benefit typically grows with prompt length. With the feature OFF, the prefill instance recomputes the entire grown context each turn, so TTFT rises steeply. Qwen3-32B climbs from 158 ms at 2k to 778 ms at 24k, and GLM-5-FP8 from 500 ms at 2k to 2,334 ms at 20k. With it ON, the decode instance’s retained KV is reused instead of recomputed, keeping TTFT far flatter. Qwen3-32B rises only from 122 ms to 460 ms over the same range (about a 1.6x reduction at the longest prompt), while GLM-5-FP8 rises from 358 ms to 702 ms and runs up to 3.3x below the recompute path at long prompts. The prefill-duration breakdown explains why.

Bi-directional transfer adds a small KV-transfer time (about 30 ms to 130 ms for Qwen3-32B; about 29 ms to 110 ms for GLM-5-FP8), but the prefill compute it removes far outweighs that KV-transfer time. As a result, the total prefill duration with transfer ON (Qwen3-32B: 128 ms to 414 ms; GLM-5-FP8: 316 ms to 541 ms) stays well below the recompute-only path (Qwen3-32B: 153 ms to 744 ms; GLM-5-FP8: 500 ms to 2,331 ms). The larger GLM-5-FP8 model shows a bigger absolute and relative gap, since its higher per-token recompute time makes KV reuse pay off even more as models and contexts scale.

Qwen3-32B

The following chart plots TTFT against prompt length for Qwen3-32B, comparing bi-directional KV transfer ON versus OFF.

Figure 3: Qwen3-32B TTFT versus prompt length with bi-directional KV transfer on and off

The next chart breaks the prefill duration for the same runs into its recompute and KV-transfer components.

Figure 4: Qwen3-32B prefill duration split into recompute and KV-transfer components

Together the charts show TTFT staying nearly flat with the feature ON while the OFF path climbs steeply as context grows, because the small KV-transfer time is far outweighed by the recompute it removes.

zai-org/GLM-5-FP8

The same two views follow for GLM-5-FP8. The first plots TTFT against prompt length with the feature ON versus OFF.

Figure 5: zai-org/GLM-5-FP8 TTFT with KV reuse on and off

The second breaks the prefill duration into its recompute and KV-transfer components.

Figure 6: zai-org/GLM-5-FP8 prefill duration breakdown

GLM-5-FP8 shows a much larger gap than Qwen3-32B, up to about 3.3x lower TTFT at long prompts, because its higher per-token recompute time makes KV reuse pay off even more as context length grows.

How to run benchmark

To reproduce our 1P1D benchmark configuration, one prefill node and one decode node, using Amazon EC2 P5en instances with EFA, follow the step-by-step instructions in the Reproducing our results guide. The guide covers environment setup, launching the vLLM prefill and decode servers, running the benchmark, and collecting results. The feature can run on other supported configurations, but performance results will vary depending on the hardware, model, workload, and network configuration.

Conclusion

In this post, we showed how bi-directional KV cache transfer can reduce TTFT as an agent’s conversation history grows. In P-D disaggregated deployments, the feature lets prefill node reuse KV blocks retained by decode node across turns, reducing repeated computation. Configurable block-retention and recompute thresholds determine when reuse occurs. On a 1P1D deployment using Amazon EC2 P5en instances connected with EFA, at the longest prompt lengths tested, we observed TTFT reductions of 1.6x for Qwen3-32B and 3.3x for zai-org/GLM-5-FP8. Results depend on the workload and deployment configuration. To implement this in your environment, use vLLM v0.21.0 or later with NixlConnector, enable bidirectional_kv_xfer on both prefill and decode servers, use a kv-cache-aware router, and follow the GitHub reproduction guide.

Acknowledgments

We thank the vLLM community for their support in maintaining this feature.

---

About the author