摘要：GPU 高带宽内存（HBM）既稀缺又昂贵，而随着聊天、Agent 循环以及文档问答等场景不断累积上下文状态，KV Cache 消耗了其中大量的空间。诸如 Mooncake、LMCache、FlexGen、InfiniGen 和 AttentionStore 等系统选择利用 CPU DRAM 和 SSD 来扩展 GPU 显存。然而更棘手的问题在于：哪些数据块（blocks）应该放置在哪个存储层级中、何时对它们进行迁移或逐出，以及预取（prefetching）是否真正有效。我们在一个涵盖 GPU HBM、CPU DRAM 和 SSD 的离散事件模拟器中对这些策略进行了深入研究，并通过随机森林执行时间预测器对模拟器进行了校准。我们横跨聊天（chat）、Agent 和文档问答（document question answering）三种工作负载，对比了最近使用（recency）、复用频率（reuse frequency）、预测复用（predicted reuse）以及带有预取前瞻窗口的 EWMA 预测器等策略。研究发现，分层存储能让每张 GPU 支持的并发会话数增加 73.02 倍，并将单会话成本降低 62.04 倍。但这些增益主要源于各层级 1 + 8 + 64 的容量扩展比例，而非放置策略本身的差异。在我们的测试配置下，当 Batch Size 为 1 时，解码阶段处于计算受限（compute bound）状态，因此数据放置策略几乎不会影响系统吞吐量，主要改变的是 PCIe 迁移流量和首字延迟（Time to First Token, TTFT）。在聊天负载中，Recency 策略产生的迁移流量比 Reuse Frequency 策略少 2.30 倍；而在 Agent 和文档问答负载中，Reuse Frequency 策略表现最佳。现有的预测复用（predicted reuse）策略在字节级行为上与 Recency 完全一致，这使得其对 Agent 的策略建议在实质上退化为了 Recency。真正的 EWMA 预测器虽然改变了行为模式，但在原本预期能获益的工作负载中，其表现依然落后于 Reuse Frequency。此外，预取机制所带来的带宽开销并不能带来相匹配的收益——在各种策略和缓存大小网格中，即使是知晓未来请求的理想预言机（Oracle），在迁移流量指标上也从未击败过「不预取（no prefetch）」方案。综上所述，针对特定工作负载的放置策略可以减少数据搬运，但现存实现中所推荐的预测复用和预取方案并未得到实验数据的支持。

Abstract:GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore extend GPU memory with CPU DRAM and SSD. The harder question is which blocks belong in each tier, when to move or evict them, and whether prefetching helps. We study these choices in a discrete event simulator spanning GPU HBM, CPU DRAM, and SSD, calibrated against a random forest execution time predictor. We compare recency, reuse frequency, predicted reuse, and an EWMA predictor with prefetch lookahead across chat, agent, and document question answering workloads. Tiering supports 73.02 times more concurrent sessions per GPU and lowers cost per session by 62.04 times. These gains come from tier capacities of 1 plus 8 plus 64, not placement policy. Decode is compute bound at batch size one in our setup, so placement barely affects throughput. It mainly changes PCIe migration traffic and time to first token. Recency produces 2.30 times less migration traffic than reuse frequency for chat. Reuse frequency performs best for agents and document question answering. The existing predicted reuse policy is byte identical to recency, making its agent recommendation effectively recency. A genuine EWMA predictor changes behavior but still ranks behind reuse frequency on the workloads prediction was expected to help. Prefetching does not justify its bandwidth cost. Across the policy and cache size grid, even an oracle with knowledge of future requests never beats no prefetch on migration traffic. Workload specific placement can reduce data movement, but the predicted reuse and prefetch recommendations are not supported as implemented.