摘要：大语言模型（LLM）推理受限于自注意力机制的二次方规模增长以及 KV cache 的线性增长；随着上下文长度扩展，延迟、能耗和 GPU 显存需求均会增加。现有软压缩方法要么在推理时缺乏查询引导的记忆选择，要么在训练时没有采用面向答案的监督，要么将压缩过程与特定解码器架构紧密耦合。我们提出 Context-to-Answer-Aligned Memory Compression（CMC）框架，可将长输入上下文压缩为紧凑的 Context Memory Embeddings（CMEs），并与任意冻结解码器的嵌入空间对齐，从而在无需修改解码器权重的情况下降低推理成本。CMC 引入双层 KV cache，将问题引导的 CME 选择与局部上下文窗口相结合，并通过来自冻结 LLM 的面向答案蒸馏训练压缩器。在 9 种编码器—解码器组合和 4 个问答基准上的实验表明，CMC 始终优于基线方法：在 SQuAD 上，EM 最高提升 7.3 个点，F1 最高提升 4.0 个点；在生成 3,000 个 token 时，推理时间和能耗最高降低 20%，GPU 峰值预留显存最高降低 50%。消融研究进一步证实，每个架构组件及训练目标都对性能提升有所贡献。

Abstract:Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing latency, energy consumption, and GPU memory demand as context length scales. Existing soft-compression methods either lack query-guided memory selection at inference time, train without answer-targeted supervision, or couple compression tightly to a specific decoder architecture. We propose a Context-to-Answer-Aligned Memory Compression (CMC) framework, which compresses long input contexts into compact Context Memory Embeddings (CMEs) aligned to any frozen decoder's embedding space, reducing inference costs without modifying decoder weights. CMC introduces a two-tier KV cache that combines question-guided CME selection with a local context window, and trains the compressor with answer-targeted distillation from a frozen LLM. Experiments across nine encoder-decoder combinations and four QA benchmarks show that CMC consistently outperforms the baseline, achieving up to 7.3 EM and 4.0 F1 point gains on SQuAD, while reducing inference time and energy consumption by up to 20% and peak reserved GPU memory by up to 50% at 3,000 generation tokens. Ablation studies confirm that each architectural component and training objective contributes to the performance.