摘要：长上下文 LLM 推理受到二次方注意力计算以及不断增长的 KV-cache 成本制约。现有稀疏注意力和 KV 压缩方法通常根据运行时注意力分数、观测窗口、校准提示词或学习得到的门控机制，决定保留哪些 token 或注意力头，这使得头部诊断依赖输入且部署成本较高。我们提出 Autonomy-of-Heads（AoH），一种基于 Query-Key 投影谱几何来识别检索头和流式处理头的数据无关方法。AoH 定义核注意力算子 $M_h = W_K^{h\top}W_Q^h$，并将其有效秩作为衡量注意力头功能的权重空间指标：集中的谱表明仅存在少数占主导地位的 Query-Key 匹配方向，并与检索头相关；而弥散的谱则表明不存在占主导地位的全局匹配方向，并与流式处理头相关。我们进一步推导出一种高效的 $d_\text{head}$ 维计算方法，避免构建完整的 $d_\text{model}\times d_\text{model}$ 矩阵。我们在多个模型上开展了广泛实验，结果表明：在 50% 稀疏度下，AoH 平均可保留完整注意力性能的 96.5%；同时，预填充和解码延迟最多可分别降低 41.4% 和 66.0%；在 256K token 条件下，KV-cache 内存可减少 50.0%。

Abstract:Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs. Existing sparse attention and KV-compression methods typically decide which tokens or heads to preserve from runtime attention scores, observation windows, calibration prompts, or learned gates, making head diagnosis input-dependent and costly to deploy. We propose Autonomy-of-Heads (AoH), a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections. AoH defines the kernel attention operator $M_h = W_K^{h\top}W_Q^h$ and uses its effective-rank as a weight-space measure of head function: concentrated spectra indicate a small number of dominant query-key matching directions and are associated with retrieval heads, whereas diffuse spectra indicate the absence of a dominant global matching direction and are associated with streaming heads. We further derive an efficient $d_\text{head}$-dimensional computation that avoids constructing the full $d_\text{model}\times d_\text{model}$ matrix. We conducted extensive experiments across models demonstrating that at 50% sparsity, AoH retains 96.5% of Full Attention performance on average while reducing prefill and decode latency by up to 41.4% and 66.0%, respectively, and KV-cache memory by 50.0% at 256K tokens.