AMD llama.cpp: reducing MTP buffer overhead gave me 64K → 149K context for Qwen 27B
英文标题：AMD llama.cpp: reducing MTP buffer overhead gave me 64K → 149K context for Qwen 27B
Available context length with and without the patch: Model: QWEN 27B ROCm stock patched Vulkan stock patched IQ4_XS Pure, single 16GB GPU 19.456 76.032 68,352 78,592 Q6_K_L on 16GB + 12GB 64,256 149,248 68,864 151,296 The issue is that llama.cpp overestimates the memory needed for MTP compu
打开原文相关 2 · 机会 0
中文
Available context length with and without the patch: Model: QWEN 27B ROCm stock patched Vulkan stock patched IQ4_XS Pure, single 16GB GPU 19.456 76.032 68,352 78,592 Q6_K_L on 16GB + 12GB 64,256 149,248 68,864 151,296 The issue is that llama.cpp overestimates the memory needed for MTP compu
中文全文翻译正在补齐，可先查看下方英文原文。
English / 原文
Available context length with and without the patch: Model: QWEN 27B ROCm stock patched Vulkan stock patched IQ4_XS Pure, single 16GB GPU 19.456 76.032 68,352 78,592 Q6_K_L on 16GB + 12GB 64,256 149,248 68,864 151,296 The issue is that llama.cpp overestimates the memory needed for MTP compu