Need help tuning cache in llama-server
英文标题:Need help tuning cache in llama-server
Hey I am running a few models on a strix halo box. Especially for the larger models (like Qwen 3.5 122B) they work okayish performance wise if the cache is utilised properly but a full cache miss at 100k context causes roughly 10-20 minute of PP time - which is extremely annoying. I will first show
打开原文相关 2 · 机会 0
中文
Hey I am running a few models on a strix halo box. Especially for the larger models (like Qwen 3.5 122B) they work okayish performance wise if the cache is utilised properly but a full cache miss at 100k context causes roughly 10-20 minute of PP time - which is extremely annoying. I will first show
中文全文翻译正在补齐,可先查看下方英文原文。
English / 原文
Hey I am running a few models on a strix halo box. Especially for the larger models (like Qwen 3.5 122B) they work okayish performance wise if the cache is utilised properly but a full cache miss at 100k context causes roughly 10-20 minute of PP time - which is extremely annoying. I will first show