We built our own inference engine so a 32B model could keep 262K context inside a 16 GB GPU budget. Here's an uncut run.
英文标题：We built our own inference engine so a 32B model could keep 262K context inside a 16 GB GPU budget. Here's an uncut run.
Hey r/LocalLLaMA, Blockway here, a small team from Hong Kong. After our v1 post, the most common question was basically "ok, but why does this model exist?" Totally fair. Here's our honest answer. The problem Long agent sessions eat VRAM. On a normal dense model every layer keeps a KV cache, a
打开原文相关 3 · 机会 1
中文
Hey r/LocalLLaMA, Blockway here, a small team from Hong Kong. After our v1 post, the most common question was basically "ok, but why does this model exist?" Totally fair. Here's our honest answer. The problem Long agent sessions eat VRAM. On a normal dense model every layer keeps a KV cache, a
中文全文翻译正在补齐，可先查看下方英文原文。
English / 原文
Hey r/LocalLLaMA, Blockway here, a small team from Hong Kong. After our v1 post, the most common question was basically "ok, but why does this model exist?" Totally fair. Here's our honest answer. The problem Long agent sessions eat VRAM. On a normal dense model every layer keeps a KV cache, a