Show HN：可在笔记本电脑上运行 Kimi K3 的新引擎
英文标题：🔶 Show HN: A new engine to run Kimi K3 on a laptop
开源项目 WASTE（Weight-Aware Streaming Tensor Engine）通过将高频使用的稠密权重保留在内存中、按需从 NVMe 存储流式加载 MoE 专家权重，使 2.78 万亿参数的 Kimi K3 能够在笔记本电脑上完整运行。在 64 GB MacBook Pro 上，982 GiB 的 Kimi K3 容器可实现约 0.32–0.34 tokens/s，4K 上下文下实测最低内存需求约为 29 GB。当前主要瓶颈是每个 token 需读取约 17 GB 专家数据，超过一半解码时间耗费在磁盘读取上。该项目提供 C 语言推理引擎、容器格式、转换工具、基准测试和验证套件，并兼容 OpenAI API 服务形式，对希望在本地运行超出 RAM 容量的大模型的开发者具有参考价值。
315 B
43 B
中文
Kimi K3 拥有 2.78 万亿参数，权重文件大小为 1.42 TB，显然无法装入笔记本电脑的内存。但 K3 是一个 Mixture-of-Experts（MoE，混合专家）模型：对于每个 token，每层 896 个专家中只有一小部分会被激活。这改变了问题的性质：只要每个 token 所需的权重能够足够快地被访问，整个模型就不必常驻于 RAM 中。
我们构建了 WASTE——Weight-Aware Streaming Tensor Engine（权重感知流式张量引擎）——以探索这一思路。
WASTE 将模型中稠密且被反复使用的部分保留在内存中，将路由专家存储于针对 NVMe 优化的容器内，并且只流式加载推理期间被选中的专家。其余 RAM 则被用作有界的专家缓存。
当前的 Kimi K3 容器大小为 982 GiB。在一台配备 64 GB 内存的 MacBook Pro 上，WASTE 可以以约每秒 0.32–0.34 个 token 的速度运行完整模型；在 4K 上下文下，实测最低内存需求约为 29 GB。
这显然还不是可交互的性能。但我们认为有趣的是，它居然能够运行：这是完整的开放权重模型，而不是蒸馏版、剪枝版，或仅使用 Kimi 名称的更小模型。
该引擎以 C 语言编写，推理路径中不依赖 BLAS、CUDA、ONNX 或 Python。同一套代码可通过 CLI 使用、作为库嵌入，或通过附带的 OpenAI-compatible server（兼容 OpenAI 的服务器）对外提供服务。
正确性是首要约束。每一层都针对 PyTorch 参考实现进行了验证，最终 logits 的匹配误差在 3.6e-06 以内。视觉塔同样得到支持，与其参考实现的匹配误差在 2.3e-06 以内。
当前瓶颈已经明确：K3 每个 token 大约需要 17 GB 的专家数据，超过一半的解码时间用于从磁盘读取专家权重。该引擎的运行吞吐量已经接近笔记本内部 SSD 的实测上限。因此，下一步改进需要在不使操作系统进入分页状态的前提下，减少每个 token 读取的字节数，并提高有效的专家复用率。
K3 是我们刻意选择的极端案例。同一引擎可在 8 GB 内存预算下，从一个 19 GB 容器运行 Kimi-Linear 48B，速度达到每秒 8.92 个 token。更广泛的目标是：让远大于可用 RAM 的模型能够在本地使用，无需将私有数据发送给 API，也无需专用的加速器硬件。
我们已经发布了引擎、容器格式、转换工具、基准测试、验证套件，以及那些失败的实验，而不是悄悄将它们删除。
所有内容均为完全开源。我们非常欢迎大家针对存储布局、量化、缓存策略、直接 I/O、可移植性及潜在优化提出反馈。我们也非常欢迎各种形式的贡献——代码、基准测试、不同硬件上的测试、文档、Bug 报告或新想法。
English / 原文
Kimi K3 has 2.78 trillion parameters and ships as 1.42 TB of weights. It clearly does not fit in the memory of a laptop. But K3 is a Mixture-of-Experts model. For each token, only a small fraction of its 896 experts per layer is activated. That changes the problem: the entire model does not need to be resident in RAM, as long as the weights required by each token can be reached quickly enough.
We built WASTE — the Weight-Aware Streaming Tensor Engine — to explore that idea.
WASTE keeps the dense, repeatedly used part of the model resident in memory, stores the routed experts in an NVMe-optimized container, and streams only the experts selected during inference. The remaining RAM is used as a bounded expert cache.
The current Kimi K3 container is 982 GiB. On a 64 GB MacBook Pro, WASTE runs the complete model at around 0.32–0.34 tokens per second, with a measured minimum memory requirement of approximately 29 GB at a 4K context.
That is obviously not interactive performance yet. But the result we found interesting is that it works at all: this is the full open-weights model, not a distillation, a pruned version, or a smaller model using the Kimi name.
The engine is written in C and has no BLAS, CUDA, ONNX, or Python dependency in the inference path. The same code can be used through the CLI, embedded as a library, or exposed through the included OpenAI-compatible server.
Correctness was the first constraint. Every layer was validated against a PyTorch reference, with final logits matching within 3.6e-06. The vision tower is supported as well and matches its reference within 2.3e-06.
The current bottleneck is understood: K3 needs roughly 17 GB of expert data per token, and more than half of the decode time is spent reading experts from disk. The engine is already operating close to the measured throughput limit of the laptop’s internal SSD. The next improvements therefore need to reduce the number of bytes read per token and increase useful expert reuse without pushing the operating system into paging.
K3 is deliberately the extreme case. The same engine runs Kimi-Linear 48B from a 19 GB container at 8.92 tokens per second with an 8 GB memory budget. The broader goal is to make models that are much larger than available RAM usable locally, without sending private data to an API and without requiring specialized accelerator hardware.
We have published the engine, container format, conversion tools, benchmarks, validation suite, and also the experiments that failed rather than quietly removing them.
Everything is fully open source. Feedback on the storage layout, quantization, caching strategy, direct I/O, portability, and potential optimizations would be very welcome. Contributions of any kind — code, benchmarks, testing on different hardware, documentation, bug reports, or new ideas — are more than appreciated.