🔶 OpenTPU：由 AI 开发的开源 AI 加速器

一个由 AI 开发的开源 AI 加速器。

openTPU 将 auto-arch-tournament 的经验带入 AI 加速器领域。它提出两个问题：AI Agent 在硬件设计上能走多远，以及它们能否构建运行自身推理任务的芯片？

左侧为 otpu-chat 在 FPGA 卡上运行 LFM2.5-230M，右侧为 otpu-smi 显示该卡的利用率和 DRAM 带宽。

openTPU 同时也是一个学习项目。整个加速器都位于一个小型 monorepo 中，可以从头读到尾：其中包括硬件设计（SystemVerilog）、指令集、位精确模拟器、内核语言及其编译器，以及驱动真实 PCIe 卡的主机软件。如果你希望理解 AI 加速器的工作原理——从 Python 中的矩阵乘法一直到底层连线——这是一个很好的起点。

该设计可在一张 Inspur YPCB-00338 卡（Xilinx Kintex-7 xc7k480t、双 DDR3 通道）上使用真实权重运行 10 个现代模型，且该卡生成的 token 与模拟器逐位完全一致。

在卡上测得：前三个模型于 2026-09-29 使用生产镜像 deploy_champ_e698dcd7 测试。LFM2-2.6B、SmolLM3-3B 和 Phi-4-mini 于 2026-09-30 测试；Qwen3.5-2B、4B 和 Gemma 4 于 2026-10-01 使用构建 B、deploy_fused133c_79c5707a 测试，该版本此后成为生产版本。与 e698dcd7 相比，构建 B 对 LFM2-2.6B、SmolLM3 和 Phi-4-mini 的解码速度提升 8—9%（Gemma 4 E2B 提升 10%）；DRAM 利用率从 82—87% 提高至峰值的 91—94%。Qwen3.5-4B 的 int8 镜像超过 4 GiB。

• 镜像：主版本 e698dcd，运行于 133.33 MHz，所有模型共用一个 bitstream。它包含由内存核心内部小型 CPU 校准的 LiteDRAM 控制器、一个四列脉动矩阵单元和流引擎（docs/stream.md）。采用 DDR3-1066，理论峰值带宽为 17.1 GB/s。
• 主机：该卡安装于 opentpu（Intel Core i7-4790）中。
• 方法，tools/qual/perf.py：解码是在一个 512-token 提示词之后贪心生成 64 个 token，循环中由主机执行 argmax（未采用流式处理）。“设备端”仅统计加速器运行的周期；“端到端”还计入主机时间。预填充为设备端处理该 512-token 提示词。
• DRAM 流量来自卡运行期间自身的计数器。
• Gemma 4 E2B 将其逐层 embedding 表保留在卡上（镜像为 3.5—3.6 GiB；docs/gemma4.md）；在 int8 模式下无法容纳。无论使用哪种 head，它都能在三个提示词上匹配 Hugging Face 的贪心 token。E4B 的表格（2.95 GB）保留于主机端，主机每个 token 向卡复制一行 11 KB 数据；其镜像大小为 3.96 GiB，head 及前 24 层的 down projection 使用 4-bit，其余为 int8（docs/gemma4_e4b.md）。在卡自身的解码循环中，即卡自行选择每一个 token 时，Gemma 4 解码更快：E2B 在设备端达到 11.01 / 12.73 tok/s（int8 / 4-bit head），E4B 为 3.83 tok/s。
• 每种配置均与模拟器逐 token、逐位置，以及使用驻留式解码程序的结果完全一致。更多细节见 docs/board.md。

在卡运行期间以流式方式回传 logits（tools/decode_profile.py，96 个 token）时，4-bit 解码速度更快；以下为设备端 / 端到端 tok/s：

• LFM2：89.5 / 84.5；
• Qwen3：33.7 / 33.3；
• Qwen3.5：24.6 / 24.2；
• LFM2-2.6B：11.07 / 11.02（构建 B）；
• SmolLM3-3B：8.92 / 8.89（构建 B）；
• Phi-4-mini：6.69 / 6.67（构建 B）。

此前的生产镜像 se-cand3 使用 Xilinx MIG、双列矩阵单元和 120.755 MHz 时钟。以相同方式测量，新镜像具有以下表现：

• **解码：**所有配置下都处于 se-cand3 性能的 2.3% 以内。解码受 DRAM 限制，而 LiteDRAM 的读取带宽达到 DDR3 峰值的 82—85%，与 MIG 相当。
• **预填充：**速度提升 1.3 倍（Qwen3.5）至 2.0 倍（LFM2 4-bit）。
• **校准：**镜像启动时，核心内的 CPU 会在 12 秒内校准两条 DDR3 通道，无需主机参与。

早期镜像及其测试数据见 docs/board.md 第 5 节。

大于该卡 4 GiB 容量的混合专家模型，可通过从主机存储流式传输专家来运行（docs/offload.md 第 10 节）。该卡为每个 token 执行路由并计算每个专家，同时将专家保存在其 DRAM 中按层划分的槽位内。主机仅以连接带宽，将池文件中缺失的专家复制到这些槽位（第 10.1 节）。以下为 2026-10-01 使用构建 B（79c5707a）测得的数据，采用卡自身的解码循环来选择每个 token、4-bit 专家和 int8 head：

• LFM2.5-8B-A1B（85 亿参数，17 亿活跃参数）：在 160 个 token 上达到 10.6 tok/s。98.5% 的专家调用命中槽位，每个 token 流式传输 5.2 MB。
• Qwen3.5-35B-A3B（347 亿参数，30 亿活跃参数）：使用 Hugging Face 的 16 个贪心 token，达到 3.95 tok/s。62% 的专家调用命中槽位，每个 token 流式传输 153 MB，经 PCIe 的传输速率为 1.41 GB/s（第 10.3 节）。
• 两者均与模拟器逐位完全一致。

4-bit 权重（docs/quant.md）采用 FP4 数值与双级块缩放，每个权重占 4.25 bit，并为保证精度而将 LM head 保持为 int8。它们将每个 token 的字节量减少约三分之一，并将解码速度提升 40%（Qwen3.5）至 45%（Qwen3、LFM2），但会带来可测量的困惑度损失，docs/quant.md 报告了各模型的具体情况。

主机几乎不再构成开销。对于 LFM2 和 Qwen3，该卡运行一个仅编译一次的解码程序：程序从寄存器读取位置，并查询自身的 embedding 和 RoPE 行；在卡仍在运行时，logits 即以流式方式回传。在 omarchy 上，主机每个 token 仅增加 0.17 至 0.30 ms；在 opentpu 上则增加 0.45 至 1.3 ms。Qwen3.5 的解码也以相同方式运行；其预填充仍会在主机上、卡运行之前，为每个数据块编译程序。

Kernels in ol              mlp, attention, full model layers
      |  @ol.jit
Language + compiler        layouts, affine loop addressing, fusion
      |
ISA                        8 x 32-bit words per instruction
      |
ISA simulator  <======>  RTL          same bits, checked by the tests
(Python)                 (SystemVerilog)
                          |  Vivado bitstream
                         FPGA card    Kintex-7 xc7k480t
                          |  PCIe
                         Host         otpu-chat, otpu-smi, otpu-lens

这台机器刻意保持简单。一个 sequencer 每个周期向少量单元发出一条指令：DMA 负责移动数据；矩阵单元对从 DRAM 流式读取的 int8 权重进行乘法；向量单元执行 fp32 数学计算；量化器将结果重新转换为 int8。系统没有缓存，也没有隐藏调度：每一次数据移动都是一条指令，因此一份 trace 能够精确展示周期消耗的位置。docs/isa.md 描述了完整的指令集。

一个内核如下所示：

from opentpu import language as ol

@ol.jit
def mlp(h, gamma, w_gate, w_up, w_down, out, eps):   # simplified; see kernels/mlp.py
    x = ol.load(h)
    xs = ol.quantize(rmsnorm(x, ol.load(gamma), eps))
    g = ol.dot(xs, w_gate)
    u = ol.dot(xs, w_up)
    a = ol.all_gather(silu(g) * u)
    y = ol.all_gather(ol.dot(a, w_down))
    if ol.program_id() == 0:
        ol.store(out, x + y)

由于每一次数据移动都是一条指令，一次运行的 trace 能够解释其速度。性能分析器 Lens 可从 RTL、模拟器或实体卡记录运行过程，并在浏览器中打开，提供 roofline 图、时间线和逐指令表格（docs/lens.md）。

Lens 正在回放 Qwen3 一次解码步骤的一部分。颜色显示每个单元在每个周期中的状态：忙碌、等待 DRAM，或等待另一条指令。

除实体卡外，所有组件都可以在笔记本电脑上运行。

pip install -e .
pip install pytest torch transformers
python3 -m pytest -q          # RTL tests also need Verilator 5

hf download LiquidAI/LFM2.5-230M --local-dir models/LFM2.5-230M
otpu-chat --model lfm2 --backend isa    # chat on the simulator

如有卡，可在 boards/ypcb-00338 中构建 bitstream（make bit），通过 JTAG 加载它，然后运行 sudo otpu-setup 和 otpu-chat --backend board。docs/board.md 介绍了完整的启动流程。

1. docs/isa.md：指令集，其他所有组件均构建于其上。
2. opentpu/kernels 和 docs/compiler.md：内核如何变成指令。
3. opentpu/isasim.py：作为规范的模拟器。
4. rtl/：硬件实现，从 rtl/top/otpu_top.sv 开始。
5. docs/lfm2.md、docs/qwen35.md、docs/llama.md、docs/benchmarks.md：完整模型及其周期消耗的位置。
6. docs/board.md：从时钟到 PCIe 的实体卡说明。

• **DRAM 的最后几个百分点。**解码受 DRAM 效率限制：其读取带宽为 DDR3-1066 峰值的 82% 至 85%。目前正持续提升 LiteDRAM 数据路径的效率。
• **时序裕量和面积。**该设计刚好能够在 133.33 MHz 完成时序收敛——这是 128 字节端口与两条 DDR3 通道相匹配的时钟频率——但裕量极小（WNS +0.032 ns）。一场由 Vivado 运行组成的 tournament 正持续优化其裕量和面积。解码受 DRAM 限制，因此更高时钟主要有助于预填充。
• **更快的预填充。**四列脉动矩阵单元已进入生产镜像；预填充仍受矩阵单元乘法速率的限制。

欢迎提交 issue 和 pull request，而且大多数工作只需要 Python 和 Verilator，不需要 FPGA。对 ISA、模拟器或 RTL 的修改必须确保 python3 -m pytest -q 仍能通过；性能声明应说明测试方法。

Apache License 2.0。参见 LICENSE。

An open-source AI accelerator, developed by AI.

openTPU brings the lessons of auto-arch-tournament to AI accelerators. It asks two questions: how far can AI agents go at hardware design, and can they build the chip that runs their own inference?

otpu-chat running LFM2.5-230M on the FPGA card (left), with otpu-smi showing the card's utilization and DRAM bandwidth (right).

openTPU is also a learning project. The whole accelerator lives in one small monorepo that you can read end to end: the hardware design (SystemVerilog), the instruction set, a bit-exact simulator, a kernel language and its compiler, and the host software that drives a real PCIe card. If you want to understand how an AI accelerator works, from a matmul in Python down to the wires, this is a good place to start.

The design runs ten modern models with their real weights on an Inspur YPCB-00338 card (Xilinx Kintex-7 xc7k480t, two DDR3 channels), and the card produces the same tokens as the simulator, bit for bit.

Measured on the card: the first three models on 2026-09-29 with the production imagedeploy_champ_e698dcd7. LFM2-2.6B, SmolLM3-3B and Phi-4-mini on 2026-09-30, and Qwen3.5-2B and 4B and Gemma 4 on 2026-10-01, with build B, deploy_fused133c_79c5707a, production since then. Build B decodes LFM2-2.6B, SmolLM3 and Phi-4-mini 8-9% faster than e698dcd7 (Gemma 4 E2B 10%), at 91-94% of the DRAM peak instead of 82-87%. Qwen3.5-4B's int8 image is over 4 GiB.

• The image: main e698dcd at 133.33 MHz, one bitstream for all models. It has LiteDRAM controllers calibrated by a small CPU inside the memory core, a four-column systolic matrix unit and the stream engine (docs/stream.md). DDR3-1066, with a 17.1 GB/s peak.
• The host: the card sits in opentpu (Intel Core i7-4790).
• Method,tools/qual/perf.py: decode is 64 greedy tokens after a 512-token prompt, with the host's argmax in the loop (not streamed). "Device" counts only the cycles the accelerator runs; "wall" adds the host. Prefill is the 512-token prompt, on the device.
• DRAM traffic comes from the card's own counters while it runs.
• Gemma 4 E2B keeps its per-layer embedding tables on the card (3.5-3.6 GiB images;docs/gemma4.md); in int8 it does not fit. It matches Hugging Face's greedy tokens on three prompts with either head. E4B's table (2.95 GB) stays on the host, which copies one 11 KB row into the card per token; its image is 3.96 GiB, int8 with the head and the first 24 layers' down projections in 4-bit (docs/gemma4_e4b.md). In the card's own decode loop (the card picking every token) Gemma 4 decodes faster: E2B 11.01 / 12.73 tok/s (int8 / 4-bit head), E4B 3.83 tok/s, on the device.
• Every configuration matches the simulator token for token, per-position and with the resident decode program. More detail indocs/board.md.

With the logits streamed back while the card runs (tools/decode_profile.py, 96 tokens), 4-bit decode is faster, in device / wall tok/s:

• LFM2: 89.5 / 84.5;
• Qwen3: 33.7 / 33.3;
• Qwen3.5: 24.6 / 24.2;
• LFM2-2.6B: 11.07 / 11.02 (build B);
• SmolLM3-3B: 8.92 / 8.89 (build B);
• Phi-4-mini: 6.69 / 6.67 (build B).

The previous production image, se-cand3, was built with the Xilinx MIG, a two-column matrix unit and a 120.755 MHz clock. Measured the same way, the new image:

• decode: within 2.3% of se-cand3's in every configuration. Decode is bound by DRAM, and LiteDRAM reads at 82-85% of the DDR3 peak, as the MIG did.
• prefill: 1.3x (Qwen3.5) to 2.0x (LFM2 4-bit) faster.
• calibration: when the image starts, the core's CPU calibrates both DDR3 channels in 12 s, with no host involvement.

The earlier images and their numbers are in docs/board.md, section 5.

Mixture-of-experts models bigger than the card's 4 GiB run with their experts streamed from host storage (docs/offload.md, section 10). The card routes each token and computes every expert, and it keeps the experts in per-layer slots in its DRAM. The host only copies missing experts from a pool file into those slots, at the link's rate (section 10.1). Measured on 2026-10-01 with build B (79c5707a), the card's own decode loop picking every token, 4-bit experts, int8 head:

• LFM2.5-8B-A1B (8.5B parameters, 1.7B active): 10.6 tok/s over 160 tokens. 98.5% of expert uses hit the slots, and 5.2 MB streamed per token.
• Qwen3.5-35B-A3B (34.7B parameters, 3.0B active): 3.95 tok/s, with Hugging Face's 16 greedy tokens. 62% of expert uses hit, and 153 MB streamed per token at 1.41 GB/s over PCIe (section 10.3).
• Both match the simulator bit for bit.

4-bit weights (docs/quant.md) use FP4 values with two-level block scales, 4.25 bits per weight, and keep the LM head in int8 for accuracy. They cut the bytes per token by about a third and raise decode speed by 40% (Qwen3.5) to 45% (Qwen3, LFM2), at a measurable cost in perplexity that docs/quant.md reports per model.

The host is nearly out of the way. For LFM2 and Qwen3 the card runs one decode program compiled once, which reads the position from a register and looks up its own embedding and RoPE rows, and the logits stream back while the card is still running: the host adds 0.17 to 0.30 ms per token on omarchy (0.45 to 1.3 ms on opentpu). Qwen3.5 runs the same way for decode; its prefill still compiles each chunk's program on the host, ahead of the card.

  Kernels in ol              mlp, attention, full model layers
        |  @ol.jit
  Language + compiler        layouts, affine loop addressing, fusion
        |
  ISA                        8 x 32-bit words per instruction
        |
  ISA simulator  <======>  RTL          same bits, checked by the tests
  (Python)                 (SystemVerilog)
                            |  Vivado bitstream
                           FPGA card    Kintex-7 xc7k480t
                            |  PCIe
                           Host         otpu-chat, otpu-smi, otpu-lens

The machine is deliberately simple. A sequencer issues one instruction per cycle to a few units: DMA moves data, the matrix unit multiplies int8 weights streamed from DRAM, the vector unit does fp32 math, and a quantizer turns results back into int8. There is no cache and no hidden scheduling: every data movement is an instruction, so a trace shows exactly where the cycles go. docs/isa.md describes the whole instruction set.

A kernel looks like this:

from opentpu import language as ol

@ol.jit
def mlp(h, gamma, w_gate, w_up, w_down, out, eps):   # simplified; see kernels/mlp.py
    x = ol.load(h)
    xs = ol.quantize(rmsnorm(x, ol.load(gamma), eps))
    g = ol.dot(xs, w_gate)
    u = ol.dot(xs, w_up)
    a = ol.all_gather(silu(g) * u)
    y = ol.all_gather(ol.dot(a, w_down))
    if ol.program_id() == 0:
        ol.store(out, x + y)

Because every data movement is an instruction, a trace of a run explains its speed. Lens, the profiler, records a run from the RTL, the simulator or the card and opens it in the browser, with a roofline, a timeline and per-instruction tables (docs/lens.md).

Lens replaying part of a Qwen3 decode step. Colours show what each unit is doing in each cycle: busy, waiting on DRAM, or waiting on another instruction.

Everything except the card runs on a laptop.

pip install -e .
pip install pytest torch transformers
python3 -m pytest -q          # RTL tests also need Verilator 5

hf download LiquidAI/LFM2.5-230M --local-dir models/LFM2.5-230M
otpu-chat --model lfm2 --backend isa    # chat on the simulator

With a card, build the bitstream (make bit in boards/ypcb-00338), load it over JTAG, then run sudo otpu-setup and otpu-chat --backend board. docs/board.md walks through the bring-up.

1. docs/isa.md: the instruction set. Everything else is built on it.
2. opentpu/kernels and docs/compiler.md: how a kernel becomes instructions.
3. opentpu/isasim.py: the simulator, which is the spec.
4. rtl/: the hardware, starting from rtl/top/otpu_top.sv.
5. docs/lfm2.md, docs/qwen35.md, docs/llama.md, docs/benchmarks.md: whole models and where their cycles go.
6. docs/board.md: the physical card, from clocks to PCIe.

• The last few percent of DRAM. Decode is bound by DRAM efficiency: it reads 82 to 85% of the DDR3-1066 peak. Work on the LiteDRAM path's efficiency is under way.
• Timing margin and area. The design closes 133.33 MHz, the clock at which the 128-byte port matches the two DDR3 channels, but only just (WNS +0.032 ns). A tournament of Vivado runs keeps working on its margin and area. Decode is bound by DRAM, so a faster clock mostly helps prefill.
• Faster prefill. The four-column systolic matrix unit is in the production image; prefill is still limited by the matrix unit's multiply rate.

Issues and pull requests are welcome, and most of the work needs only Python and Verilator, not an FPGA. Changes to the ISA, the simulator or the RTL must keep python3 -m pytest -q passing, and performance claims should say how they were measured.

Apache License 2.0. See LICENSE.