openTPU is an open-source AI accelerator project that combines RTL, ISA, simulator, compiler, and profiler tools in one monorepo. According to its documentation, the project explores how far AI agents can advance in hardware design and whether they can create chips capable of running their own inference. The entire accelerator implementation resides in a repository containing SystemVerilog hardware design, a defined instruction set, a bit-exact simulator, a custom kernel language with its compiler, and host software for driving a physical PCIe FPGA card (Inspur YPCB-00338 with Xilinx Kintex-7 xc7k480t and two DDR3 channels).
This image represents the openTPU monorepo described in the article, which contains the hardware design, instruction set, simulator, compiler, and profiler tools.
The accelerator uses a simple execution model where a sequencer issues one instruction per cycle to units for DMA, matrix multiplication (int8 weights streamed from DRAM), vector math (fp32), and quantization, with no cache or hidden scheduling. Performance measurements show the FPGA card achieves token generation rates comparable to its simulator, with the hardware producing identical output token sequences bit-for-bit. For example, when running the LFM2.5-230M model in int8 precision, the card achieves 59.0 tokens per second on device and 52.3 tokens per second wall-clock time, while utilizing 85% of the available DRAM bandwidth. Validation against Hugging Face transformers using the project's quantization scheme shows top-1 agreement over 96% and low KL divergence for models like Qwen3-0.6B and LFM2.5-230M.
This image illustrates the real‑world performance of the openTPU accelerator running a quantized language model, matching the article’s discussion of token generation rates and simulator‑exact output.
This image shows the hardware floorplan and runtime behavior of a decode step, supporting the article’s mention of the bit‑exact simulator and Qwen3 execution.
Limitations and trade-offs exist: for mixture-of-experts models exceeding the card's 4 GiB DRAM (e.g., Qwen3.5-35B-A3B), experts are streamed from host storage over PCIe at rates up to 153 MB per token; 4-bit weight quantization reduces memory usage by about a third and increases decode speed by 40%-45% but incurs a measurable perplexity cost reported in the project's documentation; and certain operations like Qwen3.5 prefill still require host-side compilation of each chunk's program ahead of execution, indicating the host is not entirely out of the loop for all workloads.