Edge0-145B Enables 145-Billion Parameter AI on Personal Laptops

Edge0-145B is the world's first open-source on-device AI model with 145 billion parameters, capable of running on a compatible personal laptop with a reported peak memory of 29GB. By moving high-parameter reasoning and coding capabilities from the cloud to local hardware, the model eliminates recurring API fees and data-handling constraints while maintaining performance that rivals frontier cloud models.

This diagram illustrates the technical architecture and performance metrics of the 145B parameter model.

Overcoming the Memory Wall

Historically, on-device AI models have been limited to smaller scales, such as the 1.8B to 3.25B parameter models used in mobile foundations. This was due to a 'memory wall' where large models required more memory and bandwidth than personal hardware could provide, forcing a tradeoff between model capability and local execution.

Edge0-145B addresses this by using a latent MoE architecture. In this design, groups of 64 experts share two large matrices, while each expert retains a small set of parameters to capture its specific differences. This sharing substantially reduces the weights that must be moved during inference while maintaining a high expert count.

“Edge0-145B is the world's first open-source on-device AI model with 145 billion parameters.”

— Edge0-145B: Open-Source On-Device AI with 145B Parameters

Multi-Tiered Inference Engine

To handle the 145B parameters, the Edge0 framework utilizes a four-layer design that pipelines GPU memory, system memory, and SSD access. Frequently used 'hot experts' remain in GPU memory, while an expert library of approximately 40GB sits in system memory, and 48GB of vocabulary data resides on the SSD to be read on demand.

The system also employs parallel scheduling, using the CPU and GPU together. The GPU computes cached experts while the CPU handles remaining work, and speculative decoding is used to verify multiple candidate tokens at once, spreading the cost of reading weights across more output tokens.

Performance and Practical Applications

In internal testing, Edge0-145B demonstrated a prefill speed of over 1,100 tokens per second and a decode rate of 80+ tokens per second. These speeds enable complex agentic workflows, such as generating 3D scenes in Blender or producing video scripts, where the model can plan, write, debug, and visually inspect its own work.

The speed table provides specific data points for the model's performance in different scenarios.

While the model eliminates cloud-model token fees and keeps data local, it still requires specific hardware. The reported results were achieved on a setup with a 29GB peak memory footprint and a typical power consumption of approximately 120W, meaning users must possess hardware capable of sustaining these demands without throttling.

This table provides the specific hardware requirements and performance metrics mentioned in the final paragraph.

Chart drawn by HN Digest from figures reported by edge0.ai; values are shown as written in the source.