Smolmaxxing Enables 16B Parameter Models on 2013 iPad Air and TV Boxes

A 13-year-old iPad Air and a standard Xiaomi TV box can now run 16B parameter mixture-of-experts (MoE) models at speeds between 2.6 and 5.1 tokens per second. This is achieved through the 'smolmaxxing' project, which optimizes for the specific hardware bottlenecks of low-power devices—namely limited RAM, slow flash storage, and basic CPU instructions—to unlock capabilities that were previously thought to be inaccessible on such hardware.

This screenshot demonstrates the actual user interface for the model's output on the TV box hardware.

This chart visualizes the performance gains across different model sizes, specifically highlighting the 16B MoE model's performance on both devices.

Overcoming Compute and Memory Limits

To run models on devices like the Xiaomi TV Box S, which features an Amlogic S7D chip and only 1.9 GB of RAM, the project implements several compute-focused innovations. These include Q2_K and ternary repacks for vdotq_laneq instructions, 1-bit bit planes, and a specialized 1-bit lookup-table kernel that works on base NEON instructions without requiring dotprod support.

Memory constraints are addressed through 'headless' modes and pruned LM heads, which can free up 0.2 to 0.5 GB of RAM per model. These optimizations allow the system to maintain a larger pool of free memory, which is critical for keeping the model resident in RAM rather than constantly swapping to disk.

“Every device around you is more capable than it looks.”

— GitHub - hyperionaidev/smolmaxxing: maxxing the ability to run different LLMs on the smallest devices possible

Weight Streaming and Expert Paging

For models larger than the device's physical RAM, smolmaxxing utilizes a weight streaming engine with O_DIRECT and an 'expert pager.' This mechanism pages non-expert weights from flash memory on demand and prefetches layers ahead to minimize latency. This is particularly effective for Mixture-of-Experts (MoE) models, which only require a fraction of their total weights to be active at any given time.

On the 2013 iPad Air, this approach allowed a 16B MoE model to run at 2.6-4.5 tokens per second. The system uses cache-aware routing and a decayed-frequency eviction policy to ensure that the most frequently used experts remain accessible, reducing the impact of flash storage's 278 MB/s speed limit.

This scoreboard provides specific data points for the iPad Air's performance, including the 16B MoE model's speed.

Performance Trade-offs and Bottlenecks

The project demonstrates that for models larger than 2B parameters, the primary bottleneck shifts from CPU compute to flash storage speeds. A model's generation speed is strictly limited by the device's ability to stream weights from flash; for example, a 9B model running on the TV box achieves only 0.27 tokens per second because it is heavily flash-bound.

There is a measurable trade-off between speed and output quality. On the iPad Air, increasing the number of experts for the Moon-Beam-8B-A1B model can boost the generation speed to 5.0 tokens per second, but this results in a 27% increase in perplexity, meaning the model's responses become less coherent.

This graph illustrates the specific trade-off mentioned in the final paragraph regarding increasing active experts.