Apple Machine Learning Research has demonstrated the capability to run frontier-class large language models (LLMs) on Mac Studio hardware using the MLX framework. MLX is an open-source array framework designed to leverage the unified memory architecture of Apple silicon, allowing developers to perform inference and fine-tuning locally. The research highlights the ability to run Muse Spark Open, a mixture-of-experts (MoE) model with nearly a trillion parameters, by utilizing a cluster of Mac Studios. In their testing, a four-machine cluster achieved over 36 tokens per second at a context size of 8,192 tokens, which exceeds average human reading speeds.
MLX is an open-source array framework designed to leverage the unified memory architecture of Apple silicon, allowing developers to perform inference and fine-tuning locally.
The performance of these models is heavily influenced by the hardware's memory architecture and specialized accelerators. For instance, the M5 Ultra chip features a GPU Neural Accelerator that supports dedicated matrix multiplication, which Apple reports can speed up 'time to first token' (TTFT) by up to 4.6x compared to the previous M3 Ultra generation. While smaller models like Qwen 3.8 (27B) and Mistral Medium 3.5 (128B) fit on a single Mac Studio with 512GB of unified memory, the Muse Spark Open model requires more than 1.6TB of RAM in its native precision. Even when quantized to 4-bit (NVFP4) precision, the model's memory footprint reaches approximately 493GB, necessitating parallelization across multiple machines to accommodate the model and the intermediate computations required during inference.
To handle these massive models, MLX supports two primary parallelization schemes: pipeline and tensor parallelization. Pipeline parallelization splits the model's layers across different machines, where one Mac Studio executes the first half of the layers and passes the output to the next. Tensor parallelization distributes the model's width, where each machine holds a fraction of every layer. To make tensor parallelization viable, MLX utilizes a low-latency RDMA (Remote Direct Memory Access) driver over Thunderbolt 5. This allows for the high-speed inter-machine communication necessary to aggregate outputs across the cluster. While pipeline parallelization offers simplicity, tensor parallelization can provide significant speedups for dense architectures as more machines are added to the cluster, provided the communication overhead remains low.",
To make tensor parallelization viable, MLX utilizes a low-latency RDMA (Remote Direct Memory Access) driver over Thunderbolt 5.