Baidu has released ERNIE 5.5 Flash Base, a language model designed as a high-efficiency starting point for developers. Unlike a finished AI assistant, a 'base' model is a checkpoint captured at the end of the pre-training phase, before any mid- or post-training steps like instruction tuning or reinforcement learning. By releasing the weights, Baidu provides a foundation that allows developers to focus their own resources on specific domain adaptation, reasoning tasks, and agentic applications rather than undertaking the prohibitive cost of large-scale pre-training from scratch.
The model utilizes a sparse Mixture-of-Experts (MoE) architecture, featuring 280B total parameters with only 14B activated per token. To balance capability with inference efficiency, the architecture incorporates several specialized components. These include a hybrid sparse attention system—combining Multi-head Latent Attention (MLA) and Heavily Compressed Attention (HCA)—and a Multi-Router Latent MoE system. The Multi-Router design uses several independently initialized sigmoid routers to score experts jointly, which Baidu claims improves routing accuracy and stability compared to conventional single-router systems. Additionally, the model uses a new BPE tokenizer that handles numbers right-to-left to improve performance on arithmetic tasks.
This diagram illustrates the specific architectural components mentioned, such as the hybrid sparse attention and Multi-Router Latent MoE system.
ERNIE 5.5 Flash Base was pre-trained on 33.5T tokens with a training context of up to 128K tokens, while supporting a 1M-token context for fine-tuning and inference. The training process utilized a custom distributed Muon optimizer and a focus on high-quality data, including repository-scale code and long-context documents. While the model offers a clean research baseline and high efficiency, it is important to note that as a base model, it is not yet fine-tuned to follow specific instructions. Users intending to use it for conversational interfaces or specific task-following behaviors will need to perform further training to achieve those capabilities. The primary trade-off is that while the model provides a powerful starting point, it requires additional development effort to transition from a raw pre-trained checkpoint to a production-ready application.
This chart provides a visual comparison of the model's performance relative to other base models across multiple benchmarks.