Baidu has released ERNIE 5.5 Flash Base, the raw pre-training checkpoint of its new sparse Mixture-of-Experts language model. ERNIE 5.5 Flash Base is a 280B-parameter text model from Baidu that activates 14B parameters per token. The weights stop at the end of pre-training, with no mid- or post-training applied. This as a clean starting point for teams that want to compare fine-tuning, instruction tuning or RL methods from one checkpoint.
TL;DR
- Size: 280B total, 14B active per token; 128K training context, 1M for fine-tuning and inference.
- Runs on: Multi-GPU servers. Baidu’s reference configs use 4-way parallelism in SGLang and vLLM. GPU model and VRAM not disclosed. FP8 and BF16 weights.
- Performance: Best score on 24 of 30 base-model benchmarks in Baidu’s unified pipeline (self-reported).
- Best: MMLU-Pro 78.4, ahead of DeepSeek-V4-Flash-Base at 69.2.
- Worst: HELMET-Rerank 56.2, behind DeepSeek-V4-Flash-Base at 60.0.
What is ERNIE 5.5 Flash Base?
It is a text-only MoE base model with 43 layers and a 4096 hidden size. It was pre-trained from scratch on 33.5T tokens with the Muon optimizer. The tokenizer has 201,216 entries and groups numbers right-to-left in chunks of up to 3 digits.
Activated compute, KV cache and long-context cost were treated as one budget. Pre-training data leaned on repository-scale code and long-context material, since those are hard to add later.
How does the ERNIE 5.5 Flash architecture work?
- Hybrid attention: 37 layers use Heavily Compressed Attention (HCA), which caches 1 entry per 128 tokens. The other 6 layers use Multi-head Latent Attention with sparse attention (MLA + DSA). That means 1 in 7 layers keeps full-context access. HCA layers use 512-dim heads; MLA layers use 256-dim heads.
- Gated and mixed heads: The attention output gate comes from the low-rank query latent, cutting gate-projection parameters by 75%. Virtual Head Attention adds a lightweight low-rank mix across heads.
- Multi-Router Latent MoE. Each MoE layer has 512 routed experts (top-10) plus 1 shared expert. Several independent sigmoid routers replace a single router, per Baidu’s Multi-Router paper. Expert FLOPs and communication volume stay unchanged. Experts run in a 2048-d latent space, half the hidden width.
- ECHO instead of MTP.:ECHO reuses the backbone’s final layer to predict future tokens. The same layer doubles as a built-in draft model for speculative decoding. ECHO adds 0.36B parameters, excluded from the headline counts.
- Other pieces: The model uses Manifold-constrained Hyper-Connections with expansion rate 4 and a MultiMax output head.
