Mixture of Experts (MoE)
Sparse architecture activating only a fraction of weights per token
Architecture
In plain English
A council of 8 specialized doctors. When a patient arrives, only the 2 most relevant doctors give advice, saving time and energy.
Technically
A model architecture where feed-forward network (FFN) layers are split into multiple independent expert subnetworks. A learned router dynamically activates top-k experts per token (e.g. 2 of 8 in Mixtral 8x7B, or 8 of 256 in DeepSeek-R1), decoupling total parameter capacity from per-token compute cost.