Mixture of experts (MoE)
A mixture-of-experts model contains many specialist sub-networks and uses only a few of them for each token, so it can be very large yet fast.
In a dense model every parameter is used for every token. An MoE model splits part of the network into 'experts', and a small router picks a few of them for each token. DeepSeek V4.1 Flash, for example, has about 552 billion parameters in total but uses 16 billion for each token.
The catch is memory. All the experts must be loaded even though only some of them run, so an MoE model needs memory for its total size, while its speed is closer to that of a much smaller dense model. That is why this site lists MoE models with both a total and an active parameter count.
Related terms
- Dense model A dense model uses all of its parameters to process every token, unlike a mixture-of-experts model, which activates only a part of them.
- Parameters Parameters are the numbers inside a model that are adjusted during training; their count, such as 7B or 70B, is the usual measure of a model's size.
- VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.