What is it?
Mixture of Experts (MoE)
A Mixture of Experts is a model design (Chapter 5) where each layer contains many sub-networks called experts, but only a few of them activate for any given token. A small router picks which experts handle each word.
The benefit is that the model can hold a very large amount of knowledge across all its experts, while only doing a small amount of work per word. You get the quality of a big model at the running cost of a much smaller one. The trade-off is higher memory, because all experts must be kept loaded even though only a few are used at a time. Mixtral (Chapter 5) is the well-known open example.
From the Foundational Modelling glossary
Where Mixture of Experts is explained
- Start Here: Foundational Modelling, Explained SimplyFoundational Modelling · The big picture
- Mixtral and Mixture of Experts, More Brain, Same SpeedFoundational Modelling
- LoRA, Fine-Tuning Giant Models on a BudgetFoundational Modelling · The one-sentence takeaway
- BIG-Bench, Testing the Full Breadth of What a Model Can DoBenchmarks · What they found