Skip to content
Agent YAP

What is it?

Mixture of Experts (MoE)

A Mixture of Experts is a model design (Chapter 5) where each layer contains many sub-networks called experts, but only a few of them activate for any given token. A small router picks which experts handle each word.

The benefit is that the model can hold a very large amount of knowledge across all its experts, while only doing a small amount of work per word. You get the quality of a big model at the running cost of a much smaller one. The trade-off is higher memory, because all experts must be kept loaded even though only a few are used at a time. Mixtral (Chapter 5) is the well-known open example.

From the Foundational Modelling glossary

Where Mixture of Experts is explained

  1. Start Here: Foundational Modelling, Explained SimplyFoundational Modelling · The big picture
  2. Mixtral and Mixture of Experts, More Brain, Same SpeedFoundational Modelling
  3. LoRA, Fine-Tuning Giant Models on a BudgetFoundational Modelling · The one-sentence takeaway
  4. BIG-Bench, Testing the Full Breadth of What a Model Can DoBenchmarks · What they found

Every term, A to Z