CH 05 · Mixtral and Mixture of Experts, More Brain, Same Speed · 01/07
Read as one pageMixtral and Mixture of Experts, More Brain, Same Speed
Mixtral sends each word to two of eight expert subnetworks, so it holds the knowledge of a large model while running at the cost of a much smaller one.
7 slides · 3 min read
In this chapter
Paper: Mixtral of Experts (2024)
We end folder 01 with a clever architecture trick. Chapter 2 taught us that bigger models are smarter but more expensive to run. What if you could have the knowledge of a big model while paying the running cost of a small one? That is exactly what a Mixture of Experts, or MoE, delivers, and Mixtral is the open model that made the idea famous.
Finished this idea?
Your learning progress stays in this browser.