Skip to slide
32 / 74

CH 05 · Mixtral and Mixture of Experts, More Brain, Same Speed · 01/07

Read as one page

Mixtral and Mixture of Experts, More Brain, Same Speed

Mixtral sends each word to two of eight expert subnetworks, so it holds the knowledge of a large model while running at the cost of a much smaller one.

7 slides · 3 min read

Contour map for Mixtral and Mixture of Experts, More Brain, Same Speed

In this chapter

  1. 01The tension we are trying to escape
  2. 02The expert panel analogy
  3. 03How Mixtral is built
  4. 04Do the experts specialize by topic?
  5. 05What you give up
  6. 06The one-sentence takeaway

Paper: Mixtral of Experts (2024)

We end folder 01 with a clever architecture trick. Chapter 2 taught us that bigger models are smarter but more expensive to run. What if you could have the knowledge of a big model while paying the running cost of a small one? That is exactly what a Mixture of Experts, or MoE, delivers, and Mixtral is the open model that made the idea famous.

Finished this idea?

Your learning progress stays in this browser.