Skip to slide
38 / 74

CH 05 · Mixtral and Mixture of Experts, More Brain, Same Speed · 07/07

Read as one page

The one-sentence takeaway

A Mixture of Experts model stores the knowledge of a very large model but, for each word, a small router activates only a couple of expert sub-networks, so you get big-model quality at small-model running cost, paying for it with extra memory to keep all the experts on standby.

Next: Chapter 6, Judging models, where we ask how anyone can possibly measure whether one model is better than another.

Finished this idea?

Your learning progress stays in this browser.