CH 02 · Scaling Laws and Chinchilla, How Big Should a Model Be? · 01/06
Read as one pageScaling Laws and Chinchilla, How Big Should a Model Be?
Scaling laws show model quality improves predictably with more parameters, data, and compute, and Chinchilla found trained models were starved of data.
6 slides · 4 min read
In this chapter
Papers: Scaling Laws for Neural Language Models (2020) and Training Compute-Optimal Large Language Models (Chinchilla, 2022)
Once you have the Transformer engine from Chapter 1, an obvious question appears: how big should you build it, and how much text should you feed it? These two papers answer that question with numbers instead of guesswork. Together they are the reason teams felt confident spending tens of millions of dollars to train ever-larger models.
Finished this idea?
Your learning progress stays in this browser.