Skip to slide
13 / 74

CH 02 · Scaling Laws and Chinchilla, How Big Should a Model Be? · 01/06

Read as one page

Scaling Laws and Chinchilla, How Big Should a Model Be?

Scaling laws show model quality improves predictably with more parameters, data, and compute, and Chinchilla found trained models were starved of data.

6 slides · 4 min read

Contour map for Scaling Laws and Chinchilla, How Big Should a Model Be?

In this chapter

  1. 01Three knobs you can turn
  2. 02Paper 1: Scaling Laws, the discovery that bigger is predictably better
  3. 03Paper 2: Chinchilla, the correction
  4. 04Putting the two papers together
  5. 05The one-sentence takeaway

Papers: Scaling Laws for Neural Language Models (2020) and Training Compute-Optimal Large Language Models (Chinchilla, 2022)

Once you have the Transformer engine from Chapter 1, an obvious question appears: how big should you build it, and how much text should you feed it? These two papers answer that question with numbers instead of guesswork. Together they are the reason teams felt confident spending tens of millions of dollars to train ever-larger models.

Finished this idea?

Your learning progress stays in this browser.