Skip to slide
Chapter 2 · Scaling Laws and Chinchilla, How Big Should a Model Be?
17 / 74

CHAPTER 02 · Scaling Laws and Chinchilla, How Big Should a Model Be? · 4 / 5

Putting the two papers together

These papers are not in conflict. The first one discovered that scaling works in a predictable way. The second one refined the recipe for how to scale, by balancing model size against data instead of just inflating the model.

QuestionScaling Laws (2020)Chinchilla (2022)
Does scaling help predictably?Yes, it follows a smooth power lawAgrees
Where to spend extra compute?Mostly on a bigger modelSplit evenly between model size and data
Practical ruleBigger is betterAbout 20 tokens of data per parameter
Lasting lessonForecast performance before trainingDo not starve your model of data
← → arrow keys work too