Skip to slide
01 / 39

CH 00 · Start Here: Benchmarks, Explained Simply · 01/06

Read as one page

Start Here: Benchmarks, Explained Simply

Three papers measure a model's breadth of skill, usefulness on real software bugs, and which responses people prefer, from static tests to live voting.

6 slides · 3 min read

Contour map for Start Here: Benchmarks, Explained Simply

In this chapter

  1. 01How to use these explanations
  2. 02The big picture
  3. 03Which paper is covered where
  4. 04Chapter map
  5. 05One idea to hold in your head

In the earlier folders you learned how a language model is built and how it is taught to reason. But there is a question hanging over all of that work: how do we actually know whether a model is any good? If one model scores 80 and another scores 82, does that mean anything? If a model aces a test, has it really learned the skill, or just seen the answers before? This folder is about the surprisingly hard problem of measuring language models honestly.

A benchmark is just a shared test that everyone runs their model on, so that scores can be compared fairly. That sounds simple, but good benchmarks are genuinely difficult to build, and the three papers here each solve a different piece of the puzzle. Together they cover the three things you most want to know about a model: how broad its abilities are, how useful it is on real work, and which model people actually prefer to talk to.

Finished this idea?

Your learning progress stays in this browser.