CH 00 · Start Here: Benchmarks, Explained Simply · 01/06
Read as one pageStart Here: Benchmarks, Explained Simply
Three papers measure a model's breadth of skill, usefulness on real software bugs, and which responses people prefer, from static tests to live voting.
6 slides · 3 min read
In this chapter
In the earlier folders you learned how a language model is built and how it is taught to reason. But there is a question hanging over all of that work: how do we actually know whether a model is any good? If one model scores 80 and another scores 82, does that mean anything? If a model aces a test, has it really learned the skill, or just seen the answers before? This folder is about the surprisingly hard problem of measuring language models honestly.
A benchmark is just a shared test that everyone runs their model on, so that scores can be compared fairly. That sounds simple, but good benchmarks are genuinely difficult to build, and the three papers here each solve a different piece of the puzzle. Together they cover the three things you most want to know about a model: how broad its abilities are, how useful it is on real work, and which model people actually prefer to talk to.
Finished this idea?
Your learning progress stays in this browser.