What is it?
Benchmark
A benchmark is a standardized test used to measure and compare model capabilities. Older benchmarks often used questions with a single correct answer, which made scoring easy but did not capture open-ended skills like helpfulness or writing quality.
Chapter 6 is about the harder problem of benchmarking open-ended chat ability, where there is no single right answer, and introduces tools like MT-Bench, Chatbot Arena, and using a strong model as an automatic judge.
From the Foundational Modelling glossary
As the Benchmarks glossary puts it
A benchmark is a shared, standardized test that many different models are run through so their results can be compared on equal footing. Think of it as a common exam: if everyone sits the same paper under the same rules, the scores mean something relative to each other. The deeper idea is explained in the folder 01 glossary; this whole folder is about what separates a good benchmark from a misleading one. The short version is that a benchmark is only useful if it is hard to fake, broad enough to be representative, and resistant to going stale.
Where Benchmark is explained
- BIG-Bench, Testing the Full Breadth of What a Model Can DoBenchmarks · The core idea
- Start Here: Benchmarks, Explained SimplyBenchmarks
- SWE-bench, Can a Model Fix a Real Bug in Real Code?Benchmarks · The problem it solves
- Chatbot Arena, Letting Real People Pick the WinnerBenchmarks · The problem it solves
- Practical Lessons from Building ManusContext Engineering · Verify with computable, binary checks: the Intern Test