Skip to content
Agent YAP

What is it?

Benchmark

A benchmark is a standardized test used to measure and compare model capabilities. Older benchmarks often used questions with a single correct answer, which made scoring easy but did not capture open-ended skills like helpfulness or writing quality.

Chapter 6 is about the harder problem of benchmarking open-ended chat ability, where there is no single right answer, and introduces tools like MT-Bench, Chatbot Arena, and using a strong model as an automatic judge.

From the Foundational Modelling glossary

As the Benchmarks glossary puts it

A benchmark is a shared, standardized test that many different models are run through so their results can be compared on equal footing. Think of it as a common exam: if everyone sits the same paper under the same rules, the scores mean something relative to each other. The deeper idea is explained in the folder 01 glossary; this whole folder is about what separates a good benchmark from a misleading one. The short version is that a benchmark is only useful if it is hard to fake, broad enough to be representative, and resistant to going stale.

See it in the Benchmarks glossary

Where Benchmark is explained

  1. BIG-Bench, Testing the Full Breadth of What a Model Can DoBenchmarks · The core idea
  2. Start Here: Benchmarks, Explained SimplyBenchmarks
  3. SWE-bench, Can a Model Fix a Real Bug in Real Code?Benchmarks · The problem it solves
  4. Chatbot Arena, Letting Real People Pick the WinnerBenchmarks · The problem it solves
  5. Practical Lessons from Building ManusContext Engineering · Verify with computable, binary checks: the Intern Test

Every term, A to Z