Skip to slide
27 / 39

CH 04 · Glossary: Benchmarks · 01/13

Read as one page

Glossary: Benchmarks

Ground truth, contamination, pairwise comparison, Elo rating, the Bradley-Terry model, and confidence intervals are defined for reading model leaderboards.

13 slides · 11 min read

Contour map for Glossary: Benchmarks

In this chapter

  1. 01Benchmark
  2. 02Static vs live benchmark
  3. 03Ground truth
  4. 04Human preference
  5. 05Task and task suite
  6. 06Metric
  7. 07Saturation
  8. 08Contamination
  9. 09Canary string
  10. 10Scoring and scaling behavior
  11. 11Measuring real coding
  12. 12Ranking by human votes

This is your shared reference for folder 04. Every term the chapters link to is explained here from scratch, in plain language, with enough depth to actually understand it rather than just recognize it. Read it straight through as a primer, or jump in whenever a chapter sends you here.

A few foundational terms live in the earlier folders and are linked across rather than repeated: what a benchmark is at heart, the Elo rating system, emergent ability, sparse and dense models, long context, and agent. This glossary focuses on the ideas specific to evaluation.

Terms are grouped by theme so related ideas sit together.


Finished this idea?

Your learning progress stays in this browser.