Skip to content
Benchmarks chapter route0001020304

Research Papers

Benchmarks

BIG-Bench, SWE-Bench and Chatbot Arena: how the headline evaluations are constructed, what they genuinely test, and where each one stops being evidence.

5 chapters39 slides25 minUpdated

Your route through the subject

0/39 slides completed · 0/5 chapters · 0 slides read

Start with: Start Here: Benchmarks, Explained Simply

Begin reading

The chapter map

Follow the route or choose a chapter.
  1. 010/6 completeStart Here: Benchmarks, Explained Simply~3 minReadCode
  2. 020/7 completeBIG-Bench, Testing the Full Breadth of What a Model Can Do~4 minReadCode
  3. 030/6 completeSWE-bench, Can a Model Fix a Real Bug in Real Code?~3 minReadCode
  4. 040/7 completeChatbot Arena, Letting Real People Pick the Winner~4 minReadField checkPlateCode
  5. 050/13 completeGlossary: Benchmarks~11 minRead

Looking for the reading order? Follow this subject’s paper path to see what each paper builds on.

Chapters

  1. Chapter 0: Start Here: Benchmarks, Explained Simply

    Three papers measure a model's breadth of skill, usefulness on real software bugs, and which responses people prefer, from static tests to live voting.

    3 min6 slides: Start Here: Benchmarks, Explained Simply0/6 complete

  2. Chapter 1: BIG-Bench, Testing the Full Breadth of What a Model Can Do

    More than 450 researchers built 204 deliberately hard tasks to map what language models can and cannot do, each one scored against a human baseline.

    4 min7 slides: BIG-Bench, Testing the Full Breadth of What a Model Can Do0/7 complete

  3. Chapter 2: SWE-bench, Can a Model Fix a Real Bug in Real Code?

    SWE-bench hands a model a real GitHub bug and a full codebase, then grades the fix by running the test suite; early top models solved under two percent.

    3 min6 slides: SWE-bench, Can a Model Fix a Real Bug in Real Code?0/6 complete

  4. Chapter 3: Chatbot Arena, Letting Real People Pick the Winner

    Chatbot Arena runs blind head-to-head battles between anonymous models and turns crowd votes into an Elo-style ranking of the answers people prefer.

    4 min7 slides: Chatbot Arena, Letting Real People Pick the Winner0/7 complete

  5. Chapter 4: Glossary: Benchmarks

    Ground truth, contamination, pairwise comparison, Elo rating, the Bradley-Terry model, and confidence intervals are defined for reading model leaderboards.

    11 min13 slides: Glossary: Benchmarks0/13 complete