Research Papers
Benchmarks
BIG-Bench, SWE-Bench and Chatbot Arena: how the headline evaluations are constructed, what they genuinely test, and where each one stops being evidence.
5 chapters39 slides25 minUpdated
Your route through the subject
0/39 slides completed · 0/5 chapters · 0 slides read
Start with: Start Here: Benchmarks, Explained Simply
The chapter map
Follow the route or choose a chapter.- 010/6 completeStart Here: Benchmarks, Explained Simply~3 minReadCode
- 020/7 completeBIG-Bench, Testing the Full Breadth of What a Model Can Do~4 minReadCode
- 030/6 completeSWE-bench, Can a Model Fix a Real Bug in Real Code?~3 minReadCode
- 040/7 completeChatbot Arena, Letting Real People Pick the Winner~4 minReadField checkPlateCode
- 050/13 completeGlossary: Benchmarks~11 minRead
Looking for the reading order? Follow this subject’s paper path to see what each paper builds on.
Chapters
Chapter 0: Start Here: Benchmarks, Explained Simply
Three papers measure a model's breadth of skill, usefulness on real software bugs, and which responses people prefer, from static tests to live voting.
3 min6 slides: Start Here: Benchmarks, Explained Simply0/6 complete
Chapter 1: BIG-Bench, Testing the Full Breadth of What a Model Can Do
More than 450 researchers built 204 deliberately hard tasks to map what language models can and cannot do, each one scored against a human baseline.
4 min7 slides: BIG-Bench, Testing the Full Breadth of What a Model Can Do0/7 complete
Chapter 2: SWE-bench, Can a Model Fix a Real Bug in Real Code?
SWE-bench hands a model a real GitHub bug and a full codebase, then grades the fix by running the test suite; early top models solved under two percent.
3 min6 slides: SWE-bench, Can a Model Fix a Real Bug in Real Code?0/6 complete
Chapter 3: Chatbot Arena, Letting Real People Pick the Winner
Chatbot Arena runs blind head-to-head battles between anonymous models and turns crowd votes into an Elo-style ranking of the answers people prefer.
4 min7 slides: Chatbot Arena, Letting Real People Pick the Winner0/7 complete
Chapter 4: Glossary: Benchmarks
Ground truth, contamination, pairwise comparison, Elo rating, the Bradley-Terry model, and confidence intervals are defined for reading model leaderboards.
11 min13 slides: Glossary: Benchmarks0/13 complete