Skip to slide
33 / 39

CH 04 · Glossary: Benchmarks · 07/13

Read as one page

Metric

A metric is the specific rule you use to turn a model's output into a number. Accuracy (the fraction of questions answered correctly) is the most common, but there are many others, and the choice of metric quietly shapes what a benchmark rewards. A poorly chosen metric can make a weak model look strong or vice versa, which is why the BIG-Bench authors warned that even automatic, programmatic metrics can be surprisingly subjective. Whenever you read a benchmark score, it is worth asking exactly what was measured.

Finished this idea?

Your learning progress stays in this browser.