CH 04 · Glossary: Benchmarks · 08/13
Read as one pageSaturation
Saturation is what happens when a benchmark becomes too easy: nearly every model scores near the top, so the test can no longer tell strong models apart. A saturated benchmark has outlived its usefulness, like an exam where everyone gets full marks. The rapid saturation of older benchmarks was a major motivation for BIG-Bench, which deliberately included tasks believed to be beyond current models so that there would be plenty of room left to measure improvement.
Finished this idea?
Your learning progress stays in this browser.