Saturation

Saturation is what happens when a benchmark becomes too easy: nearly every model scores near the top, so the test can no longer tell strong models apart. A saturated benchmark has outlived its usefulness, like an exam where everyone gets full marks. The rapid saturation of older benchmarks was a major motivation for BIG-Bench, which deliberately included tasks believed to be beyond current models so that there would be plenty of room left to measure improvement.