Skip to content
Agent YAP

What is it?

Saturation

Saturation is what happens when a benchmark becomes too easy: nearly every model scores near the top, so the test can no longer tell strong models apart. A saturated benchmark has outlived its usefulness, like an exam where everyone gets full marks. The rapid saturation of older benchmarks was a major motivation for BIG-Bench, which deliberately included tasks believed to be beyond current models so that there would be plenty of room left to measure improvement.

From the Benchmarks glossary

Where Saturation is explained

  1. Scaling and InfrastructureArchitecture and System Design · Deployment and operational basics
  2. BIG-Bench, Testing the Full Breadth of What a Model Can DoBenchmarks · The problem it solves

Every term, A to Z