Skip to slide
07 / 39

CH 01 · BIG-Bench, Testing the Full Breadth of What a Model Can Do · 01/07

Read as one page

BIG-Bench, Testing the Full Breadth of What a Model Can Do

More than 450 researchers built 204 deliberately hard tasks to map what language models can and cannot do, each one scored against a human baseline.

7 slides · 4 min read

Contour map for BIG-Bench, Testing the Full Breadth of What a Model Can Do

In this chapter

  1. 01The problem it solves
  2. 02The core idea
  3. 03What they found
  4. 04A short worked example
  5. 05Why it mattered
  6. 06The one-sentence takeaway

Paper: Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models (2022), known as BIG-Bench

This paper is the result of an unusual experiment in scale, not of the model but of the test. More than 450 researchers across 132 institutions came together to build a single giant benchmark of 204 tasks, deliberately chosen to be hard for the language models of the day. The goal was simple to state and hard to do: build a test broad and difficult enough to actually map the edges of what these models can and cannot do.

Finished this idea?

Your learning progress stays in this browser.