Canary string

A canary string is a unique, distinctive identifier embedded in every document of a benchmark so that the benchmark can be detected later. The idea, introduced by BIG-Bench, is twofold. Model builders can search their training data for the string and remove the benchmark before training, preventing contamination. And researchers can test whether a model has secretly memorized the benchmark by checking whether it reproduces the string. The name comes from the canary in a coal mine: a simple early-warning device.

---