Skip to content
Agent YAP

What is it?

Elo rating

Elo is a rating system originally designed to rank chess players, repurposed in Chapter 6 to rank chat models. Each model has a numeric rating. When two models compete and humans vote for the winner, the winner's rating goes up and the loser's goes down. Crucially, beating a strong opponent raises your rating more than beating a weak one, and losing to a weak opponent costs you more.

In Chatbot Arena, thousands of blind head-to-head human votes feed an Elo leaderboard, producing a ranking grounded directly in human preference. It is considered one of the most trustworthy ways to compare models, precisely because it is built on real human choices.

From the Foundational Modelling glossary

As the Benchmarks glossary puts it

Elo is a rating system, originally from chess, that estimates each player's strength from the outcomes of head-to-head matches. The full mechanics are explained in the folder 01 glossary. The key intuition for this folder is that beating a strong opponent raises your rating more than beating a weak one, and losing to a weak opponent costs you more. Chatbot Arena uses Elo-style ratings as an intuitive way to rank models from pairwise votes.

See it in the Benchmarks glossary

Where Elo rating is explained

  1. Judging Models, How Do We Measure Quality?Foundational Modelling · Tool 2: Chatbot Arena, let the crowd vote
  2. Start Here: Benchmarks, Explained SimplyBenchmarks · How to use these explanations
  3. Chatbot Arena, Letting Real People Pick the WinnerBenchmarks · The core idea

Every term, A to Z