Skip to slide
22 / 39

CH 03 · Chatbot Arena, Letting Real People Pick the Winner · 03/07

Read as one page

The core idea

Chatbot Arena is a free public website where the evaluation is a game. A user types a question and receives two replies from two anonymous models, side by side. The user reads both and votes for the better one. Only after voting are the two models' names revealed.

flowchart TD
    A[User asks a question] --> B[Two anonymous models<br/>each write a reply]
    B --> C[User reads both replies]
    C --> D[User votes for the better one]
    D --> E[Model names revealed<br/>after the vote]
    E --> F[Vote added to the data]
    F --> G[Update each model's rating]
    G --> H[Public leaderboard]

Hiding the names is the crucial trick. It makes the vote a blind, fair comparison, so a famous brand cannot win on reputation alone. Each vote is a single head-to-head result, like one match in a sports league: this model beat that model on this question. The paper reports the platform had already gathered over 240,000 such votes.

The remaining challenge is turning a pile of individual match results into a trustworthy ranking. You cannot make every model play every other model an equal number of times on equal questions, so the data is lopsided and noisy. Chatbot Arena handles this with established statistical tools. The familiar version of the idea is the Elo rating system borrowed from chess, where beating a strong opponent raises your score more than beating a weak one. The paper formalizes this with the Bradley-Terry model, a clean statistical method for estimating each model's underlying strength from many pairwise wins and losses, and it reports a confidence interval around each rating so readers can see how certain a ranking really is.

To get reliable rankings without wasting votes, the platform also chooses which pairs of models to show using active sampling: it preferentially matches up models whose relative strength is still uncertain, the same way a tournament organizer schedules the games that are most informative.

Ratings from pairwise votes
Ratings from pairwise votes

Field check

See it worked. Then try it.

Tie a benchmark claim to the behavior it actually measures.

Worked examples

In a blind comparison, a reader prefers one explanation’s style.

This is evidence about preference on the sampled question, not an executable proof that every factual claim is true.

A proposed patch passes the repository’s specified tests.

This is evidence about task correctness under those tests, not about which prose a reader likes best.

Your turnA model wins more sampled blind Arena comparisons. What does that support most directly?

Finished this idea?

Your learning progress stays in this browser.