The core idea

Chatbot Arena is a free public website where the evaluation is a game. A user types a question and receives two replies from two anonymous models, side by side. The user reads both and votes for the better one. Only after voting are the two models' names revealed.

flowchart TD
    A[User asks a question] --> B[Two anonymous models<br/>each write a reply]
    B --> C[User reads both replies]
    C --> D[User votes for the better one]
    D --> E[Model names revealed<br/>after the vote]
    E --> F[Vote added to the data]
    F --> G[Update each model's rating]
    G --> H[Public leaderboard]

Hiding the names is the crucial trick. It makes the vote a blind, fair comparison, so a famous brand cannot win on reputation alone. Each vote is a single head-to-head result, like one match in a sports league: this model beat that model on this question. The paper reports the platform had already gathered over 240,000 such votes.

The remaining challenge is turning a pile of individual match results into a trustworthy ranking. You cannot make every model play every other model an equal number of times on equal questions, so the data is lopsided and noisy. Chatbot Arena handles this with established statistical tools. The familiar version of the idea is the Elo rating system borrowed from chess, where beating a strong opponent raises your score more than beating a weak one. The paper formalizes this with the Bradley-Terry model, a clean statistical method for estimating each model's underlying strength from many pairwise wins and losses, and it reports a confidence interval around each rating so readers can see how certain a ranking really is.

To get reliable rankings without wasting votes, the platform also chooses which pairs of models to show using active sampling: it preferentially matches up models whose relative strength is still uncertain, the same way a tournament organizer schedules the games that are most informative.