Skip to slide
20 / 39

CH 03 · Chatbot Arena, Letting Real People Pick the Winner · 01/07

Read as one page

Chatbot Arena, Letting Real People Pick the Winner

Chatbot Arena runs blind head-to-head battles between anonymous models and turns crowd votes into an Elo-style ranking of the answers people prefer.

7 slides · 4 min read

Contour map for Chatbot Arena, Letting Real People Pick the Winner

In this chapter

  1. 01The problem it solves
  2. 02The core idea
  3. 03Why you can trust it
  4. 04A short worked example
  5. 05Why it mattered
  6. 06The one-sentence takeaway

Paper: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024)

The first two benchmarks both score answers against something objective: a known answer in BIG-Bench, a passing test in SWE-bench. But some of the most important things about a chatbot have no objective answer at all. Which of two helpful, well-written replies is better? That depends on human taste and judgment. Chatbot Arena tackles this head-on by turning evaluation into a live, crowd-powered tournament where the only judges are real users.

Finished this idea?

Your learning progress stays in this browser.