CH 03 · Chatbot Arena, Letting Real People Pick the Winner · 01/07
Read as one pageChatbot Arena, Letting Real People Pick the Winner
Chatbot Arena runs blind head-to-head battles between anonymous models and turns crowd votes into an Elo-style ranking of the answers people prefer.
7 slides · 4 min read
In this chapter
Paper: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024)
The first two benchmarks both score answers against something objective: a known answer in BIG-Bench, a passing test in SWE-bench. But some of the most important things about a chatbot have no objective answer at all. Which of two helpful, well-written replies is better? That depends on human taste and judgment. Chatbot Arena tackles this head-on by turning evaluation into a live, crowd-powered tournament where the only judges are real users.
Finished this idea?
Your learning progress stays in this browser.