What is it?
Human preference
Human preference is a way of scoring answers not against a correct solution but by asking people which answer they like better. It is the natural metric for open-ended tasks where quality is a matter of judgment rather than correctness. Its strength is that it captures what users actually value; its difficulty is that human opinions are noisy and can disagree, so you need many judgments and careful statistics to turn them into a reliable signal. Chatbot Arena is built entirely on human preference.
From the Benchmarks glossary
Where Human preference is explained
- Alignment, Turning a Text Predictor Into a Helpful AssistantFoundational Modelling · Putting the two papers together
- Judging Models, How Do We Measure Quality?Foundational Modelling · Tool 2: Chatbot Arena, let the crowd vote
- Chatbot Arena, Letting Real People Pick the WinnerBenchmarks
- Start Here: Benchmarks, Explained SimplyBenchmarks · The big picture