Skip to content
Agent YAP

What is it?

Human preference

Human preference is a way of scoring answers not against a correct solution but by asking people which answer they like better. It is the natural metric for open-ended tasks where quality is a matter of judgment rather than correctness. Its strength is that it captures what users actually value; its difficulty is that human opinions are noisy and can disagree, so you need many judgments and careful statistics to turn them into a reliable signal. Chatbot Arena is built entirely on human preference.

From the Benchmarks glossary

Where Human preference is explained

  1. Alignment, Turning a Text Predictor Into a Helpful AssistantFoundational Modelling · Putting the two papers together
  2. Judging Models, How Do We Measure Quality?Foundational Modelling · Tool 2: Chatbot Arena, let the crowd vote
  3. Chatbot Arena, Letting Real People Pick the WinnerBenchmarks
  4. Start Here: Benchmarks, Explained SimplyBenchmarks · The big picture

Every term, A to Z