Skip to slide
39 / 74

CH 06 · Judging Models, How Do We Measure Quality? · 01/08

Read as one page

Judging Models, How Do We Measure Quality?

Judging open-ended chat quality is hard with no single right answer, and a strong model can substitute for a human judge about eighty percent of the time.

8 slides · 3 min read

Contour map for Judging Models, How Do We Measure Quality?

In this chapter

  1. 01Why measuring chat models is genuinely hard
  2. 02Tool 1: MT-Bench, a hard conversation quiz
  3. 03Tool 2: Chatbot Arena, let the crowd vote
  4. 04The big idea: let an LLM be the judge
  5. 05Be careful: judges have biases
  6. 06How the three pieces fit together
  7. 07The one-sentence takeaway

Paper: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)

We have built, scaled, aligned, and optimized our model. One question remains, and it is harder than it sounds: how do we know if it is any good? This final chapter of folder 01 is about measurement, which is the unglamorous but essential foundation of all progress. If you cannot measure quality, you cannot improve it.

Finished this idea?

Your learning progress stays in this browser.