CH 06 · Judging Models, How Do We Measure Quality? · 01/08
Read as one pageJudging Models, How Do We Measure Quality?
Judging open-ended chat quality is hard with no single right answer, and a strong model can substitute for a human judge about eighty percent of the time.
8 slides · 3 min read
In this chapter
Paper: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
We have built, scaled, aligned, and optimized our model. One question remains, and it is harder than it sounds: how do we know if it is any good? This final chapter of folder 01 is about measurement, which is the unglamorous but essential foundation of all progress. If you cannot measure quality, you cannot improve it.
Finished this idea?
Your learning progress stays in this browser.