What is it?
LLM-as-a-judge
Using one language model to grade the output of another, against a rubric (criteria like accuracy, completeness, and citation quality), often producing a score and a pass or fail label. It is a scalable way to evaluate agents when there is no single exact correct answer, though it is not perfectly reliable and is usually paired with human review.
From the AI agent glossary
Where LLM-as-a-judge is explained
- Judging Models, How Do We Measure Quality?Foundational Modelling
- Practical Lessons from Building ManusContext Engineering · Verify with computable, binary checks: the Intern Test
- How Memory Is Built and RetrievedAgentic Memory · The agent loop and context tools: the Claude Agent SDK
- The Case For Multi-Agent: Anthropic's Research SystemMulti-Agent Systems · Evaluating a system that has no single right path
- Why Multi-Agent Systems Fail: The MAST TaxonomyMulti-Agent Systems · The contributions, in plain terms