What is it?
Metric
A metric is the specific rule you use to turn a model's output into a number. Accuracy (the fraction of questions answered correctly) is the most common, but there are many others, and the choice of metric quietly shapes what a benchmark rewards. A poorly chosen metric can make a weak model look strong or vice versa, which is why the BIG-Bench authors warned that even automatic, programmatic metrics can be surprisingly subjective. Whenever you read a benchmark score, it is worth asking exactly what was measured.
From the Benchmarks glossary
Where Metric is explained
- Observability and EvaluationArchitecture and System Design · From observability to evaluation
- Domain-Specific and Regulated-Industry AgentsArchitecture and System Design · Evaluation with domain rigor
- Practical Lessons from Building ManusContext Engineering · Verify with computable, binary checks: the Intern Test
- Practical Guidance: When and How to Use Agentic RAGAgentic RAG · Lesson 5: Evaluation must account for process, not just outcomes
- What Is an Agent Harness?Building Coding Agents and Harnesses · Real code: Codex splits the harness into crates