What is it?
Reward model
A reward model is a separate model trained to predict how much a human would like a given answer. It is built from preference data: humans rank several answers, and the reward model learns to reproduce those judgments, outputting a score for any answer it sees.
Its purpose is to bottle human judgment into automatic software, so that during reinforcement learning the main model can be graded on millions of its own attempts without needing a human in the loop each time.
From the Foundational Modelling glossary
As the Planning and Reasoning glossary puts it
A reward model is a model trained to score how good an output is, standing in for a human judge so that scoring can be done automatically and at scale. You met it first in folder 01 as part of the alignment pipeline. In folder 02, the process and outcome reward models are specialized reward models aimed at grading reasoning. (More background in the folder 01 glossary.)
Where Reward model is explained
- Alignment, Turning a Text Predictor Into a Helpful AssistantFoundational Modelling · Paper 1: InstructGPT and the RLHF recipe
- Let's Verify Step by Step, Rewarding Good ReasoningPlanning and Reasoning · Two kinds of graders
- DeepSeek-R1, Learning to Reason Through Reinforcement LearningPlanning and Reasoning · The setup: reward correct answers, nothing else