What is it?
RLHF
RLHF stands for Reinforcement Learning from Human Feedback. It is the three-step recipe (Chapter 3) that turns a raw pre-trained model into a helpful assistant: first supervised fine-tuning on human-written answers, then training a reward model from human rankings, then using reinforcement learning (specifically PPO) to optimize the model against that reward.
RLHF was the breakthrough that made ChatGPT possible. Its central trick is learning human taste from comparisons rather than trying to write down rules for good behavior.
From the Foundational Modelling glossary
Where RLHF is explained
- Alignment, Turning a Text Predictor Into a Helpful AssistantFoundational Modelling · Paper 2: DPO, the same goal with far less machinery
- Start Here: Foundational Modelling, Explained SimplyFoundational Modelling · The big picture
- Scaling Laws and Chinchilla, How Big Should a Model Be?Foundational Modelling · The one-sentence takeaway