Skip to content
Agent YAP

What is it?

PPO

PPO, short for Proximal Policy Optimization, is the specific reinforcement learning algorithm used in the RLHF recipe. You do not need its math, only its role: it is the method that adjusts the model to earn higher scores from the reward model, while taking care not to change the model too drastically in any one step. The "proximal" in the name refers to that caution, staying close to the previous version to keep training stable.

From the Foundational Modelling glossary

Where PPO is explained

  1. Deep Research Agents and How They Are TrainedBuilding Coding Agents and Harnesses · How they are trained
  2. Alignment, Turning a Text Predictor Into a Helpful AssistantFoundational Modelling · Paper 1: InstructGPT and the RLHF recipe

Every term, A to Z