Preference data

Preference data is the fuel of modern alignment. It consists of comparisons: for a given prompt, a human is shown two or more answers and indicates which is better. A single item is often a triple of (prompt, preferred answer, rejected answer).

Comparisons are used because humans are far better and more consistent at judging "which of these two is better" than at writing perfect answers from scratch or assigning absolute scores. Both RLHF and DPO run on this kind of data.