Alignment: RLHF, PPO, DPO, GRPO and Constitutional AI

Why imitation is not enough

SFT (8.2 Fine-tuning and PEFT) trains the model to reproduce demonstrations. But demonstrations are expensive, inconsistent, and bounded by what the demonstrator can write. Comparisons are easier than demonstrations: a person who could never write a great proof can still tell which of two attempts is better. Reinforcement learning from human feedback (RLHF) turns such judgments into a training signal. InstructGPT (2022) established the recipe behind ChatGPT: SFT → reward model → RL.

Step 1: a reward model from preferences

Collect prompts $x$ with two responses, a preferred $y_w$ and a rejected $y_l$. The Bradley–Terry model says the probability of preferring $y_w$ depends on the difference of latent scores:

Train a reward model $r_\theta$ (an LLM with a scalar head) by maximum likelihood on the comparisons:

Only differences matter: adding a constant to $r$ changes nothing, which is why reward values are not comparable across prompts. Reward models are themselves imperfect classifiers (agreement with humans is often in the 70–80% range), and that imperfection is the root of what follows.

Step 2: the RLHF objective

The policy $\pi_\theta$ is the LLM itself. We want high reward without drifting far from the SFT reference model $\pi_{\text{ref}}$:

The KL term (1.9 Information theory: entropy, cross-entropy and KL divergence) is the safety rail: a reward model is only trustworthy near the data it was trained on, so wandering far from $\pi_{\text{ref}}$ means exploiting its errors (reward hacking or over-optimization — Goodhart's law). $\beta$ trades off reward against conservatism. Implementations fold the KL into a per-token reward, $r_t = -\beta\log\frac{\pi_\theta(y_t\mid\cdot)}{\pi_{\text{ref}}(y_t\mid\cdot)}$, plus the sequence-level reward at the last token.

Optimization uses PPO (4.11 Reinforcement learning essentials): sample responses, score them, and update with the clipped surrogate

where the advantage $A_t$ comes from a learned value model (critic). That is four large networks in memory — policy, reference, reward, critic — and a notoriously fiddly training loop. Typical symptoms of over-optimization: verbose, flattering (sycophantic) answers that the reward model loves and humans do not.

DPO: skip the reward model

Direct Preference Optimization (2023) starts from a known fact: the KL-regularized objective above has a closed-form optimum,

with partition function $Z(x)$. Solve for the reward:

Substitute into the Bradley–Terry likelihood — the intractable $\log Z(x)$ cancels in the difference — and parametrize $\pi^*$ directly by the policy $\pi_\theta$:

Intuition. The policy is the reward model in disguise: $\beta\log(\pi_\theta/\pi_{\text{ref}})$ is an implicit reward. The gradient raises the log-probability of $y_w$ and lowers that of $y_l$, weighted by how badly the implicit reward currently misranks the pair. No sampling, no critic, no separate reward model — a plain supervised loss over offline comparisons, usually run with LoRA adapters (8.2 Fine-tuning and PEFT).

DPO's weakness is that it learns from a fixed dataset: if the comparisons were generated by another model, the policy never sees its own mistakes (the "off-policy" problem). Variants address different gaps:

Constitutional AI and RLAIF

Human preference labels are slow, expensive and expose annotators to harmful content. Constitutional AI (Anthropic, 2022) replaces much of that with a written set of principles. In a supervised phase, the model critiques and revises its own responses against the principles; in the RL phase, AI feedback produces the preference labels (RLAIF), and training proceeds as in RLHF. The approach scales oversight and makes the governing values explicit and inspectable. Most labs now combine human and AI feedback, plus rule-based rewards for specific behaviors.

GRPO and RL with verifiable rewards

For tasks with checkable answers — math with a reference solution, code with unit tests, formats that can be validated — a learned reward model is unnecessary: the environment verifies. Group Relative Policy Optimization (DeepSeekMath, 2024) makes the RL loop cheap: for each prompt, sample a group of $G$ responses, score them $r_1,\dots,r_G$, and use the group-normalized advantage

inside a PPO-style clipped objective with a KL penalty toward the reference model. The group mean replaces the critic as the baseline, removing one large network from memory. DeepSeek-R1-Zero (2025) showed that applying this to a base model with only answer-correctness and format rewards produces long chains of thought, self-verification and backtracking that nobody wrote into the data — the mechanism behind reasoning models (8.6 Reasoning models and test-time compute). Rubric-graded rewards extend the idea to less verifiable domains.

What alignment does — and does not — guarantee

Alignment training shapes behavior on the distribution it was trained on. It does not make the model understand or share human values, and it leaves known gaps: jailbreaks and prompt injection (8.12 Safety, risks and responsible AI, 10.10 Agent safety and security), sycophancy, reward hacking that generalizes in odd ways, and findings such as sleeper agents (backdoors that survive safety training) and alignment faking (models strategically complying during training). This is why evaluation (8.7 Evaluating LLMs), interpretability (8.11 Interpretability: what is happening inside) and system-level safeguards complement training rather than replace it.

Where this shows up later

• Reasoning models are GRPO-style RL on verifiable rewards scaled up — 8.6 Reasoning models and test-time compute.
• The same Bradley–Terry model ranks LLMs in human-preference arenas — 8.7 Evaluating LLMs.
• Alignment's limits motivate the safety and security lessons — 8.12 Safety, risks and responsible AI, 10.10 Agent safety and security; its mechanics reuse policy gradients from 4.11 Reinforcement learning essentials.