RLHF & PPO Mechanics: Reward Modeling, Advantage, and Trust Regions
How is a generative autoregressive transformer modified into a scalar reward model, and how is it trained on preference pairs?
How this lesson unfolds.
Before DPO, RLHF powered every milestone from InstructGPT to ChatGPT. In this chapter, we master the complete classical RLHF architecture.
How is a generative autoregressive transformer modified into a scalar reward model, and how is it trained on preference pairs?
Think of converting a novelist into a literary critic. A novelist uses their full vocabulary of 50,000 words to write the next line. A critic reads the finished chapter and raises a single number paddle from 1 to 10. We replace the 50,000-word vocabulary matrix with a single scalar output head at the end of the text.
- $$\mathcal{L}_{\text{RM}}(\psi) = -\ln\sigma(r_\psi(x, y_w) - r_\psi(x, y_l))$$
- $$A reward model replaces the causal language model vocabulary projection head with a single linear scalar projection $\mathbf{w}_{\text{rm}} \in \mathbb{R}^{d \times 1}$ evaluated at the terminal token, trained via pairwise binary ranking loss.$$
- Learning goalArchitect a scalar reward head on a causal LLM backbone and implement pairwise ranking loss training.
- Helpful foundationsForward logit extraction · Bradley-Terry preference model
- Common trapAveraging reward scores across all token positions; reward models pool representations exclusively from the final token containing full bidirectional context.
- Next connectionFeed these scalar rewards into the PPO clipped surrogate objective in Screen 2.
More context & mechanics
A reward model r_ψ(x, y) repurposes a pre-trained base transformer:
- Architecture Surgery: The final language modeling head W_(vocab) ∈ R^(d × V) is replaced by a scalar projection w_(rm) ∈ R^(d × 1).
- Terminal Token Pooling: The scalar score is extracted from the hidden state of the final token h_(eos):
r_ψ(x, y) = w_(rm)^T h_(eos) + b
- Pairwise Ranking Objective: Given preferred response y_w and dispreferred response y_l for prompt x:
L_(RM)(ψ) = -E_((x, y_w, y_l)) [ ln σ(r_ψ(x, y_w) - r_ψ(x, y_l)) ]
Crucially, rewards are invariant to scalar shifts (r → r + c); only the relative margin matters.
A reward model evaluates two responses to prompt x:
- Terminal hidden vector h_(w) = [1.2, -0.5], h_(l) = [0.2, 0.8].
- Reward head weights: w_(rm) = [2.0, -1.0]^T, bias b = 0.1.
- Computed Scalar Rewards:
r(x, y_w) = (1.2 × 2.0) + (-0.5 × -1.0) + 0.1 = 2.4 + 0.5 + 0.1 = 3.0
r(x, y_l) = (0.2 × 2.0) + (0.8 × -1.0) + 0.1 = 0.4 - 0.8 + 0.1 = -0.3
- Reward Difference & Ranking Loss:
Δ r = 3.0 - (-0.3) = 3.3
L_(RM) = -ln σ(3.3) = -ln((1) / (1 + e^(−3.3))) = -ln(0.9644) = 0.036
The model successfully scores the preferred completion significantly higher with minimal loss.
Standard reward architecture used in OpenAI InstructGPT, Anthropic Constitutional AI, and DeepMind Sparrow.
Christiano et al. (2017) introduced deep reinforcement learning from human preferences using pairwise trajectory comparisons.
Swap vocabulary for a scalar head, read the last token: rank winning over losing, score the thought.
Why do standard policy gradients suffer from destructive policy collapse, and how does PPO's clipped surrogate corridor tame updates?
Imagine training a prize racehorse. A standard policy gradient trainer uses an electric cattle prod: when the horse runs well, it gets an apple; when it stumbles, it gets a massive 10,000-volt shock. One oversized shock terrifies the horse so severely that it collapses in panic and refuses to ever run again (destructive policy collapse). PPO is a gentle trainer with a safety harness: it limits how much the horse's gait can change in any single session. If the horse tries to leap too violently, the harness gently catches it, ensuring consistent, injury-free daily improvement.
- Learning goalExplain the policy collapse defect in standard policy gradients and formulate PPO's clipped surrogate objective corridor.
- Helpful foundationsPolicy gradient theorem and advantage estimation · Proximal Policy Optimization Algorithms (Schulman et al., 2017)
- Common trapBelieving that clipping limits how much the model can learn over an entire training run. Clipping only limits the step size per mini-batch update; over thousands of steps, the policy can traverse vast distances safely.
- Next connectionNext, derive the dual-clipping calculus, piecewise gradient conditions, and numerical traces in Part B.
More context & mechanics
Reinforcement learning in high-dimensional language models faces the Policy Collapse Hazard:
- The Policy Gradient Catastrophe:
Standard policy gradient algorithms (REINFORCE) update parameters via:
g = E[ ∇_θ ln π_θ(a | s) Â ]
If an action happens to receive an outsized positive advantage Â, the gradient step can dramatically increase probability ratio r_t(θ). In a deep network, a large step in parameter space can cause a catastrophic shift in output distributions, turning a coherent model into a gibberish-spewing broken network in a single step.
- The Clipped Surrogate Solution (Schulman et al., 2017):
Proximal Policy Optimization defines the probability ratio:
r_t(θ) = (π_θ(a_t | s_t)) / (π_(old)(a_t | s_t))
Instead of maximizing r_t(θ) Â_t without limits, PPO clips the ratio inside an ε-corridor:
clip(r_t(θ), 1 - ε, 1 + ε)
(Typically ε = 0.2, restricting ratio changes to [0.8, 1.2]).
- Pessimistic Lower Bound:
By taking the minimum of the unclipped and clipped objectives, PPO ignores gradients that would push the policy too far from π_(old), providing a rock-solid mathematical safety corridor.
PPO was the foundational alignment engine used by OpenAI to train ChatGPT (InstructGPT) in 2022. It remains the core algorithm for complex multi-turn reasoning and agentic reinforcement learning where online exploration is required.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov published PPO at OpenAI in 2017. PPO simplified the complex second-order Hessian constraints of Trust Region Policy Optimization (TRPO) into a first-order clipped scalar objective.
- Clip the ratio, save the policy: Never let the probability ratio stray beyond 1 ± ε.
- Pessimistic bound: Always take the minimum between raw gain and clipped limit.
How is the PPO clipped surrogate objective mathematically formulated, and how do piecewise conditions eliminate destructive gradients?
Think of a safety governor on an engine. If the engine is spinning smoothly at 2,000 RPM, the throttle cable responds linearly to your foot on the pedal. But if the engine suddenly revs past the redline of 6,000 RPM, a centrifugal clutch disengages the throttle cable entirely: pushing down harder on the pedal does nothing. The PPO objective is that centrifugal clutch: as soon as the probability ratio exceeds 1 + ε, the gradient drops to zero.
- $$The PPO objective $L^{\text{CLIP}} = \mathbb{E}[\min(r_t \hat{A}_t, \, \text{clip}(r_t, 1-\epsilon, 1+\epsilon)\hat{A}_t)]$ sets gradients to zero when updates exceed trust boundaries.$$
- Learning goalDerive the mathematical proof of PPO's clipped surrogate objective and trace piecewise gradient behaviors across positive and negative advantage scenarios.
- Helpful foundationsPPO clipped surrogate concepts · Piecewise calculus and gradient vanishing conditions
- Common trapAssuming that clipping only applies to positive advantages. Clipping applies symmetrically to both positive and negative advantages, preventing catastrophic policy updates in either direction.
- Next connectionNext, discover how FlashAttention optimizes hardware memory hierarchies to eliminate attention bottlenecks in Chapter 46 Screen 5.
More context & mechanics
#### 1. Mathematical Formulation
Let probability ratio be r_t(θ) = (π_θ(a_t | s_t)) / (π_(old)(a_t | s_t)) and estimated advantage be Â_t.
The PPO Clipped Surrogate Objective is:
L^(CLIP)(θ) = Ê_t [ min( r_t(θ) Â_t, clip(r_t(θ), 1 - ε, 1 + ε) Â_t ) ]
Where ε ∈ (0, 1) (standard ε = 0.2).
The objective branches into two distinct operational regimes:
- Case 1: Positive Advantage (Â_t > 0, Good Action):
L^(CLIP) = min(r_t, 1 + ε) Â_t
- When r_t < 1 + ε: Objective is r_t Â_t. Gradient is ∇_θ r_t Â_t > 0, encouraging the action.
- When r_t ≥ 1 + ε: The objective caps at (1 + ε) Â_t. The derivative with respect to θ is identically zero!
The policy is forbidden from becoming overconfident on a single batch.
- Case 2: Negative Advantage (Â_t < 0, Bad Action):
L^(CLIP) = max(r_t, 1 - ε) Â_t
- When r_t > 1 - ε: Objective is r_t Â_t. Gradient actively drives probability down.
- When r_t ≤ 1 - ε: The objective caps at (1 - ε) Â_t. Gradient drops to zero!
The policy is prevented from excessively penalizing a bad action into complete collapse.
Let ε = 0.2 (Clipping bounds: [0.8, 1.2]).
- Scenario A (Positive Advantage  = +2.0):
- Suppose r_t = 1.15 (inside corridor):
L = min(1.15 × 2.0, 1.20 × 2.0) = min(2.30, 2.40) = 2.30 ⟹ Active gradient!
- Suppose r_t = 1.35 (outside corridor):
L = min(1.35 × 2.0, 1.20 × 2.0) = min(2.70, 2.40) = 2.40 ⟹ Gradient is ZERO!
- Scenario B (Negative Advantage  = -1.5):
- Suppose r_t = 0.70 (outside corridor):
L = min(0.70 × (-1.5), 0.80 × (-1.5)) = min(-1.05, -1.20) = -1.20 ⟹ Gradient is ZERO!
In RLHF implementations (e.g. DeepSpeed-Chat, Hugging Face TRL), the PPO loss also adds a value function loss L^(VF) and an entropy bonus S[π]:
L_(PPO)(θ) = Ê_t [ L_t^(CLIP)(θ) - c₁ L_t^(VF)(θ) + c₂ S[π_θ](s_t) ]
John Schulman introduced this clipping formula in 2017 to eliminate the expensive matrix inversions of TRPO's Fisher Information Matrix, establishing PPO as the default algorithm across robotics, gaming (OpenAI Five), and foundation model alignment.
- min(r Â, clip(r) Â): Protects against excessive greed on both positive and negative updates.
- Flat slopes mean zero gradients: Once you leave the corridor, the gradient disengages.
How does Generalized Advantage Estimation balance immediate reward variance against long-term critic estimation bias?
Imagine predicting your final semester grade. If you only look at yesterday's quiz (TD-0), your forecast is biased by one test. If you wait until graduation day (Monte Carlo), you get the true score but zero guidance during the semester. GAE is an exponential blender: blending near-term facts with long-term forecasts smoothly.
- $$\hat{A}_t^{\text{GAE}} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}^V, \quad \delta_t^V = r_t + \gamma V(s_{t+1}) - V(s_t)$$
- Learning goalCompute Generalized Advantage Estimates recursively and analyze the bias-variance tradeoff controlled by lambda.
- Helpful foundationsPPO Clipped Surrogate Objective · Value functions and Bellman equations
- Common trapAssuming the advantage is simply the raw scalar reward; raw rewards lack baseline normalization and cause unstable policy gradients.
- Next connectionExplore how critic baseline subtraction stabilizes variance in Screen 4.
More context & mechanics
Define the temporal difference residual at step t:
δ_t^V = r_t + γ V(s_(t+1)) - V(s_t)
Where V(s) is the value function estimated by the critic.
Generalized Advantage Estimation (GAE):
Â_t^(GAE(γ, λ)) = ∑_(l=0)^∞ (γ λ)^l δ_(t+l)^V = δ_t^V + (γ λ) Â_(t+1)^(GAE)
The λ Tradeoff Continuum:
- λ = 0 (TD-0 Advantage): Â_t = δ_t^V = r_t + γ V(s_(t+1)) - V(s_t).
- Properties: Lowest variance, but heavily biased by errors in the critic's value estimates.
- λ = 1 (Monte Carlo Advantage): Â_t = ∑_(l=0)^∞ γ^l r_(t+l) - V(s_t).
- Properties: Completely unbiased, but high variance due to trajectory stochasticity.
- λ ≈ 0.95 (Standard Sweet Spot): Achieves minimal mean-squared error by dampening noise while correcting critic bias.
Let discount γ = 1.0 (standard in finite-horizon LLM generation) and λ = 0.8.
Consider a 3-token generation with TD residuals: δ₀ = 0.5, δ₁ = -0.2, δ₂ = 1.0:
- Backward recursive computation:
- Token 2 (terminal): Â₂ = δ₂ = 1.0
- Token 1: Â₁ = δ₁ + (γ λ) Â₂ = -0.2 + (0.8 × 1.0) = -0.2 + 0.8 = 0.6
- Token 0: Â₀ = δ₀ + (γ λ) Â₁ = 0.5 + (0.8 × 0.6) = 0.5 + 0.48 = 0.98
Each token receives a smoothed, temporally consistent credit score reflecting its contribution to the final reward.
Standard advantage calculation engine across TRL, Ray RLlib, and cleanRL.
John Schulman et al. (2015) introduced GAE, resolving high variance in policy gradient estimation.
Lambda zero trusts the critic; lambda one trusts the roll; lambda 0.95 blends them both to reach the goal.
Why does subtracting a baseline value function dramatically reduce policy gradient variance without introducing bias?
Imagine grading exams where the class average was 90%. If a student scores 92%, they shouldn't get an astronomical celebration—they only beat expectations by +2%. If another scores 88%, they underperformed by -2%. Subtracting the baseline (expected class average) keeps rewards centered around zero so you only reward true outperformance.
- $$\mathbb{E}_{a \sim \pi}[\nabla_\theta \ln \pi_\theta(a \mid s) \cdot V(s)] = V(s) \nabla_\theta \sum_a \pi_\theta(a \mid s) = 0$$
- $$Subtracting state value baseline $V(s_t)$ from total return reduces gradient variance by centering updates around zero expectations; because baseline expectation $\mathbb{E}[\nabla \ln \pi \cdot V(s)] = 0$, the mathematical expectation remains strictly unbiased.$$
- Learning goalProve mathematically that subtracting a state-dependent baseline leaves policy gradient expectation unbiased while slashing variance.
- Helpful foundationsGeneralized Advantage Estimation · Probability conservation
- Common trapBelieving that subtracting a baseline alters the target optimal policy; because the gradient of a constant sum is zero, the optimal policy is invariant.
- Next connectionInvestigate failure modes of reward models such as length exploitation and Goodhart's law in Screen 5.
More context & mechanics
In the REINFORCE policy gradient:
g = E_(τ ~ π) [ ∑_(t=0)^T ∇_θ ln π_θ(a_t | s_t) · (R_t - b(s_t)) ]
Proof of Unbiasedness:
E_(a_t ~ π) [∇_θ ln π_θ(a_t | s_t) · b(s_t)] = b(s_t) ∑_(a_t) π_θ(a_t | s_t) (∇_θ π_θ(a_t | s_t)) / (π_θ(a_t | s_t))
= b(s_t) ∇_θ ∑_(a_t) π_θ(a_t | s_t) = b(s_t) ∇_θ (1) = 0
The baseline b(s_t) contributes identically zero to the gradient expectation!
Variance Reduction Impact:
By setting baseline b(s_t) = V_φ(s_t) (the state-value predicted by the critic network), the policy gradient variance is reduced by over 80\%, accelerating training convergence.
A batch has three prompt-response rollouts with returns R = [10.5, 10.0, 9.5]:
- Without Baseline (Raw Returns):
- All returns are large positive numbers (>9).
- Gradients push parameter updates in the same direction for all three rollouts, regardless of quality!
- Mean gradient magnitude is huge; variance is high.
- With Critic Baseline V(s) = 10.0:
- Centered Advantages:
- Rollout 1: 10.5 - 10.0 = +0.5 (Reinforce / Step Forward)
- Rollout 2: 10.0 - 10.0 = 0.0 (No change)
- Rollout 3: 9.5 - 10.0 = -0.5 (Suppress / Step Away)
- Result: Updates cleanly differentiate superior from inferior outputs with centered zero-mean dynamics.
Actor-Critic RL architectures in continuous control, game agents, and LLM alignment.
Ronald Williams established baseline subtraction in his landmark 1992 REINFORCE paper.
Baselines center the score around zero; expectations remain pure: subtract the average, amplify the edge.
Why do aligned models exploit reward model vulnerabilities, and how does Goodhart's law manifest as verbosity bias?
Imagine paying a programmer per line of code written. Within a week, the codebase will be filled with 50-line functions that do nothing but print blank spaces! When a reward model associates longer answers with thoroughness, the LLM stops giving concise answers and starts writing rambling filler essays to game the score.
- $$r_{\text{penalized}}(x, y) = r_\psi(x, y) - \alpha \cdot |y|$$
- Learning goalIdentify reward hacking mechanisms, diagnose Goodhart's law inflection points, and apply length-normalization penalties.
- Helpful foundationsReward Model Architecture · PPO Clipped Surrogate Objective
- Common trapBelieving higher proxy reward scores always indicate a better model; beyond the overoptimization threshold, quality deteriorates rapidly.
- Next connectionExplore modern reference-free and unpaired preference optimization methods in Chapter 44.
More context & mechanics
Goodhart's Law in RLHF:
As optimization proceeds against proxy reward model r_ψ:
max_θ E_(y ~ π_θ) [r_ψ(x, y)]
Proxy reward steadily climbs, but true human evaluation diverges and collapses after an inflection point!
Common Reward Hacking Vulnerabilities:
- Length Exploitation (Verbosity Bias): Human annotators historically rated longer answers as more helpful; reward models learn this correlation. The policy pads completions with filler phrases ('Furthermore, it is important to remember that...').
- Sycophancy: Fawning over user misconceptions or agreeing with false premises to score higher.
- Punctuation & Formatting Exploitation: Stuffing bullet points and bold tags regardless of relevance.
Mitigations:
- Length-normalized rewards: r_(norm) = r - α · Length(y).
- Regularized online rejection sampling and ensemble reward models.
A model generates two answers to prompt x: 'What is 2+2?'
- Answer A (Concise): '4' (1 token) arrow Raw Reward: 1.8.
- Answer B (Hacked Verbose): 'The sum of two and two is a fundamental mathematical fact resulting in four. In many mathematical contexts...' (60 tokens) arrow Raw Reward: 3.2.
Without length penalty, the RL loop strongly reinforces Answer B!
- With Length Penalty (α = 0.03 per token):
- Score A: 1.8 - (0.03 × 1) = 1.77
- Score B: 3.2 - (0.03 × 60) = 3.2 - 1.80 = 1.40
Now Answer A correctly wins, suppressing verbosity exploitation.
Production alignment safety protocols in OpenAI, Anthropic, and Google DeepMind alignment pipelines.
Formulated by British economist Charles Goodhart in 1975, formalized in machine learning by Gao et al. (2022).
When a metric becomes a goal, the policy exploits the hole: penalize length, ensemble the judge.
How does Reinforcement Learning from Human Feedback (RLHF): The PPO Triad operate mathematically at scale?
An interconnected system diagram showing four model roles:
- Learning goalMaster the algebraic and geometric derivation of Reinforcement Learning from Human Feedback (RLHF): The PPO Triad.
- Helpful foundationsFoundations of Build Llm From Scratch · Coordinate Invariance & Geometric Projections · Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
- Common trapAssuming Reinforcement Learning from Human Feedback (RLHF): The PPO Triad is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
A reinforcement learning multi-model architecture diagram in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.
Top section: Title in navy serif: "RLHF with PPO: The Multi-Model Ecosystem". Subtitle: "The four models simultaneously coordinated in GPU memory during Proximal Policy Optimization."
Middle section: An interconnected system diagram showing four model roles:
- Actor / Policy (π_θ): The active LLM being trained to generate responses that maximize reward.
- Reference Model (π_(ref)): Frozen copy of initial SFT model. Enforces KL divergence penalty D_(KL)(π_θ || π_(ref)) to prevent policy collapse.
- Reward Model (r_ψ): Trained on human preference pairs to output scalar score r(x, y).
- Critic / Value Model (V_φ): Estimates expected future reward to compute generalized advantage estimation (GAE).
Show composite reward formula:
"R(x, y) = r_ψ(x, y) - β D_(KL)(π_θ(y|x) || π_(ref)(y|x))".
Bottom section: Engineering challenge: "Running 4 full LLMs simultaneously in VRAM requires distributed model sharding (DeepSpeed ZeRO / Megatron-LM)."
Clean multi-container layout, distinct model color codes, balanced square framing.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
"Reinforcement Learning from Human Feedback (RLHF): The PPO Triad: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."