Modern Preference Optimization: KTO, ORPO, IPO, and GRPO
How does KTO align models using unpaired binary feedback (thumbs up / thumbs down) inspired by behavioral economics?
How this lesson unfolds.
Alignment research moves at breakneck speed. In this chapter, we master the modern alignment suite that powers today's top open-weights reasoning models.
How does KTO align models using unpaired binary feedback (thumbs up / thumbs down) inspired by behavioral economics?
In real products, users rarely write two answers and pick the better one. Instead, they give a single 'thumbs up' or 'thumbs down'. Behavioral economics teaches that humans feel the pain of a bad experience twice as intensely as the pleasure of a good experience. KTO embeds this exact human psychology into the loss function.
- $$\mathcal{L}_{\text{KTO}} = \mathbb{E}_{(x, y)} [w(y) \cdot (1 - \sigma(\lambda (z - z_0)))]$$
- $$Kahneman-Tversky Optimization (KTO) eliminates the requirement for paired preference data by applying Kahneman-Tversky Prospect Theory: modeling human utility with loss aversion where losses hurt more than equal-sized gains reward.$$
- Learning goalExplain the behavioral economics foundations of KTO and implement unpaired preference alignment with loss aversion.
- Helpful foundationsDirect Preference Optimization (DPO) · Binary logistic loss
- Common trapAssuming preference tuning strictly requires pairs of winning and losing answers; KTO aligns models using individual thumbs-up/down ratings.
- Next connectionEliminate the need for a separate reference model in memory via ORPO in Screen 2.
More context & mechanics
While DPO requires matched pairs (x, y_w, y_l), real-world telemetry produces singletons (x, y, label ∈ +1, -1).
KTO Formulation:
Inspired by Kahneman & Tversky's Prospect Theory (1979), human perception of utility is concave for gains and convex/steeper for losses:
v(z) = 1 - σ(λ_(gain) (z - z₀)) & if y is desirable \ 1 - σ(λ_(loss) (z₀ - z)) & if y is undesirable
Where z(x, y) = β ln (π_θ(y | x)) / (π_(ref)(y | x)) and z₀ = E[z] is the reference anchor point.
Loss Aversion Hyperparameter:
Setting λ_(loss) > λ_(gain) (typically λ_(loss) ≈ 1.33 λ_(gain)) penalizes bad outputs more aggressively than it rewards good ones, preventing toxic outputs without requiring paired data.
Let reference anchor point z₀ = 0.0.
Consider a desirable completion y₁ and an undesirable completion y₂:
- Desirable Output (y₁): Model increases log-ratio to z = +1.0 (λ_(gain) = 1.0).
- Utility loss: 1 - σ(1.0 × (1.0 - 0.0)) = 1 - σ(1.0) = 1 - 0.731 = 0.269.
- Undesirable Output (y₂): Model slips and produces z = +0.5 (λ_(loss) = 1.5, loss averse!).
- Utility loss: 1 - σ(1.5 × (0.0 - 0.5)) = 1 - σ(-0.75) = 1 - 0.321 = 0.679.
Notice that the penalty for the undesirable output (0.679) is 2.5× larger due to loss aversion weighting, forcefully stamping out bad habits.
Direct alignment on user interaction telemetry (thumbs up/down, upvotes, accepts) in production chat interfaces.
Ethayarajh et al. (Contextual AI & Stanford, 2024) developed KTO, translating Nobel-prize winning Prospect Theory into LLM alignment.
No pairs needed, just thumbs up and down: losses hurt twice as much, KTO aligns the town.
How does Odds Ratio Preference Optimization eliminate the memory overhead of maintaining a separate reference model during training?
DPO requires keeping two full copies of the model in GPU memory: the active student and the frozen reference teacher. ORPO says: why carry two models? By comparing the odds of generating the good answer versus the bad answer within the student model itself, you cut GPU memory consumption in half!
- $$\mathcal{L}_{\text{ORPO}} = \mathcal{L}_{\text{SFT}} - \lambda \ln\sigma\left(\ln\frac{\text{odds}_\theta(y_w)}{\text{odds}_\theta(y_l)}\right)$$
- Learning goalFormulate the ORPO objective function and explain how odds ratios eliminate the reference model from GPU memory.
- Helpful foundationsDirect Preference Optimization · Odds and probability relationships
- Common trapBelieving reference models are mathematically mandatory for alignment; odds ratios penalize undesirable tokens within the active model alone.
- Next connectionAddress the unbounded margin drift of DPO using Identity Preference Optimization (IPO) in Screen 3.
More context & mechanics
In DPO, computing ln (π_θ) / (π_(ref)) requires evaluating both π_θ and frozen π_(ref), increasing VRAM by ≈ 50-80\%.
ORPO Formulation:
ORPO defines the odds of generation under the current policy alone:
odds_θ(y | x) = (P_θ(y | x)) / (1 - P_θ(y | x))
The odds ratio between winning response y_w and losing response y_l is:
OR_θ(y_w, y_l) = (odds_θ(y_w | x)) / (odds_θ(y_l | x))
The Unified ORPO Objective:
L_(ORPO) = L_(SFT)(y_w) + λ_(OR) · L_(OR)
L_(OR) = -ln σ( ln OR_θ(y_w, y_l) )
Where L_(SFT) trains fluency on winning answers, while L_(OR) directly penalizes the probability of dispreferred tokens.
A model generates token probabilities on winning sequence P(y_w) = 0.80 and losing sequence P(y_l) = 0.20:
- Calculate Individual Odds:
odds(y_w) = (0.80) / (1 - 0.80) = (0.80) / (0.20) = 4.0
odds(y_l) = (0.20) / (1 - 0.20) = (0.20) / (0.80) = 0.25
- Odds Ratio & Log Odds Ratio:
OR = (4.0) / (0.25) = 16.0 ⟹ ln(OR) = ln(16.0) ≈ 2.772
- Odds Penalty Loss:
L_(OR) = -ln σ(2.772) = -ln((1) / (1 + e^(−2.772))) = -ln(0.941) = 0.060
Zero reference model forward passes were executed; memory remained strictly capped at single-model capacity.
High-efficiency finetuning on memory-constrained GPUs (e.g. RTX 4090, single A10G).
Jiwoo Hong et al. (KAIST, 2024) published ORPO, introducing reference-free monolithic alignment.
Odds ratio in one forward pass, no reference in memory: SFT plus odds penalty equals pure efficiency.
Why does standard DPO risk over-fitting on deterministic preferences, and how does IPO enforce bounded quadratic regularization?
Imagine adjusting the tension on a guitar string. DPO keeps tightening the peg as long as the pitch rises, eventually snapping the string! IPO adds a physical stopper: once the string reaches the target tension, the wrench stops turning, preventing catastrophic over-tightening.
- $$\mathcal{L}_{\text{IPO}} = \mathbb{E}\left[ \left( \Delta h_\theta(y_w, y_l) - \frac{1}{2\tau} \right)^2 \right]$$
- $$DPO assumes deterministic preferences under the Bradley-Terry model, causing gradient descent to drive margins to infinity; IPO adds a quadratic regularization term $\left( \hat{r}_\theta - \frac{1}{2\tau} \right)^2$ that bounds margin growth and prevents over-fitting.$$
- Learning goalDerive the quadratic IPO objective and explain how it prevents over-fitting on noisy preference labels.
- Helpful foundationsDirect Preference Optimization (DPO) · Mean squared error loss calculus
- Common trapBelieving margins should always be as large as possible; unbounded margins cause severe likelihood displacement and vocabulary degeneration.
- Next connectionScale to multi-candidate group comparisons using Group Relative Policy Optimization (GRPO) in Screen 4.
More context & mechanics
In Bradley-Terry modeling, if human preference is noisy (e.g., y_w is only preferred 70\% of the time), DPO's unconstrained cross-entropy loss continues driving the log-ratio Δ r → ∞, causing policy collapse.
The IPO Formulation:
Identity Preference Optimization (Azar et al., DeepMind 2023) applies root-finding to the general alignment objective directly without assuming the Bradley-Terry model:
L_(IPO)(θ) = E_((x, y_w, y_l)) [ ( ln (π_θ(y_w | x)) / (π_(ref)(y_w | x)) - ln (π_θ(y_l | x)) / (π_(ref)(y_l | x)) - (1) / (2τ) )² ]
Key Differences from DPO:
- Target Margin Constraint: Rather than driving margin to +∞, IPO drives the margin toward an explicit target (1) / (2τ).
- Quadratic Penalty: If the model over-shoots the target margin, IPO pulls it back!
- No Sigmoid Saturation: Eliminates vanishing gradients on easy examples.
Let regularization parameter τ = 0.5, so the target margin is (1) / (2(0.5)) = 1.0.
- Case 1 (Under-optimized Margin Δ = 0.2):
- Residual: 0.2 - 1.0 = -0.8. Loss: (-0.8)² = 0.64.
- Gradient pushes margin higher toward 1.0.
- Case 2 (Optimal Margin Δ = 1.0):
- Residual: 1.0 - 1.0 = 0.0. Loss: (0.0)² = 0.00.
- System is in perfect equilibrium.
- Case 3 (Over-optimized / Overfitting Margin Δ = 2.5):
- Residual: 2.5 - 1.0 = +1.5. Loss: (1.5)² = 2.25.
- Reversed Gradient: The quadratic penalty forcefully decreases the margin, preventing policy collapse!
Robust preference tuning on noisy, crowdsourced preference datasets where labels have high variance.
Mohammad Gheshlaghi Azar et al. (Google DeepMind, 2023) introduced IPO, establishing theoretical bounds on preference over-fitting.
DPO chases infinity; IPO anchors to a target: square the difference from the goal, keep the policy whole.
How does GRPO eliminate the critic network in mathematical and logical reasoning alignment by normalizing across output groups?
In standard RL, you need a full-time critic network predicting the value of every situation. GRPO says: instead of asking a critic, just ask the student to solve the problem 4 different times! If the student gets it right twice and wrong twice, the average of those 4 attempts serves as the perfect baseline. You eliminate the critic network entirely!
- $$\hat{A}_i = \frac{r_i - \text{mean}(\{r_j\})}{\text{std}(\{r_j\}) + \epsilon}$$
- Learning goalImplement Group Relative Policy Optimization and explain how group normalization eliminates critic networks in reasoning tasks.
- Helpful foundationsPPO Clipped Surrogate Objective · Critic baseline subtraction
- Common trapBelieving RL always requires a value critic network; group normalization computes empirical baselines on-the-fly across candidate rollouts.
- Next connectionExamine inference-time search and best-of-N sampling in Screen 5.
More context & mechanics
In complex reasoning and mathematical verification, training a token-level critic network consumes massive memory and introduces value estimation errors.
The GRPO Architecture (DeepSeek, 2024):
For each question q, sample a group of G outputs o₁, o₂, ṡ, o_G from the old policy π_(old).
Each completion receives a verified scalar reward rᵢ ∈ 0, 1 from automated rule-based verifiers (e.g. sympy or unit tests).
Group-Relative Advantage Normalization:
Instead of a critic V(s), the baseline is computed directly from group statistics:
μ_q = 1/G ∑_(i=1)^G rᵢ, σ_q = √(1/G ∑_(i=1)^G (rᵢ - μ_q)²)
Âᵢ = (rᵢ - μ_q) / (σ_q + ε)
The GRPO Clipped Objective:
L_(GRPO)(θ) = 1/G ∑_(i=1)^G min( (π_θ(oᵢ | q)) / (π_(old)(oᵢ | q)) Âᵢ, clip( (π_θ) / (π_(old)), 1-ε, 1+ε ) Âᵢ ) - β D_(KL)(π_θ ∥ π_(ref))
A math question is sampled with group size G = 4:
- Rollouts 1 & 2 produce correct proofs (r₁ = 1.0, r₂ = 1.0).
- Rollouts 3 & 4 make arithmetic errors (r₃ = 0.0, r₄ = 0.0).
- Group Statistics:
μ = (1 + 1 + 0 + 0) / (4) = 0.50, σ = √((2(0.5)² + 2(-0.5)²) / (4)) = √(0.25) = 0.50
- Group Normalized Advantages:
Â₁ = Â₂ = (1.0 - 0.50) / (0.50) = +1.00 (Strong Positive Reinforcement)
Â₃ = Â₄ = (0.0 - 0.50) / (0.50) = -1.00 (Strong Negative Suppression)
The advantages are zero-centered and scaled without allocating a single byte of memory for a critic network!
Core alignment algorithm powering DeepSeek-Math, DeepSeek-V2, and DeepSeek-R1 reasoning models.
Zhihong Shao et al. (DeepSeek AI, 2024) introduced GRPO in 'DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models'.
Sample a group of four, normalize against the crew: no critic network needed, GRPO makes math dreams come true.
How does scaling inference-time compute via Best-of-N sampling rival the gains of scaling parameter size?
If you ask a student to solve a difficult math problem once under a 30-second timer, they might slip. But if you give them 10 sheets of scratch paper and let them try 10 times, picking their most verified answer, their success rate skyrockets! Test-time search gives models the scratchpad and attempts to solve problems they could never solve in a single shot.
- $$\text{Pass}@N = 1 - (1 - p)^N$$
- Learning goalCalculate pass@N coverage scaling and evaluate tradeoffs between pre-training compute and test-time search compute.
- Helpful foundationsGroup Relative Policy Optimization · Binomial probability distributions
- Common trapBelieving a model's capabilities are strictly locked after training; test-time search dramatically expands reasoning horizons without weight changes.
- Next connectionDive deep into tree search, MCTS, and process verifiers in Chapter 45.
More context & mechanics
Traditional LLM deployment spends fixed compute per token: O(1) forward passes per token.
Test-Time Compute Frontier:
Instead of training a 10× larger model, generate N independent candidate sequences y₁, ṡ, y_N ~ π_θ(· | x) and rank via verifier r(x, y):
y^* = argmax_(i ∈ [1, N]) r(x, yᵢ)
The Pass@N Scaling Dynamics:
If a single completion has probability p of being correct, the probability of at least one correct completion in N independent attempts is:
Pass@N = 1 - (1 - p)^N
As N scales from 1 to 64, a 7B parameter model with p = 0.20 reaches:
Pass@64 = 1 - (0.80)^(64) = 1 - 0.00000062 = 99.9999\%
The challenge shifts entirely from generation to verification: ranking the candidate pool accurately.
A coding model attempts a difficult algorithm question where single-pass accuracy is p = 0.30 (30\%):
- With N = 1 attempt: Success = 30.0\%.
- With N = 4 attempts: Success = 1 - (0.70)⁴ = 1 - 0.2401 = 75.99\%.
- With N = 8 attempts: Success = 1 - (0.70)⁸ = 1 - 0.0576 = 94.24\%.
- With N = 16 attempts: Success = 1 - (0.70)^(16) = 1 - 0.0033 = 99.67\%.
By running 16 forward passes at test time, the small model outperforms a massive 70B model evaluated at single-pass (N=1).
Inference engine architectures powering OpenAI o1, DeepSeek-R1, and competitive programming bots (AlphaCode).
Brown et al. (2024) published 'Large Language Monkeys: Scaling Inference Compute with Repeated Sampling', establishing modern test-time scaling laws.
One shot is gambling, Best-of-N is science: trade inference compute to beat model scale.
How does Modern Preference Optimization: KTO, IPO, and ORPO operate mathematically at scale?
Three progressive alignment method cards:
- Learning goalMaster the algebraic and geometric derivation of Modern Preference Optimization: KTO, IPO, and ORPO.
- Helpful foundationsFoundations of Build Llm From Scratch · Coordinate Invariance & Geometric Projections · Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS
- Common trapAssuming Modern Preference Optimization: KTO, IPO, and ORPO is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
A comparative paradigm chart in square 1:1 format on warm cream parchment (#fcf0db) with a sepia frame.
Top section: Title in navy serif: "Modern Alignment: Beyond Pairwise DPO". Subtitle: "Evolution of preference optimization: KTO (Binary), IPO (Identity Regularized), and ORPO (Reference-Free)."
Middle section: Three progressive alignment method cards:
- KTO (Kahneman-Tversky Optimization):
Does not require paired data (chosen vs rejected)! Operates directly on unpaired thumbs-up / thumbs-down binary feedback using Prospect Theory loss.
- IPO (Identity Preference Optimization):
Adds an explicit L2 regularization term on the implicit reward, preventing DPO from overfitting and driving log-likelihoods to zero on deterministic datasets.
- ORPO (Odds Ratio Preference Optimization):
Monolithic single-stage training! Fuses SFT cross-entropy loss with an odds-ratio preference penalty, eliminating the need for a separate reference model π_(ref) altogether.
Bottom section: Production trend: "Reference-free methods (ORPO) cut training memory by 50% by removing the frozen reference model from VRAM."
Clean 3-panel card comparison, distinct mathematical objectives, balanced square styling.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
"Modern Preference Optimization: KTO, IPO, and ORPO: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."