How ChatGPT Works
How does minimizing cross-entropy on next-token prediction induce high-level reasoning, code synthesis, and world models?
How this lesson unfolds.
Begin with How does minimizing cross-entropy on next-token prediction induce high-level reasoning, code synthesis, and world models?. Follow the visual diagrams to build an intuitive mental model.
How does minimizing cross-entropy on next-token prediction induce high-level reasoning, code synthesis, and world models?
If you ask a child to predict the very next word in a mystery novel: 'The detective opened the safe and found the stolen...', they cannot guess the word 'diamond' by memorizing letter frequencies. To guess 'diamond' with high probability, the child's brain must have tracked who the suspect was, what was stolen in chapter 2, how safes work, and what motives existed. In an identical way, to predict the next token across 15 trillion tokens of science, law, and code, a 70-billion parameter transformer must compress an internal simulation of physics, logic, grammar, and human psychology.
- Learning goalFormulate next-token prediction as maximum likelihood estimation and prove how loss minimization compels the formation of internal world models.
- Helpful foundationsMaximum Likelihood Estimation (MLE) · Cross-entropy loss and Kullback-Leibler (KL) divergence · Language Models are Unsupervised Multitask Learners (Radford et al., 2019)
- Common trapBelieving next-token prediction is mere superficial 'stochastic parrot' pattern matching. Perfect next-token prediction is formally AI-complete: predicting the next token of arbitrary human text requires the computational capacity to simulate the cognitive and physical processes that generated that text.
- Next connectionBecause the base model learns rich general world representations during next-token prediction, it can perform novel tasks through in-context learning without updating a single weight.
More context & mechanics
Next-token prediction optimizes the empirical log-likelihood of a sequence:
max_θ ∑_(t=1)^T log P_θ(w_t | w₁, ṡ, w_(t-1))
Equivalently, minimizing cross-entropy is minimizing the Kullback-Leibler (KL) divergence between true data distribution P and model distribution P_θ:
D_(KL)(P ∥ P_θ) = ∑_(w) P(w) log (P(w)) / (P_θ(w)) = H(P, P_θ) - H(P)
To drive cross-entropy below 2.0 bits/token on arbitrary internet text, the network cannot rely on surface memorization. It must learn:
- Linguistic Syntax: Agreement, verb conjugation, parse tree structures.
- Algorithmic State Machines: Execution of Python code, tracking variable assignments and loops.
- World Facts and Commonsense: Physics of gravity, temporal causality, geographical knowledge.
This single objective powers generative engines from code completion (GitHub Copilot) to legal analysis and creative writing. When conditioned on a prompt like `def binary_search(arr, target):`, the model's next-token predictions construct a mathematically valid algorithm.
Ilya Sutskever championed the hypothesis that predicting the next token with extreme accuracy forces the emergence of true understanding. While early AI researchers viewed text prediction as a trivial toy task, OpenAI demonstrated that scaling autoregression transformed language models into general problem solvers.
- To predict the future token, you must understand the entire past universe.
- Compression of text is induction of world models.
How do zero-shot, one-shot, and few-shot prompting steer frozen transformer representations to perform new tasks?
Imagine hiring a genius linguist who has never heard of your specific company acronyms. You don't send them back to university for four years (fine-tuning). Instead, you write a short sticky note: 'Here are two examples: ACME-1 means Priority Delivery; ACME-2 means Standard Ground. Now classify ACME-1.' The linguist glances at the two examples, understands the pattern immediately, and executes the classification. That sticky note is few-shot prompting: demonstration examples provided directly inside the context window that configure the frozen model's attention state on the fly.
- Learning goalConstruct zero-shot, one-shot, and few-shot prompt templates and analyze the empirical scaling curves of in-context task adaptation.
- Helpful foundationsPrompt engineering syntax · Language Models are Few-Shot Learners / GPT-3 (Brown et al., 2020) · An Explanation of In-context Learning as Implicit Bayesian Inference (Xie et al., 2021)
- Common trapAssuming that few-shot learning requires gradient backpropagation or parameter updates. In-context learning is entirely non-parametric at test time: all model weights Theta remain 100% frozen; adaptation occurs through dynamic attention key-value activations.
- Next connectionHaving understood how next-token prediction and in-context learning create conversational systems, Lesson 6 summarizes the four comprehensive stages of building an LLM from scratch.
More context & mechanics
In-context learning modalities:
- Zero-Shot: Direct instruction without examples:
`Prompt: "Translate English to German: cheese => "` → `Output: "Käse"`
- One-Shot: A single demonstration pair:
`Prompt: "sea otter => loutre de mer \n cheese => "` → `Output: "fromage"`
- Few-Shot (k-shot): k demonstrations (x₁, y₁), ṡ, (xₖ, yₖ) followed by query x_(k+1):
X = [x₁, y₁, x₂, y₂, ṡ, xₖ, yₖ, x_(k+1)]
The attention mechanism calculates cross-token dependencies:
Attn(q_(k+1), K_(demos), V_(demos))
The demonstration tokens locate the latent task concept in activation space, acting as an implicit gradient descent update in forward activations (von Oswald et al., 2023).
Modern RAG systems combine few-shot prompts with dynamic vector search, injecting 3 to 5 domain demonstrations into the context window to enforce rigid JSON schemas, tone constraints, or citation guidelines without fine-tuning weights.
Brown et al. unveiled GPT-3 in 2020 ('Language Models are Few-Shot Learners'), demonstrating that as models scaled from 1.3B to 175B parameters, few-shot prompt performance climbed steeply along smooth power-law curves, establishing prompt engineering as a core discipline.
- Zero-shot asks; one-shot hints; few-shot teaches by pattern.
- No weight updates needed: the context window is the dynamic canvas of intelligence.
Why is a base pretrained language model dangerous or unhelpful for conversational interaction, and how does RLHF align its outputs with human intent?
Imagine raising a prodigy child in a library filled with every book, web forum, and argument in the world. The prodigy can recite medical textbooks, compose Shakespearean sonnets, or write venomous insult rants on command—but has zero manners, zero concept of helpfulness, and no filter. If you ask 'How do I pick a lock?', a raw base model doesn't answer your question; it continues the text as a burglar mystery story or generates another question. Supervised Fine-Tuning (SFT) teaches the prodigy how to follow instructions, and RLHF teaches them ethical boundaries and conversational tact.
- Learning goalFormulate the PPO-RLHF objective function with the KL divergence penalty and calculate step-by-step reward updates.
- Helpful foundationsTraining language models to follow instructions with human feedback (Ouyang et al., OpenAI, 2022) · Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023) · Language Models are Few-Shot Learners / GPT-3 (Brown et al., 2020)
- Common trapAssuming ChatGPT's factual knowledge comes from RLHF. RLHF imparts almost zero new factual knowledge; all world knowledge is acquired during pretraining. RLHF merely acts as a stylistic and behavioral steering wheel.
- Next connectionTo understand where pretraining ends and alignment begins in an end-to-end production workflow, Lesson 06 maps out The Four Stages of Building an LLM.
More context & mechanics
RLHF optimizes the policy network π_θ(y | x) using Proximal Policy Optimization (PPO) against a scalar reward model r_φ(x, y) while penalizing divergence from the frozen base model π_(ref):
max_θ E_(x, y) ~ D_(π_θ) [ r_φ(x, y) - β D_(KL)(π_θ(y | x) ∥ π_(ref)(y | x)) ]
Where the Kullback-Leibler (KL) divergence penalty ensures the policy does not drift into nonsensical reward-hacking regions:
D_(KL)(π_θ ∥ π_(ref)) = ∑_(t=1)^T ln ( (π_θ(y_t | x, y_(<t))) / (π_(ref)(y_t | x, y_(<t))) )
Toy Numerical Trace: Suppose prompt x produces response token y. The reward model awards r_φ = +2.4. The base model assigned probability π_(ref) = 0.08, while the updated policy assigns π_θ = 0.16 (ratio = 2.0). With KL penalty weight β = 0.10:
KL penalty = β ln(2.0) = 0.10 × 0.6931 = 0.0693
Net Objective = 2.4 - 0.0693 = 2.3307
The model is rewarded for producing a high-quality answer while staying safely within the reference distribution.
RLHF turned raw GPT-3 (released in 2020 as a research curiosity) into ChatGPT (November 2022), sparking the consumer AI revolution. Modern variants like Direct Preference Optimization (DPO) and KTO use closed-form analytical solutions to achieve the same alignment without training a separate reward model.
Formulated by Christiano et al. (2017) at OpenAI and DeepMind in 'Deep Reinforcement Learning from Human Preferences', and scaled to dialogue systems by Ouyang et al. (2022) in the landmark 'InstructGPT' paper.
- Pretraining gives the model raw intelligence; alignment gives the model a conscience and conversational purpose.
- Without the KL penalty, models hack the reward function into repetitive gibberish.