Pretraining Pipeline & Sampling Dynamics
How do numerical stability shields, temperature scaling, and multinomial sampling steer the autoregressive pretraining engine?
How this lesson unfolds.
Training large language models requires orchestrating massive matrix multiplications while avoiding floating-point overflows and probability crashes. Once trained, generation requires sampling from probability simplexes using calibrated thermodynamic temperature dials.
How does scaling logits by temperature T control the entropy and sharpness of autoregressive token selection?
Imagine holding a blowtorch to a tray of frozen wax figurines. At near-zero temperature (T = 0.1), the wax is rock-solid: the tallest figurine towers undisputed, and all smaller figurines are completely ignored (argmax greedy choice). As you crank up the flame to high heat (T = 2.0), the wax melts into a flat, boiling puddle: every figurine flattens to the same height, and choosing a figurine becomes an unpredictable roll of the dice.
- Learning goalAdjust softmax temperature to systematically calibrate distribution entropy between greedy argmax and uniform exploration.
- Helpful foundationsSoftmax function · Statistical Mechanics (Pathria & Beale, 2011)
- Common trapBelieving temperature modifies the underlying model weights; temperature is applied purely at post-processing inference time to the output logits.
- Next connectionDividing logits by extreme numbers or taking raw exponentials can cause float32 numbers to instantly explode into NaN; how do we shield against floating-point underflow and overflow?
More context & mechanics
Given logit vector z ∈ R^K, temperature-scaled softmax is defined as:
P(y = i | z, T) = (e^(zᵢ / T)) / (∑_(j=1)^K e^(zⱼ / T))
We analyze the asymptotic behavior:
- Low Temperature (T → 0^+):
lim_(T → 0^+) P(y = i) = 1 & if i = argmaxⱼ zⱼ \ 0 & otherwise
The probability mass collapses into a Dirac delta distribution on the maximal logit (zero entropy).
- High Temperature (T → ∞):
lim_(T → ∞) zᵢ/T = 0 ⟹ e^(zᵢ / T) → 1 ⟹ P(y = i) = 1/K
The distribution approaches the maximum entropy uniform distribution U(1, K).
In LLM generation APIs (OpenAI, Anthropic, Google Gemini), T=0.2 is used for structured code generation and deterministic math reasoning, while T=0.8 to 1.0 is chosen for creative writing, brainstorms, and narrative generation.
The temperature parameter T is taken directly from the Boltzmann factor e^(−E / (k_B T)) in 19th-century statistical thermodynamics, where physical temperature determines the probability of particles occupying excited energy states.
- Freeze (T → 0) for precision; boil (T → ∞) for chaos.
- Temperature divides the exponents before the budget is split.
How does subtracting the maximum logit eliminate catastrophic float32 overflow and underflow in softmax?
Standard computer numbers (IEEE 754 float32) hit a concrete ceiling: any number larger than 88.7 produces `inf` when passed to e^x, and anything smaller than -87.3 underflows to zero. If your neural network outputs a logit of 95, computing e^(95) instantly crashes your program with `NaN`. The underflow shield simply subtracts the highest score from every score before exponentiating. If your scores are [90, 95, 92], you subtract 95 to get [-5, 0, -3]. The highest value is now exactly e⁰ = 1.0, completely eliminating floating-point crashes.
- $$Exploit the shift-invariance identity exp(z - c) / sum exp(z_j - c) = exp(z) / sum exp(z_j) with c = max(z) to prevent IEEE 754 float overflow.$$
- Learning goalProve the shift-invariance of the softmax operator and implement the numerically stable Log-Sum-Exp algorithm.
- Helpful foundationsIEEE 754 floating point standard · Logarithm and exponential rules · What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg, 1991)
- Common trapThinking that subtracting a constant changes the output probabilities; since both numerator and denominator scale by e^(-c), the constant factor cancels out exactly.
- Next connectionWith a numerically stable probability distribution in hand, how does the model actually pick the next word? It rolls a high-dimensional die.
More context & mechanics
Mathematical proof of shift-invariance:
(e^(zᵢ - c)) / (∑_(j=1)^K e^(zⱼ - c)) = (e^(−c) · e^(zᵢ)) / (e^(−c) ∑_(j=1)^K e^(zⱼ)) = (e^(zᵢ)) / (∑_(j=1)^K e^(zⱼ))
By choosing c = maxⱼ zⱼ:
- The maximal shifted logit is z_(max) - c = 0.
- The maximal exponentiated term is e⁰ = 1.0.
- All shifted logits satisfy zᵢ - c ≤ 0, ensuring 0 < e^(zᵢ - c) ≤ 1.0.
- The denominator sum satisfies 1.0 ≤ ∑_(j=1)^K e^(zⱼ - c) ≤ K, preventing division by zero or overflow to +∞.
This identity is baked into PyTorch's `torch.nn.functional.log_softmax` and the GPU Triton kernels inside FlashAttention. Calculating log ∑ⱼ e^(zⱼ) stably via c + log ∑ⱼ e^(zⱼ - c) prevents gradient explosions during large-batch training.
Numerical analysts recognized exponential overflow in early floating-point hardware in the 1960s. The Log-Sum-Exp trick became standard canon in machine learning software with the release of NumPy, Theano, and early CUDA BLAS libraries.
- Subtract the max before you take the exp.
- Top score becomes zero; top exponent becomes one; NaN is conquered.
How does multinomial sampling pick the next token using cumulative probability thresholds?
Imagine a 20-sided die where faces are not equal, but sized according to the model's confidence: token A gets 50% of the surface, token B gets 30%, and token C gets 20%. Instead of rolling physical dice, the computer draws a random decimal between 0.000 and 1.000 from a uniform generator. If the number lands between 0.000 and 0.500, it selects token A; if between 0.500 and 0.800, token B; and if between 0.800 and 1.000, token C. Cumulative probability turns random numbers into proportional text generation.
- Learning goalImplement inverse transform multinomial sampling from discrete probability mass vectors using cumulative sum thresholds.
- Helpful foundationsCumulative distribution functions · Pseudo-random number generators · Non-Uniform Random Variate Generation (Devroye, 1986)
- Common trapConfusing temperature with sampling method; temperature modifies the probability distribution, whereas sampling determines how a token is chosen from that distribution.
- Next connectionIn long sequences, tokens need to know where they stand relative to each other; how do we encode word order without rigid absolute coordinates? We introduce clock-face rotations.
More context & mechanics
Given a categorical distribution p = [p₁, p₂, ṡ, p_K] with ∑_(k=1)^K pₖ = 1:
- Construct the Cumulative Distribution Function (CDF):
Cₘ = ∑_(k=1)^m pₖ, C₀ = 0, C_K = 1.0
- Sample a uniform pseudo-random scalar:
u ~ U[0, 1)
- Identify the token index i satisfying:
C_(i-1) ≤ u < Cᵢ
By the Inverse Transform Sampling theorem, the probability of selecting index i is exactly Cᵢ - C_(i-1) = pᵢ.
In `torch.multinomial(probs, num_samples=1)`, this inverse CDF lookup is implemented using parallel binary search (`std::upper_bound`) across the cumulative sum buffer, executing in O(log |V|) time on modern GPUs.
Stanislaw Ulam and John von Neumann invented the Monte Carlo method and inverse transform sampling at Los Alamos in 1946 while calculating neutron diffusion during the Manhattan Project.
- Lay probabilities end-to-end on a 0-to-1 ruler; throw a dart.
- The wider the segment, the more often the dart lands there.