Temperature Scaling, Top-k & Nucleus (Top-p) Sampling

Temperature scaling divides unnormalized logits by scalar T > 0 before computing softmax:

P(wᵢ) = (exp(zᵢ / T)) / (∑ⱼ exp(zⱼ / T))

• As T → 0, P(w_(max)) → 1.0 (recovering greedy argmax).
• At T = 1.0, standard model probabilities are preserved unchanged.
• As T → ∞, P(wᵢ) → 1/V (converging to a uniform distribution).

Used in all conversational LLM APIs (OpenAI, Anthropic, Google Gemini). Default temperature is typically 0.7 for conversational chat, 0.2 for coding and reasoning, and 1.0 for creative brainstorming.

Borrowed directly from statistical thermodynamics, specifically Ludwig Boltzmann's 1877 Boltzmann distribution Pᵢ ∝ e^(−Eᵢ / k_B T) relating particle states to thermal temperature.

Temperature does not change logit ranking; it only changes the sharpness of the probability peaks.

The relative probability ratio between two candidate tokens i and j is given by:

(P(wᵢ)) / (P(wⱼ)) = exp((zᵢ - zⱼ) / (T))

When T = 0.5, a logit difference of 2.0 becomes an effective gap of 4.0 (e⁴ ≈ 54.6× preference). When T = 2.0, that same gap shrinks to 1.0 (e¹ ≈ 2.7× preference).

Foundational in simulated annealing algorithms, diffusion model score scaling, and temperature-scheduled fine-tuning regimes.

Formulated by Kirkpatrick, Gelatt, and Vecchi (1983) in 'Optimization by Simulated Annealing', linking statistical mechanics to discrete optimization.

Higher temperatures flatten energy landscapes; lower temperatures freeze parameters into the deepest local minimum.

Top-k filtering sorts logits in descending order, identifies the k-th highest value z^((k)), and sets all logits strictly below this threshold to -∞:

z'ᵢ = zᵢ & if zᵢ ≥ z^((k)) \ -∞ & otherwise

When softmax is computed over z', values with -∞ evaluate to e^(−∞) = 0, completely eliminating the bottom V - k tokens.

Standard setting in open-source inference engines (e.g. `topₖ = 40` or `topₖ = 50`), ensuring zero probability is allocated to gibberish tokens.

Popularized by Fan, Lewis, and Dauphin (2018) in 'Hierarchical Neural Story Generation', dramatically reducing story degradation.

Cut the tail with top-k to banish bizarre, catastrophic hallucinations.

Nucleus (top-p) sampling sorts probabilities in descending order p_((1)) ≥ p_((2)) ≥ ṡ ≥ p_((V)) and finds the minimal subset whose cumulative mass exceeds threshold p ∈ (0, 1):

∑_(i=1)^(k^*) p_((i)) ≥ p

All tokens with rank index > k^ have their logits set to -∞. The set size k^ expands dynamically when the model is uncertain, and contracts to 1 when confident.

The gold-standard decoding strategy used across modern AI models (ChatGPT, Claude, LLaMA-3), typically configured with `top_p = 0.9` or `0.95`.

Introduced by Ari Holtzman et al. (2019) in 'The Curious Case of Neural Text Degeneration', demonstrating that human text does not follow fixed-k distributions.

Top-p expands when uncertain and contracts when sure: dynamic probability mass keeps sampling natural.

The complete generation sampling pipeline applies transformations in strict order:

1. Repetition penalty: zᵢ arrow zᵢ / θ for tokens already in context
2. Temperature scaling: z arrow z / T
3. Top-k truncation: mask tokens beyond rank k
4. Top-p nucleus masking: mask tokens beyond cumulative mass p
5. Categorical sampling: w_(next) ~ Categorical(softmax(z))
6. Append and repeat until `<eos>` or max length.

Core decoding algorithm embedded in production inference servers such as vLLM, HuggingFace TGI, and llama.cpp.

Synthesized through years of open-source community experimentation, merging thermodynamic scaling, statistical filtering, and penalty heuristics.

Penalty → Temperature → Top-k → Top-p → Softmax → Sample.

In statistical physics, the probability of a particle occupying state with energy Eᵢ at temperature T is governed by the Boltzmann distribution:

Pᵢ = (e^(−Eᵢ / (k_B T))) / (Z)

In language models, logits act as negative energy levels (zᵢ = -Eᵢ / k_B):

qᵢ(T) = (e^(zᵢ / T)) / (∑ⱼ e^(zⱼ / T))

We analyze the thermodynamic limits:

1. Freezing Regime (T → 0, Absolute Zero):

lim_(T → 0) qᵢ(T) = 1/M & if zᵢ = maxⱼ zⱼ (M-fold degenerate ground state) \ 0 & if zᵢ < maxⱼ zⱼ

Entropy collapses to H → 0. The model selects the argmax token deterministically.

1. Boiling Regime (T → ∞, Infinite Thermal Energy):

lim_(T → ∞) zᵢ/T = 0 ⟹ e^(zᵢ / T) → 1 ⟹ qᵢ(T) = 1/K

Entropy reaches maximum H → log K. Every word in the 50,000-token dictionary becomes equally likely, producing unreadable noise.

LLM serving runtimes (Ollama, vLLM, HuggingFace) expose temperature `T` directly to users: T = 0.0 for SQL query generation, T = 0.7 for assistant chat, T = 1.2 for creative poetry.

Ludwig Boltzmann established statistical thermodynamics in 1877. Geoffrey Hinton applied simulated annealing and temperature to Boltzmann Machines in 1985.

• Cold (T -> 0) freezes into ice (argmax).
• Hot (T -> inf) boils into steam (uniform noise).

Consider logits z = [2.0, 1.0, 0.0] for classes [A, B, C].

1. Neutral Temperature (T = 1.0):

• Scaled logits: z / 1.0 = [2.0, 1.0, 0.0]
• Exponentials: e^(2.0) ≈ 7.3891, e^(1.0) ≈ 2.7183, e^(0.0) = 1.0000
• Partition sum: Z = 7.3891 + 2.7183 + 1.0000 = 11.1074
• Probabilities:
• P(A) = 7.3891 / 11.1074 = 0.6652 (66.5\%)
• P(B) = 2.7183 / 11.1074 = 0.2447 (24.5\%)
• P(C) = 1.0000 / 11.1074 = 0.0900 (9.0\%)

2. Cold Temperature (T = 0.5 — Freezing):

• Scaled logits: z / 0.5 = [4.0, 2.0, 0.0]
• Exponentials: e^(4.0) ≈ 54.5982, e^(2.0) ≈ 7.3891, e^(0.0) = 1.0000
• Partition sum: Z = 54.5982 + 7.3891 + 1.0000 = 62.9873
• Probabilities:
• P(A) = 54.5982 / 62.9873 = 0.8668 (86.7\% — dominant peak!)
• P(B) = 7.3891 / 62.9873 = 0.1173 (11.7\%)
• P(C) = 1.0000 / 62.9873 = 0.0159 (1.6\% — crushed!)

3. Hot Temperature (T = 2.0 — Boiling):

• Scaled logits: z / 2.0 = [1.0, 0.5, 0.0]
• Exponentials: e^(1.0) ≈ 2.7183, e^(0.5) ≈ 1.6487, e^(0.0) = 1.0000
• Partition sum: Z = 2.7183 + 1.6487 + 1.0000 = 5.3670
• Probabilities:
• P(A) = 2.7183 / 5.3670 = 0.5065 (50.7\%)
• P(B) = 1.6487 / 5.3670 = 0.3072 (30.7\%)
• P(C) = 1.0000 / 5.3670 = 0.1863 (18.6\% — flattened toward uniform!)

This exact arithmetic trace is calculated for every token across 50,000 vocabulary logits on GPU hardware during model inference.

This four-decimal worked example serves as the canonical diagnostic check used in unit testing LLM sampling pipelines in PyTorch and Hugging Face Transformers.

• Halving temperature squares the odds ratio.
• Doubling temperature pulls everyone toward a flat 33/33/33 draw.

Let sorted probabilities be p₁ ≥ p₂ ≥ ṡ ≥ p_(|V|).

1. Top-K Sampling (Fan et al., 2018):

Truncate the distribution to a fixed integer K:

V^((K)) = 1, 2, ṡ, K

Set all logits outside V^((K)) to -∞ and re-normalize:

qᵢ' = (pᵢ) / (∑_(j=1)^K pⱼ) & if i ≤ K \ 0 & if i > K

• Flaw: When the distribution is flat, K=5 arbitrarily cuts off valid candidates. When the distribution is peaked, K=5 includes unwanted tail garbage.

1. Nucleus (Top-p) Sampling (Holtzman et al., 2019):

Find the minimal set of top tokens whose cumulative probability exceeds threshold p ∈ (0, 1) (typically p = 0.90):

V^((p)) = 1, 2, ṡ, k^ where k^ = min k | ∑_(i=1)^k pᵢ ≥ p

• Dynamic Adaptability:
• High Confidence: p₁ = 0.95 ⟹ k^* = 1 (pool collapses to 1 token).
• High Uncertainty: pᵢ = 0.05 ⟹ k^* = 18 (pool dynamically expands to 18 tokens).

Nucleus sampling (Top-p = 0.9) combined with mild temperature (T = 0.7) is the default decoding strategy in OpenAI ChatGPT, Anthropic Claude, and Meta LLaMA.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi introduced Nucleus Sampling at ICLR 2020 ('The Curious Case of Neural Text Degeneration'), proving it eliminated repetitive loops and incoherent tail hallucinations.

• Top-K is a rigid box of size K.
• Top-p is a dynamic net that catches 90% of the probability mass.

A comparative decoding probability chart on natural warm linen (#f5f4e9) with a slate border.

Top section: Title in bold sans-serif: "Temperature Scaling: Softening the Probability Simplex". Subtitle: "How temperature T transforms raw logits z = [2.0, 1.0, 0.0] into sampling distributions."

Middle section: Three horizontal probability bar charts side-by-side:

1. Cold Temperature (T = 0.5): Probabilities = [0.867, 0.117, 0.016]. Sharp, decisive peak at token 0; near-greedy argmax.
2. Standard Baseline (T = 1.0): Probabilities = [0.665, 0.245, 0.090]. Balanced natural language distribution.
3. Hot Temperature (T = 2.0): Probabilities = [0.468, 0.316, 0.216]. Flattened distribution; tokens compete evenly.

Show formula: P(wᵢ | z, T) = (exp(zᵢ / T)) / (∑ exp(zⱼ / T)).

Bottom section: Boundary Limits Container: "T → 0: P collapses to Dirac delta δ_(i, argmax(z)) (Hard argmax) | T → ∞: P collapses to Uniform 1/V (Maximum entropy white noise)."

Clear bar charts, generous whitespace, pristine lines.

This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.

"Temperature Scaling: Softening the Probability Simplex: Invariance and structure determine stability under spatial and numerical transformations."

A thermodynamic mathematical proof diagram on natural warm linen (#f5f4e9) with a slate border.

Top section: Title in bold sans-serif: "The Entropy Monotonicity Theorem: Temperature as Uncertainty". Subtitle: "Mathematical proof that Shannon entropy H(P_T) strictly increases with temperature."

Middle section: A clean 2D plot of Shannon entropy H in bits against temperature T ∈ [0, 3].

Mark the calculated points from Slide 085:

• H(T = 0.5) = 0.640 bits (Low uncertainty).
• H(T = 1.0) = 1.250 bits (Moderate uncertainty).
• H(T = 2.0) = 1.512 bits (Approaching theoretical maximum log₂(3) = 1.585 bits).

Boxed Derivative Proof:

"(d H(P_T)) / (d T) = (1) / (T²) Var_(P_T)(z) ≥ 0".

Annotate: "Because variance is strictly non-negative (Var(z) ≥ 0), the derivative is unconditionally ≥ 0; temperature is guaranteed to be a monotonic entropy dial."

Bottom section: Card: "Increasing temperature NEVER reduces uncertainty; it is mathematically impossible for an LLM to become more confident as T rises."

Spacious curve layout, sharp mathematical typography, zero clutter.

This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.

"The Entropy Monotonicity Theorem: Temperature as Uncertainty: Invariance and structure determine stability under spatial and numerical transformations."

A probability softening trace diagram in square 1:1 format on warm cream parchment (#fcf0db) with a sepia frame.

Top section: Title in navy serif: "Temperature Scaling: Softening the Simplex". Subtitle: "How dividing logits by scalar T controls generation entropy and diversity."

Middle section: Comparison of three probability distributions from identical logits z = [4.0, 2.0, 1.0]:

1. Low Temperature (T = 0.2, Greed/Cold):

Scaled logits: [20.0, 10.0, 5.0].

Probabilities: [0.9999, 0.0001, 0.0000]. One-hot collapse; completely deterministic.

1. Standard Temperature (T = 1.0, Base):

Scaled logits: [4.0, 2.0, 1.0].

Probabilities: [0.844, 0.114, 0.042]. Balanced selection reflecting raw training distribution.

1. High Temperature (T = 2.0, Creative/Hot):

Scaled logits: [2.0, 1.0, 0.5].

Probabilities: [0.551, 0.276, 0.173]. Softened distribution; increases entropy and diversity.

Bottom section: Mathematical limit: "lim_(T → 0) Softmax(z/T) = argmax(z) (Argmax), lim_(T → ∞) Softmax(z/T) = Uniform(1/K)."

Clear bar charts for each temperature level, crisp numeric annotations, balanced square frame.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI ↗

"Temperature Scaling: Softening the Probability Simplex: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A mathematical theorem card in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.

Top section: Title in navy serif: "The Entropy Monotonicity Theorem". Subtitle: "Proof that Shannon entropy H(P_T) is a strictly increasing function of temperature T."

Middle section: The mathematical derivation:

1. Define temperature-scaled probability: pᵢ(T) = (e^(zᵢ / T)) / (∑ⱼ e^(zⱼ / T)).
2. Define Shannon entropy: H(T) = -∑ᵢ pᵢ(T) log pᵢ(T).
3. Compute derivative with respect to temperature:

dH/dT = (1) / (T³) Var_(i ~ p)[zᵢ] = (1) / (T³) ( ∑ᵢ pᵢ zᵢ² - (∑ᵢ pᵢ zᵢ)² ).

1. Variance of non-identical logits is strictly positive: Var[z] > 0.
2. Therefore: dH/dT > 0 for all T > 0.

Bottom section: Thermodynamic parallel: "As temperature increases, entropy strictly increases. LLM temperature scaling directly mimics Boltzmann thermodynamic entropy."

Elegant LaTeX lines, clean positive variance highlight, balanced square styling.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI ↗

"The Entropy Monotonicity Theorem: Temperature as Uncertainty: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."