LLM Loss Engine: Target Shifting, Cross-Entropy & Perplexity

The objective function measures the discrepancy between predicted probability distribution p_t = softmax(z_t) and empirical one-hot target distribution q_t = e_(y_t):

L = E_((x, y) ~ D) [-∑_(t=1)^T log p_t(y_t)]

Minimizing cross-entropy is mathematically equivalent to minimizing the Kullback-Leibler (KL) divergence between empirical data and model predictions.

Universal objective for pretraining foundation models (Chinchilla, LLaMA, GPT). Loss curves serve as the primary monitoring metric across distributed clusters.

Rooted in maximum likelihood estimation (Fisher, 1922) and information theory (Shannon, 1948). First applied to neural language modeling by Bengio et al. (2003).

The loss function is the mathematical compass of gradient descent; without it, optimization has no direction.

Under uniform probability distribution pᵢ = 1/V, the negative log-likelihood evaluates to:

L_(init) = -∑_(i=1)^V qᵢ ln(1/V) = -ln(1/V) = ln(V)

For GPT-2 vocabulary V = 50,257, ln(50257) = 10.8249. For LLaMA vocabulary V = 32,000, ln(32000) = 10.373.

Essential diagnostic sanity check: if epoch 0 step 0 loss is not ≈ ln(V), your weight initialization or label shifting logic is broken.

Standardized by Andrej Karpathy and modern deep learning educators as the essential 'sanity check' in generative modeling.

Initial loss is the logarithm of ignorance: ln(V) is the mathematical signature of zero knowledge.

Self-supervised language modeling solves the data labeling bottleneck through Temporal Target Shifting:

1. The Self-Supervision Miracle:

Supervised deep learning typically requires expensive manual annotations (e.g. labeling bounding boxes or sentiment tags). In autoregressive language modeling, the text itself contains both the input and the answer.

1. The Shift-by-One Contract:

Given a raw text sequence of T+1 tokens S = [w₀, w₁, w₂, ṡ, w_T]:

• Input Sequence X: The first T tokens:

X = S[0 : T] = [w₀, w₁, ṡ, w_(T-1)]

• Target Sequence Y: The sequence shifted right by 1 position:

Y = S[1 : T+1] = [w₁, w₂, ṡ, w_T]

1. Massive Parallelism:

Because causal attention masks future tokens, a single forward pass over input sequence X simultaneously generates predictions for all T positions in parallel:

• Position 0 predicts w₁.
• Position 1 predicts w₂.
• Position T-1 predicts w_T.

A sequence of length 2,048 yields 2,048 distinct supervision targets in one single matrix multiply!

This single mathematical principle enables pretraining on trillions of tokens of web text, books, code, and Wikipedia. Llama-3 was trained on over 15 trillion tokens using target-shifted self-supervision, requiring zero manual human labeling for its foundation stage.

Self-supervised target shifting was popularized in NLP by Mikolov's Skip-gram/CBOW (2013) and generalized to autoregressive sequence models in Char-RNN (Karpathy 2015) and GPT-1 (Radford 2018).

• Shift by one to teach yourself: X is tokens 0 to T-1; Y is tokens 1 to T.
• The text is its own teacher: Every word in the library is a test question for the words that preceded it.

#### 1. Python Slicing Syntax and Shapes

Let a mini-batch of tokenized text be represented as a 2D integer tensor:

tokens ∈ Z^(B × (T+1))

Where B is the batch size and T+1 is the total loaded sequence length.

Using standard Python slicing:

```python

x = tokens[:, :-1] # Shape: [B, T]

y = tokens[:, 1:] # Shape: [B, T]

```

• `x` omits the final token, serving as the model input.
• `y` omits the initial token, serving as the ground-truth target.

The model computes logits for input `x`:

Z = model(x) ∈ R^(B × T × V)

Notice that Z[:, t, :] contains the unnormalized logit distribution predicting the token at index t+1.

To compute cross-entropy loss using PyTorch's `F.cross_entropy`, the 3D tensor must be flattened into 2D:

```python

logits_flat = Z.contiguous().view(B * T, V) # Shape: [N, V]

targets_flat = y.contiguous().view(B * T) # Shape: [N]

loss = F.cross_entropy(logits_flat, targets_flat)

```

Where N = B × T is the total count of prediction instances in the batch.

Let B = 1, T = 3, with token IDs:

tokens = [[101, 2045, 3012, 102]]

Applying slicing:

• x = tokens[:, :-1] = [[101, 2045, 3012]] (Length 3)
• y = tokens[:, 1:] = [[2045, 3012, 102]] (Length 3)

Alignment check:

• Input `[101]` → target is `2045` ⟹ Logits Z[0, 0, :] evaluated against target `2045`.
• Input `[101, 2045]` → target is `3012` ⟹ Logits Z[0, 1, :] evaluated against target `3012`.
• Input `[101, 2045, 3012]` → target is `102` ⟹ Logits Z[0, 2, :] evaluated against target `102`.

Every token is evaluated against its true successor.

In instruction fine-tuning (SFT), prompts must not be trained on. Masking is applied by setting prompt token targets in `y` to `-100`: `y[prompt_mask] = -100`. PyTorch's `F.cross_entropy(..., ignore_index=-100)` automatically skips masked positions, computing loss strictly on assistant responses.

The `ignore_index=-100` convention was established in PyTorch's early codebase (circa 2017) to support variable-length padded batches in sequence-to-sequence models, and remains the universal standard for SFT target masking today.

• `x = tokens[:, :-1]`, `y = tokens[:, 1:]`: Slice the head off y; slice the tail off x.
• Flatten to B · T: Collapse batch and time into a single vertical list of predictions.

PyTorch `torch.nn.CrossEntropyLoss` expects logits of shape (N, C) and targets of shape (N) where N is total sample count and C is class count:

Z_(flat) = Z.view(-1, V) ∈ R^((B · T) × V)

Y_(flat) = Y.view(-1) ∈ Z^(B · T)

Contiguous memory layout ensures flattening executes as a zero-copy pointer view.

Standard tensor transformation in all GPT implementations prior to calling `F.cross_entropy(logits, targets)`.

Developed in early Torch/Caffe tensor backends where linear algebra operations required 2D matrix representations for BLAS acceleration.

Collapse batch and time into a single unified stream of token decisions: view(-1, V) feeds the loss engine.

Cross-entropy measures the divergence between the true distribution q and the predicted distribution p:

1. The Negative Log-Likelihood Formulation:

For a discrete categorical target where ground truth is a one-hot vector y (y_c = 1 for target class c, and 0 otherwise):

L_(CE) = - ∑_(k=1)^V yₖ ln(pₖ) = - ln(p_c)

Where p_c is the model's predicted softmax probability for the correct token c.

1. The Asymmetry of Logarithmic Penalties:

• As p_c → 1.0: L_(CE) → -ln(1.0) = 0.0 (Zero penalty for perfect foresight).
• As p_c → 0.5: L_(CE) = -ln(0.5) ≈ 0.693.
• As p_c → 0.01: L_(CE) = -ln(0.01) ≈ 4.605.
• As p_c → 0.0: L_(CE) → +∞ (Infinite penalty!).

1. The Gradient Driving Force:

The derivative with respect to pre-softmax logit zₖ simplifies to the elegant prediction error:

(∂ L_(CE)) / (∂ zₖ) = pₖ - yₖ

When the model assigns p_c = 0.01 to the target class (y_c = 1), the gradient error is 0.01 - 1.0 = -0.99, driving a massive gradient update to push the weight toward the correct answer.

Cross-entropy is the universal loss function for pretraining all autoregressive foundation models. Training progress is monitored via training loss curves, where loss directly reflects the statistical perplexity of the language model (PPL = e^(L)).

The concept of entropy was formulated by Rudolf Clausius in thermodynamics (1865) and translated to information theory by Claude Shannon in 1948. The cross-entropy loss function is mathematically equivalent to minimizing the Kullback-Leibler (KL) divergence between empirical data and model predictions.

• -ln(p): As probability shrinks to zero, penalty explodes to infinity.
• Gradient is prediction minus truth: (∂ L) / (∂ z) = p - y; the error is simply what you predicted minus what actually happened.

#### 1. The Log-Sum-Exp Derivation

Let logits be z ∈ R^V and target token index be c.

Substituting the softmax definition into cross-entropy:

L = -ln(p_c) = -ln( (e^(z_c)) / (∑_(j=1)^V e^(zⱼ)) ) = -[ ln(e^(z_c)) - ln( ∑_(j=1)^V e^(zⱼ) ) ]

Using ln(e^(z_c)) = z_c:

L = -z_c + ln( ∑_(j=1)^V e^(zⱼ) )

This is the celebrated Log-Sum-Exp (LSE) formulation. It avoids computing raw probabilities entirely, preventing floating-point underflow.

Consider an untrained GPT-2 model at step 0 with random weight initialization:

1. Symmetry Assumption: Because weights are initialized with independent zero-mean Gaussian noise (W ~ N(0, σ²)), the expected output logits for all vocabulary tokens are identical:

E[z₁] = E[z₂] = ṡ = E[z_V] = z̄

1. Uniform Probability: The softmax probabilities over vocabulary size V are uniform:

pₖ = (e^(z̄)) / (∑_(j=1)^V e^(z̄)) = (e^(z̄)) / (V e^(z̄)) = 1/V

1. Initial Loss Evaluation:

Substituting p_c = 1/V into the cross-entropy formula:

L_(init) = -ln( 1/V ) = ln(V)

1. Exact Numerical Evaluation for GPT-2 (V = 50,257):

L_(init) = ln(50,257) = 10.824905ṡ ≈ 10.82

This serves as the universal golden unit test for transformer training pipelines. If step 0 loss is not ≈ 10.82, check for bad target shifting or faulty vocab indexing immediately! \blacksquare

Verify for smaller vocabularies:

• For V = 10: L_(init) = ln(10) ≈ 2.3026.
• For V = 32,000 (Llama-1/2): L_(init) = ln(32,000) ≈ 10.3735.
• For V = 128,256 (Llama-3): L_(init) = ln(128,256) ≈ 11.7618.

If a Llama-3 training run begins at loss 11.76, engineers confirm that vocabulary heads, loss masking, and tensor shapes are mathematically certified.

In PyTorch, `torch.nn.CrossEntropyLoss()` fuses Log-Sum-Exp internally via C++/CUDA kernels, using the max-subtraction trick ln(∑ e^zⱼ - z_(max)) + z_(max) to guarantee zero floating-point overflow.

The mathematical equivalence between maximum likelihood estimation and cross-entropy minimization was proven by Sir Ronald Fisher in 1922. The 10.82 initial loss rule of thumb became famous across AI engineering after Andrej Karpathy highlighted it as the ultimate sanity check for deep learning codebases.

• L_(0) = ln(|V|): Step zero loss is the natural logarithm of the vocabulary size.
• 10.82 for GPT-2, 11.76 for Llama-3: Memorize your architecture's baseline; never train without verifying step 0.

Perplexity is the exponential of the cross-entropy loss:

PPL = e^(L) = exp(-1/T ∑_(t=1)^T ln P(w_t | w_(<t)))

For an untrained model with L = 10.82, PPL = e^(10.82) = 50,257 (pure random choice). A trained model with L = 3.0 achieves PPL = e^(3.0) ≈ 20.08.

The standard evaluation metric reported in academic literature (Chinchilla, LLaMA, GPT-3) on benchmark corpora such as WikiText-103 and C4.

Introduced by Jelinek et al. (1977) in speech recognition, named 'perplexity' because it measures how 'perplexed' the statistical model is by the text.

Perplexity is the geometric branching factor of language: lower perplexity means fewer blind guesses.

The masked cross-entropy loss excludes target positions where yᵢ = ignore\_index (conventionally -100):

L = -(1) / (∑_(i) I(yᵢ ≠ -100)) ∑_(i, yᵢ ≠ -100) ln (e^z_(i, yᵢ)) / (∑ⱼ e^z_(i, j))

Tokens with target -100 contribute 0 to the loss numerator and 0 to the denominator count.

Crucial in multi-turn conversational finetuning and batch inference where variable-length sequences are padded with `<pad>` tokens.

Implemented as a standard feature in PyTorch `torch.nn.CrossEntropyLoss` by Adam Paszke and the PyTorch core team to streamline NLP pipelines.

Never penalize a model for the empty void: mask padding to keep gradients focused on real text.

Exponential moving average smooths the volatile per-batch loss L_t using a smoothing factor β ∈ [0.9, 0.99]:

L̄_t = β L̄_(t-1) + (1 - β) L_t

Bias correction for early iterations is given by L̂_t = (L̄_t) / (1 - β^t).

Integrated into training dashboards like Weights & Biases (WandB) and TensorBoard to provide clean loss convergence trajectories.

Developed by Charles Holt (1957) in time-series forecasting and adopted in neural network training monitors to filter stochastic minibatch noise.

Minibatches are noisy samples; exponential smoothing reveals the true gradient trajectory.

Label smoothing replaces hard one-hot target vectors with a mixture of ground truth and uniform noise:

q_(smooth)(k) = (1 - ε) · I(k = y) + (ε) / (V)

For smoothing parameter ε = 0.1 and V = 50,000, target probability is 0.90 for the correct class and 0.000002 for all other classes.

Widely used in machine translation (Transformer, Vaswani et al.) and vision architectures to improve calibration and reduce overconfidence.

Proposed by Szegedy et al. (2016) in 'Rethinking the Inception Architecture for Computer Vision' to regularize deep classification networks.

Absolute certainty breeds brittleness: smooth your targets to foster adaptive humility.

The analytical derivative of cross-entropy loss with respect to logit zₖ simplifies through the softmax derivative cancellation:

(∂ L) / (∂ zₖ) = pₖ - qₖ = softmax(zₖ) - I(k = y)

This elegant result proves that cross-entropy gradient is simply the residual prediction error.

High-performance GPU fused cross-entropy kernels (FlashCrossEntropy, Liger-Kernel) compute this subtraction directly in SRAM, bypassing intermediate activation saves.

Derived by Rumelhart, Hinton, and Williams (1986) in the foundation of backpropagation, highlighting the remarkable mathematical synergy between softmax and cross-entropy.

Gradient equals prediction minus truth: p - y is the universal pulse of neural backpropagation.

The unified loss engine executes five connected transformations:

1. Target shift: x_(in) = x[:, :-1], y_(tgt) = x[:, 1:]
2. Forward pass: Z = Model(x_(in)) ∈ R^(B × (T-1) × V)
3. Flatten: Z_(flat) = Z.reshape(-1, V), Y_(flat) = y_(tgt).reshape(-1)
4. Loss evaluation: L = F.cross\_entropy(Z_(flat), Y_(flat), ignore\_index=-100)
5. Metric tracking: PPL = e^(L)

The standard loss module implemented across all modern open-source LLM training codebases (Torchtitan, Megatron-LM, nanoGPT).

Refined across seven years of transformer engineering, crystallizing into the canonical training loop architecture used worldwide.

Shift → Flatten → Mask → Softmax-Cross-Entropy → Perplexity.

Under the autoregressive chain rule of probability:

P(x₁, x₂, ṡ, x_T) = ∏_(t=1)^T P(x_t | x_(<t))

In IEEE 754 float32, the minimum representable positive normal float is:

f_(min) ≈ 1.175 × 10^(-38)

For any sequence where ∏_(t=1)^T P(x_t | x_(<t)) < f_(min), hardware registers trigger an underflow exception, rounding the likelihood to 0.0.

Taking the natural logarithm transforms the product into an additive accumulation:

log P(x₁, ṡ, x_T) = ∑_(t=1)^T log P(x_t | x_(<t))

Since log(0.01) ≈ -4.605, summing 1,000 tokens yields ≈ -4605.0, which sits well within float32 range (-3.4 × 10^(38) to +3.4 × 10^(38)).

This is why all generative LLMs are trained to minimize Negative Log-Likelihood (NLL) Loss rather than maximizing raw probability products:

L_(NLL) = -∑_(t=1)^T log P(x_t | x_(<t))

John Napier invented logarithms in 1614 specifically to simplify tedious astronomical multiplications into additions. Ronald Fisher adopted log-likelihood in 1922 as the foundation of modern statistical estimation.

• Logarithms turn multiplication into addition.
• Products crash to zero; log-sums stay in safe arithmetic waters.

Shannon defined the self-information (surprisal) of an event x with probability P(x) as:

I(x) = -log₂ P(x) = log₂ ( (1) / (P(x)) )

Measured in bits (shannons) when using base 2.

Key mathematical axioms:

1. Certainty has zero information: If P(x) = 1.0 ⟹ I(x) = -log₂(1) = 0 bits.
2. Positivity: Since 0 < P(x) ≤ 1 ⟹ I(x) ≥ 0.
3. Monotonicity: If P(A) < P(B) ⟹ I(A) > I(B) (rarer events carry more information).
4. Additivity of Independent Events: If events A and B are independent:

P(A ∩ B) = P(A) P(B)

I(A ∩ B) = -log₂(P(A)P(B)) = -log₂ P(A) - log₂ P(B) = I(A) + I(B)

Surprisal is the fundamental basis for Huffman coding, arithmetic coding, and LLM text generation metrics. When a language model outputs a high negative log-likelihood on a token, it indicates high surprisal (the word was unpredictable given context).

Claude Shannon founded Information Theory in 1948 in his landmark paper 'A Mathematical Theory of Communication' at Bell Labs, formalizing the bit as the fundamental atomic unit of information.

• Common events carry no news; rare events carry pure information.
• Surprisal is minus log P: I(x) = -log₂ P(x).

For a discrete random variable X ∈ x₁, ṡ, x_K with probability distribution P:

H(P) = E_(X ~ P)[I(X)] = -∑_(i=1)^K pᵢ log₂ pᵢ

Extremal bounds on Δ^(K-1):

1. Minimum Entropy (H = 0 bits): Occurs when one outcome has probability 1.0 (deterministic Dirac distribution):

H = -(1 log₂ 1 + 0 + ṡ + 0) = 0

1. Maximum Entropy (H = log₂ K bits): Occurs when all outcomes are equally likely (uniform distribution pᵢ = 1/K):

H = -∑_(i=1)^K 1/K log₂(1/K) = -log₂(1/K) = log₂ K

By Shannon's Source Coding Theorem, H(P) represents the absolute theoretical lower bound on the expected code length per symbol for lossless compression.

In LLMs, output token entropy monitors generation confidence. When entropy is low (H < 0.5 bits), the model is certain of the next word (e.g. 'United States of [America]'); when entropy is high (H > 4.0 bits), the story has reached an open-ended narrative branch.

Shannon coined the term 'entropy' in 1948 upon the advice of John von Neumann, who noted: 'You should call it entropy... nobody knows what entropy really is, so in a debate you will always have the advantage.'

• Entropy is the average surprisal: expected number of yes/no questions.
• Dirac delta has 0 entropy; uniform distribution has log₂(K) maximum entropy.

A loss curve mathematical diagram on natural warm linen (#f5f4e9) with a slate border.

Top section: Title in bold sans-serif: "Categorical Cross-Entropy: The Surprisal Penalty". Subtitle: "L_(CE) = -ln(p_(target)) heavily penalizes overconfident incorrect predictions."

Middle section: A clean 2D coordinate plot of the negative logarithm function y = -ln(p) over probability domain p ∈ (0, 1].

Mark three points along the curve with color-coded callouts:

1. p = 0.99 ⟹ Loss = -ln(0.99) ≈ 0.010 (Near zero penalty for confident correct prediction).
2. p = 0.50 ⟹ Loss = -ln(0.50) ≈ 0.693 (Moderate penalty for uncertain prediction).
3. p = 0.01 ⟹ Loss = -ln(0.01) ≈ 4.605 (Severe penalty; curve shoots upward asymptotically toward infinity as p -> 0).

Bottom section: Formula card: "L_(batch) = -(1) / (B · T) ∑_(i=1)^B ∑_(t=1)^T ln(p_i, t, y_(i, t)) ⟹ Perplexity = exp(L_(CE))".

Generous whitespace, smooth logarithmic curve, clear annotations.

This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.

"Categorical Cross-Entropy: The Surprisal Penalty: Invariance and structure determine stability under spatial and numerical transformations."

An educational derivation card on natural warm linen (#f5f4e9) with a 1px slate frame.

Top section: Title in bold sans-serif: "From Fisher's MLE to Shannon's Cross-Entropy". Subtitle: "Step-by-step mathematical bridge showing why maximizing likelihood minimizes cross-entropy."

Middle section: A 5-step algebraic equivalence chain:

1. Maximum Likelihood Estimation: θ^* = argmax_θ ∏_(t=1)^N P_θ(w_t | w_(<t)).
2. Monotonic Log Transformation: θ^* = argmax_θ ∑_(t=1)^N ln P_θ(w_t | w_(<t)).
3. Multiply by -1/N to turn maximization into minimization:

θ^* = argmin_θ ( -1/N ∑_(t=1)^N ln P_θ(w_t | w_(<t)) ) = argmin L_(NLL).

1. Express target token as one-hot empirical distribution q:

L = -∑_(k=1)^V qₖ ln p_(θ, k) = H(q, p_θ) (Shannon's Cross-Entropy).

1. KL Divergence Identity: H(q, p_θ) = H(q) + D_(KL)(q ∥ p_θ) ⟹ Minimizing CE minimizes KL divergence!

Bottom section: Card: "MLE, Empirical Risk Minimization under Cross-Entropy, and KL Divergence Minimization are mathematically identical problems."

Clear step-by-step numbering, ample breathing space, crisp typography.

This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.

"From Fisher's MLE to Shannon's Cross-Entropy: Invariance and structure determine stability under spatial and numerical transformations."

A loss function geometric curve diagram in square 1:1 format on warm cream parchment (#fcf0db) with a sepia frame.

Top section: Title in navy serif: "Categorical Cross-Entropy: The Surprisal Penalty". Subtitle: "L = -log(p_(target)) penalizes overconfident wrong predictions asymptotically."

Middle section: A clean 2D coordinate plot of the negative log curve:

X-axis: Predicted probability of the correct target token p ∈ (0, 1].

Y-axis: Cross-entropy loss value L ∈ [0, ∞).

Show key points on the curve:

• When p = 1.0 (Certain Truth): Loss = -log(1.0) = 0.0 (Zero penalty).
• When p = 0.5 (Uncertain): Loss = -log(0.5) = 0.693.
• When p = 0.01 (Near Blind): Loss = -log(0.01) = 4.605.
• When p -> 0 (Confident Error): Loss -> ∞ (Infinite explosive penalty).

Bottom section: Gradient behavior: "(∂ L) / (∂ zᵢ) = pᵢ - yᵢ. The gradient is simply the prediction error (predicted probability minus one-hot ground truth)."

Pristine mathematical curve, highlighted tangent slopes, clear coordinate labels.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI ↗

"Categorical Cross-Entropy: The Surprisal Penalty Function: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A mathematical equivalence derivation card in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.

Top section: Title in navy serif: "From MLE to Cross-Entropy: Exact Mathematical Equivalence". Subtitle: "Why maximizing joint data likelihood is identical to minimizing cross-entropy loss."

Middle section: Step-by-step algebraic bridge across 4 numbered panels:

1. Joint Likelihood: Under independent tokens, likelihood is the product of probabilities:

L(θ) = ∏_(t=1)^N P(x_t | x_(<t); θ).

1. Log-Likelihood: Taking the natural logarithm converts product into a sum:

log L(θ) = ∑_(t=1)^N log P(x_t | x_(<t); θ).

1. Negative Log-Likelihood (NLL): Invert the sign to convert maximization into minimization:

NLL(θ) = -∑_(t=1)^N log P(x_t | x_(<t); θ).

1. Scale by 1/N: Dividing by total tokens N yields the empirical cross-entropy loss:

L(θ) = -1/N ∑_(t=1)^N log p_t = H(P_(data), P_θ).

Bottom section: Unification axiom: "argmax_θ ∏ P(x_t) ≡ argmin_θ -∑ log P(x_t). Statistics (Fisher) and Information Theory (Shannon) unite in modern LLM training."

Elegant typography, clean numbered steps, balanced square layout.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI ↗

"From Fisher's Maximum Likelihood Estimation to Shannon's Cross-Entropy: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."