Loss Function Backpropagation: Categorical Cross-Entropy Calculus
How do prediction errors transform into initial gradient vectors at the exit of a neural network during backpropagation?
How this lesson unfolds.
Backpropagation begins at the very end of the network: the loss function. The derivative of the loss with respect to model probabilities determines the magnitude and direction of the initial error signals sent backward through all earlier layers.
What is the mathematical derivative of categorical cross-entropy with respect to predicted class probabilities, and how is it batched across samples?
Imagine a penalty fine for being surprised. If the correct answer is 'Cat' (y = 1) and you assigned 99% probability (p = 0.99), your fine slope is small: -1 / 0.99 ≈ -1.01. But if you only assigned 1% probability (p = 0.01), your fine slope is enormous: -1 / 0.01 = -100! The less confident you were in the truth, the harder the loss function kicks you to correct your error.
- $$\frac{\partial \mathcal{L}}{\partial p_i} = -\frac{y_i}{p_i}, \quad \frac{\partial \mathcal{L}_{batch}}{\partial \mathbf{P}} = -\frac{1}{B} \frac{\mathbf{Y}}{\mathbf{P}}$$
- $$For true target distribution y and predicted probability p, the derivative of categorical cross-entropy L = -sum(y_i * log(p_i)) with respect to probability p_i is exactly -y_i / p_i, scaling inversely with confidence in the true class.$$
- Learning goalDerive the partial derivative of categorical cross-entropy with respect to class probabilities and formulate batch-averaged gradient injection.
- Helpful foundationsForward categorical cross-entropy formulation · Natural logarithm derivative calculus · A Mathematical Theory of Communication (Shannon, 1948)
- Common trapBelieving that non-target classes receive direct loss gradients; because yⱼ = 0, ∂ L / ∂ pⱼ = 0. (They only receive gradients after coupling through the softmax normalization denominator).
- Next connectionWitness the magic of combining Softmax and Cross-Entropy in Chapter 19, where derivatives collapse into p - y.
More context & mechanics
For a single sample with true one-hot target vector y ∈ 0, 1^C and predicted probability vector p ∈ (0, 1)^C, categorical cross-entropy is:
L = -∑_(c=1)^C y_c ln(p_c)
Differentiating with respect to the probability pᵢ of class i:
(∂ L) / (∂ pᵢ) = -(∂) / (∂ pᵢ)[yᵢ ln(pᵢ) + ∑_(j ≠ i) yⱼ ln(pⱼ)] = -yᵢ/pᵢ
For non-target classes where yⱼ = 0, (∂ L) / (∂ pⱼ) = 0.
Averaged across a batch of size B:
(∂ L_(batch)) / (∂ p_(b, i)) = -1/B (y_(b, i)) / (p_(b, i))
Consider a 3-class classification problem (Classes 0, 1, 2) where the true label is Class 1 (one-hot target y = [0, 1, 0]):
- Scenario A (High Confidence): Model predicts p = [0.05, 0.90, 0.05].
(∂ L) / (∂ p) = [0, -(1) / (0.90), 0] = [0.0, -1.11, 0.0]
- Scenario B (Low Confidence): Model predicts p = [0.45, 0.10, 0.45].
(∂ L) / (∂ p) = [0, -(1) / (0.10), 0] = [0.0, -10.0, 0.0]
The gradient magnitude is 9× stronger when the model is wrong, generating urgent corrective momentum.
Computed directly inside classification loss engines. In practice, this step is combined with the softmax layer to form the numerically unified Softmax-Cross-Entropy gradient.
Derived from Shannon's Information Theory (1948) and cross-entropy minimization principles formalized in machine learning in the late 1980s.
Derivative of negative log is negative reciprocal: -y / p punishes doubt with ferocious force.
Why does evaluating cross-entropy against one-hot supervised labels collapse the 50,000-term sum into a single negative log scalar?
Suppose your model outputs predictions across a massive vocabulary of 50,257 possible words. In theory, cross-entropy says you must calculate: -(y₁ log q₁ + y₂ log q₂ + ṡ + y_(50257) log q_(50257)). But in supervised text training, the correct ground truth answer is one single word (e.g. 'banana', token ID 9871). The target vector y is 50,256 zeros and one single 1 at index 9871. Every zero multiplies its term to zero! The entire 50,000-term summation collapses into one single number: -log(q_(9871)).
- Learning goalProve algebraically that cross-entropy loss against one-hot targets collapses to the negative log-probability of the true class.
- Helpful foundationsKronecker delta function · Cross-entropy loss · Pattern Recognition and Machine Learning (Bishop, 2006)
- Common trapBelieving non-target tokens don't affect the loss; while non-target yᵢ are zero, non-target logits zⱼ appear in the softmax denominator of q_c, so raising non-target logits directly inflates the loss.
- Next connectionHow does the Softmax Machine generate that vector q from raw logits? We examine the exponentiate-and-normalize pipeline.
More context & mechanics
Let y ∈ 0, 1^K be a one-hot ground truth label vector for target class c:
yᵢ = 1 & if i = c \ 0 & if i ≠ c = δ_(i, c)
The general cross-entropy loss formula is:
L(y, q) = -∑_(i=1)^K yᵢ ln qᵢ
Split the sum into target index c and non-target indices i ≠ c:
L(y, q) = -y_c ln q_c - ∑_(i ≠ c) yᵢ ln qᵢ = -1 · ln q_c - ∑_(i ≠ c) 0 · ln qᵢ = -ln q_c
Evaluating loss dynamics:
- If model assigns target q_c = 1.0 ⟹ L = -ln(1) = 0 (perfect loss!).
- If model assigns target q_c = 0.50 ⟹ L = -ln(0.50) ≈ 0.693.
- If model assigns target q_c → 0 ⟹ L = -ln(0^+) = +∞ (infinite penalty!).
This is why `torch.nn.CrossEntropyLoss` takes integer class indices `targets = torch.tensor([c])` rather than memory-heavy one-hot tensors. The CUDA kernel reads q_c directly via pointer lookup, executing the loss evaluation in O(1) time.
One-hot target encoding originates from the Kronecker delta notation formalized by Leopold Kronecker in the 19th century and adapted into modern logistic regression.
- All zeros vanish; only the target survives.
- Loss is simply minus log of the correct token's probability: -log(q_c).
How does multiplying the Softmax Jacobian by the Cross-Entropy derivative collapse the gradient into pure error subtraction?
Imagine an elaborate mechanical clock with 50 spinning gears, pulleys, and counterweights. You expect the final output to be an impossible mathematical equation. But when you turn the master crank, all 50 gears mesh into each other with perfect precision: gear 1 cancels gear 2, pulley 3 cancels lever 4, and the grandfather clock chimes a single, pristine note: 'Prediction minus Target' (q - y). That is the miracle cancellation of softmax and cross-entropy.
- $$Prove that J^T * grad_q L = (diag(q) - q q^T) * (-y / q) simplifies algebraically to q - y.$$
- Learning goalExecute the full algebraic derivation of grad_z L = q - y and explain the cancellation of intermediate denominators.
- Helpful foundationsSoftmax Jacobian derivation · Cross-entropy loss · Generalized Linear Models (Nelder & Wedderburn, 1972)
- Common trapBelieving q - y only works for one-hot labels; this proof holds identically for ANY soft label distribution y satisfying sum(y) = 1 (e.g. label smoothing, distillation).
- Next connectionWhat does this gradient q - y do geometrically on the probability simplex? We look at restoring spring forces.
More context & mechanics
Let predicted probabilities be q = softmax(z) and loss be cross-entropy L = -∑ₖ yₖ ln qₖ.
By the vector chain rule:
∇_(z) L = J^T ∇_(q) L
Recall:
∇_(q) L = [ -y₁/q₁, -y₂/q₂, ṡ, -y_K/q_K ]^T
And J_(k, i) = qᵢ (δ_(ki) - qₖ).
Now evaluate the i-th component of ∇_(z) L:
(∂ L) / (∂ zᵢ) = ∑_(k=1)^K (∂ L) / (∂ qₖ) (∂ qₖ) / (∂ zᵢ) = ∑_(k=1)^K ( -yₖ/qₖ ) qᵢ (δ_(ki) - qₖ)
Expand the summation:
(∂ L) / (∂ zᵢ) = -∑_(k=1)^K (yₖ qᵢ δ_(ki)) / (qₖ) + ∑_(k=1)^K (yₖ qᵢ qₖ) / (qₖ)
- In the first term, δ_(ki) = 0 for all k ≠ i. For k = i, qₖ cancels qᵢ:
-(yᵢ qᵢ) / (qᵢ) = -yᵢ
- In the second term, qₖ cancels directly in numerator and denominator:
qᵢ ∑_(k=1)^K yₖ
Because y is a valid probability distribution (one-hot or soft labels), ∑_(k=1)^K yₖ = 1.0:
qᵢ ∑_(k=1)^K yₖ = qᵢ (1.0) = qᵢ
Combine both terms:
(∂ L) / (∂ zᵢ) = -yᵢ + qᵢ = qᵢ - yᵢ ⟺ ∇_(z) L = q - y
This identity is the core engine of deep learning classification. It means computing gradients for 50,000 vocabulary logits requires zero matrix multiplications: just a single element-wise subtraction `grad = probs - targets`.
This result arises from exponential family duality. In 1972, John Nelder and Robert Wedderburn proved that for canonical link functions, the gradient of the log-likelihood is always the raw residual y - μ.
- Gradient is Prediction minus Truth: q - y.
- All dense Jacobian fractions cancel into pure subtraction.
How does the backward pass of categorical cross-entropy evaluate gradients using one-hot indexing arrays?
Imagine grading a test with 32 students. Each student guessed probabilities for 3 answers. You want to calculate the error gradient for each student. If student 1 picked the correct answer with probability 0.80, their error slope is -1 / 0.80 = -1.25. For the incorrect answers, the true label was 0, so their direct error slope is 0. You don't need to loop over all 50,000 dictionary words: you simply look up the single index where the student's true answer sits, write -1 / p_(true), and divide by 32.
- $$Derive dL/dy_pred = -y_true / y_pred and implement fast vector indexing dinputs[range(samples), y_true] = -1 / y_pred[range(samples), y_true] / samples.$$
- Learning goalImplement the backward pass of categorical cross-entropy loss in pure NumPy using vector indexing.
- Helpful foundationsDerivative of natural logarithm · Advanced NumPy integer array indexing · Neural Networks from Scratch in Python (Kinsley & Kukiela, 2020)
- Common trapForgetting to divide by the batch size N; without dividing by N, the loss gradient scales with the batch size, causing learning rates to blow up when batch size changes.
- Next connectionHow do all these modular forward and backward methods assemble into one complete computational graph? We review the complete graph.
More context & mechanics
For a single sample with target class c and predicted probability q_c:
L = -ln q_c ⟹ (∂ L) / (∂ qᵢ) = -1/q_c & if i = c \ 0 & if i ≠ c
Across a mini-batch of N samples, the total loss is the mean:
L = 1/N ∑_(k=1)^N Lₖ
Therefore, the gradient with respect to predicted probabilities is scaled by 1/N:
(∂ L) / (∂ q_(k, i)) = -(1) / (N · q_(k, c)) & if i = c \ 0 & if i ≠ c
Pure NumPy implementation supporting both categorical integer labels and one-hot matrices:
```python
class Loss_CategoricalCrossentropy:
def backward(self, dvalues, y_true):
samples = len(dvalues)
labels = len(dvalues[0])
# If labels are one-hot encoded, convert to discrete integers
if len(y_true.shape) == 2:
y_true = np.argmax(y_true, axis=1)
self.dinputs = dvalues.copy()
# Calculate gradient: -1 / (samples * q_true) for the correct class
self.dinputs = np.zeros_like(dvalues)
self.dinputs[range(samples), y_true] = -1.0 / dvalues[range(samples), y_true]
self.dinputs = self.dinputs / samples
```
This sparse indexing gradient is the exact entry point that kicks off the backward pass in modular neural network frameworks.
Categorical cross-entropy and its negative reciprocal derivative arise from the Fisher score of the multinomial likelihood function, formalized in modern statistics in the 1920s.
- Gradient of -log(p) is -1/p.
- Divide by batch size to compute the mean batch gradient.
How does Backpropagating Through Softmax and Categorical Cross-Entropy operate mathematically at scale?
Step-by-step mathematical cancellation:
- Learning goalMaster the algebraic and geometric derivation of Backpropagating Through Softmax and Categorical Cross-Entropy.
- Helpful foundationsFoundations of Neural Networks From Scratch · Coordinate Invariance & Geometric Projections · Goodfellow, Bengio, and Courville (2016), Deep Learning, MIT Press
- Common trapAssuming Backpropagating Through Softmax and Categorical Cross-Entropy is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
An analytical gradient simplification diagram in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.
Top section: Title in navy serif: "Softmax + Cross-Entropy: Elegant Gradient Collapse". Subtitle: "Why combining Softmax with Cross-Entropy simplifies derivative calculation to pᵢ - yᵢ."
Middle section: Step-by-step mathematical cancellation:
- Softmax derivative Jacobian has two cases:
(∂ pᵢ) / (∂ zⱼ) = pᵢ (1 - pᵢ) if i = j, -pᵢ pⱼ if i ≠ j.
- Cross-entropy loss: L = -∑ₖ yₖ log(pₖ).
- Apply multivariable chain rule to find (∂ L) / (∂ zᵢ):
"(∂ L) / (∂ zᵢ) = ∑ₖ (∂ L) / (∂ pₖ) (∂ pₖ) / (∂ zᵢ) = -∑ₖ yₖ/pₖ (∂ pₖ) / (∂ zᵢ)".
- Substitute both cases and simplify:
All complex cross-terms cancel completely!
"(∂ L) / (∂ zᵢ) = pᵢ - yᵢ".
Bottom section: Beautiful simplicity: "The gradient driving the entire neural network is simply the raw prediction error: Predicted Probability minus One-Hot Ground Truth."
Clean algebraic cancellation steps, prominent green result box, balanced square layout.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
Goodfellow, Bengio, and Courville (2016), Deep Learning, MIT Press
"Backpropagating Through Softmax and Categorical Cross-Entropy: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."