Backpropagation Through Activations: ReLU, LeakyReLU & Sigmoid Derivatives
How do non-linear activation functions modulate backward error flow, and why does ReLU masking preserve gradient highway conductivity while Sigmoid triggers exponential vanishing?
How this lesson unfolds.
Forward propagation transforms linear affine combinations via non-linear activation functions. During backpropagation, the chain rule requires multiplying incoming gradients by the local derivative of each activation. Understanding how activations act as dynamic gradient gates is the key to mastering deep network optimization.
How does the Rectified Linear Unit (ReLU) derivative act as an on/off gate for backpropagating error signals, and why do permanently negative inputs kill neurons?
Think of a one-way security turnstile. During the morning rush (forward pass), people with valid positive tickets walk through freely, while anyone with negative balance is stopped. In the evening (backward pass), feedback only flows back to the people who were allowed through. Those stopped at the gate receive zero feedback.
- $$\frac{\partial \mathcal{L}}{\partial \mathbf{z}} = \frac{\partial \mathcal{L}}{\partial \mathbf{a}} \odot \mathbb{I}(\mathbf{z} > 0)$$
- $$Because the derivative of ReLU is 1 for positive inputs and 0 for negative inputs, backpropagating through ReLU simply requires elementwise multiplication with a boolean mask of the forward activations: dL/dz = dL/da * (z > 0).$$
- Learning goalDerive the backward pass for ReLU, implement boolean gradient masking in Python/NumPy, and explain the dying ReLU pathology.
- Helpful foundationsForward ReLU activation definition · Single neuron chain rule calculus · Rectified Linear Units Improve Restricted Boltzmann Machines (Nair & Hinton, 2010)
- Common trapThinking backprop through ReLU requires computing a dense Jacobian matrix; because activation functions are strictly elementwise, the Jacobian is diagonal, collapsing into a simple Hadamard product.
- Next connectionExamine how LeakyReLU cures the dying neuron problem in Vectorized Activation Derivatives (Screen 2).
More context & mechanics
The forward ReLU operation is aᵢ = max(0, zᵢ).
The derivative g'(zᵢ) with respect to pre-activation zᵢ is a piecewise step function:
g'(zᵢ) = 1 & if zᵢ > 0 \ 0 & if zᵢ ≤ 0
Applying the multivariable chain rule for loss L:
(∂ L) / (∂ zᵢ) = (∂ L) / (∂ aᵢ) · g'(zᵢ) = (∂ L) / (∂ aᵢ) \odot I(zᵢ > 0)
In vectorized tensor computation, this is computed with zero floating-point multiplication using a cached boolean bitmask.
Let incoming loss gradient (∂ L) / (∂ a) = [2.5, -4.0, 1.2, -0.8] and forward pre-activations z = [1.8, -0.5, 0.4, -2.1]:
- Compute Boolean Mask: m = z > 0 = [True, False, True, False] = [1, 0, 1, 0].
- Apply Hadamard Product:
(∂ L) / (∂ z) = [2.5 × 1, -4.0 × 0, 1.2 × 1, -0.8 × 0] = [2.5, 0.0, 1.2, 0.0]
Notice: indices 1 and 3 receive exactly zero gradient. If neuron 1's bias drops so low that z is negative across all dataset samples, it becomes a dead neuron and never learns again.
Standard backward kernel in PyTorch autograd engine (`torch.relu_backward`), optimized with AVX-512 and CUDA bitmask operations.
Popularized by Nair & Hinton (2010) and Alex Krizhevsky in AlexNet (2012), proving that non-saturating gradients enable 6x faster training than Sigmoids.
Positive allows full gradient through; negative clamps to absolute zero: forward sign is backward shield.
How does LeakyReLU preserve gradient highway conductivity for negative values, and why does Sigmoid's maximum derivative of 0.25 cause deep networks to stall?
Imagine sending water through a series of pipes. LeakyReLU keeps a small leak (0.01) open even when the valve is shut, so water never stagnates completely. Sigmoid is like a muddy sponge that absorbs 75% of the water at every single bend: after passing through just 5 bends, only 0.1% of the water emerges!
- $$\sigma'(z) \le 0.25, \quad g'_{LeakyReLU}(z) \ge \alpha > 0$$
- Learning goalContrast the mathematical gradient flow of LeakyReLU versus Sigmoid and prove why LeakyReLU prevents vanishing gradients in deep architectures.
- Helpful foundationsBackward ReLU boolean masking · Exponential vanishing gradient calculus · Rectifier Nonlinearities Improve Neural Network Acoustic Models (Maas et al., 2013)
- Common trapAssuming Sigmoid's derivative is large near the origin; even at its theoretical maximum (z=0), Sigmoid shrinks incoming gradients by a factor of 4, guaranteeing rapid vanishing in deep networks.
- Next connectionTransitions directly to Chapter 18 to backpropagate through Categorical Cross-Entropy Loss.
More context & mechanics
- LeakyReLU Derivative:
g_(LeakyReLU)'(z) = 1 & if z > 0 \ α & if z ≤ 0 (typically α = 0.01)
Gradient flow: (∂ L) / (∂ z) = (∂ L) / (∂ a) \odot (1.0 · (z > 0) + α · (z ≤ 0)). No neuron ever completely dies.
- Sigmoid Derivative:
σ'(z) = σ(z)(1 - σ(z))
Because σ(z) ∈ (0, 1), its maximum value occurs at z = 0: σ'(0) = 0.5(1 - 0.5) = 0.25.
Across L layers, gradient magnitude diminishes exponentially:
\|∇_(W₁) L\| ≤ (0.25)^L \|∇_(a_L) L\|
Consider a 4-layer network where each layer has an incoming gradient norm of 1.0:
- Under Sigmoid:
- Layer 4: 1.0 × 0.25 = 0.25
- Layer 3: 0.25 × 0.25 = 0.0625
- Layer 2: 0.0625 × 0.25 = 0.0156
- Layer 1: 0.0156 × 0.25 = 0.0039 (Grades dropped by 99.6%! Layer 1 cannot learn).
- Under LeakyReLU (with 50% negative activations and α = 0.01):
- Average layer scaling factor: 0.5(1.0) + 0.5(0.01) = 0.505. Gradients remain over 100× stronger than Sigmoid.
LeakyReLU and PReLU are standard in GAN discriminators and deep regression networks where dead neurons must be strictly avoided.
Proposed by Andrew Maas, Awni Hannun, and Andrew Ng (Stanford, 2013) in 'Rectifier Nonlinearities Improve Neural Network Acoustic Models'.
Sigmoid divides by 4 at every wall; LeakyReLU leaves a crack in the door: small slope beats zero gradient.
How does the Jacobian matrix of softmax balance self-reinforcing diagonal growth against mutual cross-suppression?
Imagine a room of politicians running for class president. If Candidate A gains 10 new supporters, Candidate A's popularity rises (+Δ q_A), but because there is only one fixed 100% electorate, that gain must come at the direct expense of Candidate B and Candidate C (-Δ q_B, -Δ q_C). The Softmax Jacobian matrix tracks this political tug-of-war: along the diagonal, nudging your own logit boosts your own probability (qᵢ(1 - qᵢ)); off the diagonal, nudging your logit crushes your rivals (-qᵢ qⱼ).
- $$Derive the Softmax Jacobian J_ij = q_i (delta_ij - q_j), showing diagonal terms grow elastically as q_i(1 - q_i) while off-diagonal terms suppress rivals as -q_i q_j.$$
- Learning goalDerive the complete Softmax Jacobian matrix J = diag(q) - q q^T and prove that row sums equal zero.
- Helpful foundationsMultivariate quotient rule · Kronecker delta · The Softmax Function and its Derivatives (Eli Bendersky, 2016)
- Common trapAssuming the Softmax Jacobian is a diagonal matrix like ReLU or Sigmoid; because softmax normalizes across all coordinates, every logit influences every probability, creating a dense K x K Jacobian.
- Next connectionWhen this dense Jacobian matrix multiplies the cross-entropy loss gradient, how does it cancel out? We re-verify the Miracle Cancellation Proof.
More context & mechanics
Let qᵢ = (e^(zᵢ)) / (∑ₖ e^(zₖ)). We compute the partial derivative (∂ qᵢ) / (∂ zⱼ) using the quotient rule:
(∂ qᵢ) / (∂ zⱼ) = ((∂ (e^(zᵢ))) / (∂ zⱼ) ( ∑ₖ e^(zₖ) ) - e^(zᵢ) (∂ ( ∑ₖ e^(zₖ) )) / (∂ zⱼ)) / (( ∑ₖ e^(zₖ) )²)
Two cases arise:
- Diagonal Case (i = j): (∂ (e^(zᵢ))) / (∂ zᵢ) = e^(zᵢ).
(∂ qᵢ) / (∂ zᵢ) = (e^(zᵢ) ∑ₖ e^(zₖ) - e^(zᵢ) e^(zᵢ)) / (( ∑ₖ e^(zₖ) )²) = (e^(zᵢ)) / (∑ₖ e^(zₖ)) - ( (e^(zᵢ)) / (∑ₖ e^(zₖ)) )² = qᵢ - qᵢ² = qᵢ (1 - qᵢ)
- Off-Diagonal Case (i ≠ j): (∂ (e^(zᵢ))) / (∂ zⱼ) = 0.
(∂ qᵢ) / (∂ zⱼ) = (0 - e^(zᵢ) e^(zⱼ)) / (( ∑ₖ e^(zₖ) )²) = -( (e^(zᵢ)) / (∑ₖ e^(zₖ)) ) ( (e^(zⱼ)) / (∑ₖ e^(zⱼ)) ) = -qᵢ qⱼ
Unifying both cases with the Kronecker delta δ_(ij):
J_(i, j) = (∂ qᵢ) / (∂ zⱼ) = qᵢ (δ_(ij) - qⱼ)
In matrix form:
J = diag(q) - q q^T
This Jacobian structure governs gradient backpropagation through attention weights and classification layers. Notice that J is a symmetric, rank-(K-1) matrix whose rows sum to zero: J 1 = q - q(q^T 1) = q - q(1) = 0.
Carl Gustav Jacob Jacobi formalized the Jacobian matrix of multi-variable transformations in 1841 in 'De determinantibus functionalibus'.
- Diagonal: qᵢ (1 - qᵢ). Off-diagonal: -qᵢ qⱼ.
- Self-boost on the diagonal; rival suppression off the diagonal.
How do weight and bias gradients evaluate during the backward pass of an object-oriented Layer_Dense class?
Imagine you run a factory where 32 workers (batch size 32) report errors. For the weights, each error depends on what raw material that specific worker was given: worker 1's mistake multiplies worker 1's input (X^T × Δ). But for the bias, the bias was added equally to all 32 workers without any multiplying factor! Therefore, to find how much the bias contributed to the total factory error, you simply add up the errors from all 32 workers directly: `np.sum(dvalues, axis=0)`. Bias gradients are pure sum.
- $$Derive dL/dW = X^T @ dvalues and dL/db = sum(dvalues, axis=0), proving that bias gradients are the direct column sum of incoming upstream gradients.$$
- Learning goalDerive and implement the backward pass of a dense neural layer in pure NumPy, evaluating dweights, dbiases, and dinputs.
- Helpful foundationsMatrix calculus · NumPy axis reductions · Matrix Calculus for Deep Learning (Parr & Howard, 2018)
- Common trapOmitting keepdims=True in np.sum(dvalues, axis=0); dropping keepdims turns dbiases into a 1D vector of shape (D_out,), which breaks shape consistency during optimizer updates.
- Next connectionHow does the loss layer begin this backward chain? We inspect the OOP cross-entropy loss backward pass.
More context & mechanics
In the forward pass of `Layer_Dense`:
Z = X W + b
Where X ∈ R^B × D_(in), W ∈ R^D_(in) × D_(out), and b ∈ R^1 × D_(out).
Given upstream gradient dvalues = (∂ L) / (∂ Z) ∈ R^B × D_(out):
- Weight Gradient ((∂ L) / (∂ W)):
By matrix calculus:
(∂ L) / (∂ W) = X^T × dvalues ∈ R^D_(in) × D_(out)
(D_(in) × B) × (B × D_(out)) ⟶ (D_(in) × D_(out))
- Bias Gradient ((∂ L) / (∂ b)):
Since b was broadcast across all B rows of the batch:
(∂ L) / (∂ b) = ∑_(i=1)^B dvaluesᵢ = np.sum(dvalues, axis=0, keepdims=True) ∈ R^1 × D_(out)
- Downstream Input Gradient ((∂ L) / (∂ X)):
To propagate back to previous layers:
(∂ L) / (∂ X) = dvalues × W^T ∈ R^B × D_(in)
This is the complete backward method of `Layer_Dense` in pure NumPy:
```python
def backward(self, dvalues):
self.dweights = np.dot(self.inputs.T, dvalues)
self.dbiases = np.sum(dvalues, axis=0, keepdims=True)
self.dinputs = np.dot(dvalues, self.weights.T)
```
Paul Werbos derived backpropagation through matrix layers in 1974. Rumelhart, Hinton, and Williams (1986) popularized this exact vector-matrix backward formulation.
- dweights is X.T dot dvalues.
- dbiases is sum dvalues down axis 0; dinputs is dvalues dot W.T.