Gradients and Derivatives: The Chain Rule Engine
How does the chain rule decompose multi-layer neural network credit assignment into local gear ratios?
How this lesson unfolds.
Explore how backpropagation passes error gradients through deep composite functions by multiplying local rates of change, exactly like bicycle gears transmitting rotation.
How does the chain rule allow each neural network layer to compute its gradient using only local derivatives and upstream feedback?
Picture a bicycle gear train. If your pedal gear A drives intermediate gear B twice as fast (gear ratio dB/dA = 2), and gear B drives wheel C three times as fast (gear ratio dC/dB = 3), how fast does the wheel rotate relative to your pedals? You simply multiply the gear ratios: dC/dA = dC/dB · dB/dA = 3 × 2 = 6. The chain rule in neural networks is identical: when loss error flows backward through layers, each layer does not need to know the global architecture; it simply multiplies the incoming upstream gradient by its own local gear ratio.
- Learning goalDerive the scalar and multivariate chain rule, formulate the product of local Jacobians, and verify credit assignment through composed differentiable layers.
- Helpful foundationsCalculus: Early Transcendentals (Stewart, 8th ed, Cengage) · Backpropagation and the Chain Rule in Matrix Operations (Rumelhart et al., 1986) · Automatic Differentiation in Machine Learning: A Survey (Baydin et al., JMLR 2018)
- Common trapAssuming backpropagation requires computing one gigantic symbolic derivative formula for the entire deep network. In reality, reverse-mode automatic differentiation decomposes composite functions into a computational graph of elementary operations; each node evaluates only its local derivative and multiplies it by incoming adjoint values.
- Next connectionSingle-neuron calculus derivations and matrix gradients (dW, db, dX) are detailed in Module 5 of this course.
More context & mechanics
Let y = f(u) and u = g(x) be differentiable functions. The composite function y = (f ∘ g)(x) has derivative with respect to x given by Leibniz's chain rule:
dy/dx = dy/du · du/dx
In a deep neural network consisting of L sequential transformations a^((l)) = f^((l))(z^((l))) and z^((l)) = W^((l)) a^((l-1)) + b^((l)), the loss gradient with respect to an early activation a^((0)) is the continuous product of local rates:
(∂ L) / (∂ a^((0))) = (∂ L) / (∂ a^((L))) · (∂ a^((L))) / (∂ z^((L))) · (∂ z^((L))) / (∂ a^((L-1))) … (∂ z^((1))) / (∂ a^((0)))
- Local Gradient Independence:
A layer at depth l only computes two quantities:
- The derivative of its output with respect to its parameters: (∂ z^((l))) / (∂ W^((l))) = a^((l-1)).
- The derivative of its output with respect to its input: (∂ z^((l))) / (∂ a^((l-1))) = W^((l)).
- Backward Adjoint Propagation:
Given the upstream gradient δ^((l)) = (∂ L) / (∂ z^((l))), parameter updates and downstream gradients follow immediately:
(∂ L) / (∂ W^((l))) = δ^((l)) (a^((l-1)))^T, δ^((l-1)) = (W^((l)))^T δ^((l)) \odot f'(z^((l-1)))
No layer ever needs to inspect the rest of the network; it acts as an autonomous local gear in a synchronized computational train.
Reverse-mode automatic differentiation (Autograd in PyTorch, JAX, TensorFlow) records forward operations on a Directed Acyclic Graph (DAG) and executes this exact chain rule backward during `loss.backward()`. This enables end-to-end training of models with billions of parameters across thousands of GPUs without manually deriving full-network gradients.
Gottfried Wilhelm Leibniz formalized the chain rule in 1676 during the invention of infinitesimal calculus. In 1974, Paul Werbos formulated reverse-mode automatic differentiation for neural networks in his Harvard doctoral dissertation, and in 1986 David Rumelhart, Geoffrey Hinton, and Ronald Williams popularized the method as 'Backpropagation', launching the modern deep learning paradigm.
- Golden Axiom: "Gears in sequence multiply speed: upstream error times local rate gives the gradient you need."
- Backprop is local: receive upstream gradient, multiply by local derivative, pass downstream.
How does the derivative quantify the ratio of output nudge to input wiggle in multi-variable functions?
Imagine standing in front of a mixing console with 50 slider knobs controlling room temperature. If you nudge Slider 1 upward by a tiny millimeter, the room temperature rises by 3 degrees. The partial derivative of temperature with respect to Slider 1 is simply the wiggle ratio: Δ Output / Δ Input = +3 / 1 = 3. If Slider 2 drops the temperature by 2 degrees for that same 1 millimeter nudge, its partial derivative is -2. A derivative is not abstract calculus: it is the sensitivity knob of a machine.
- Learning goalCompute partial derivatives of multi-variable linear and non-linear expressions and construct gradient vectors.
- Helpful foundationsLimits and continuous functions · Calculus (Spivak, 2008)
- Common trapAssuming the gradient points toward the minimum; the gradient always points in the direction of steepest INCREASE, which is why gradient descent must subtract the gradient (-eta * grad).
- Next connectionWhen calculations are chained sequentially through multiple layers, how do these individual wiggle ratios combine? We look at bicycle gears.
More context & mechanics
For a multi-variable function f(x₁, x₂, ṡ, xₙ):
(∂ f) / (∂ xᵢ) = lim_(h → 0) (f(x₁, ṡ, xᵢ + h, ṡ, xₙ) - f(x₁, ṡ, xᵢ, ṡ, xₙ)) / (h)
Key rules for deep learning:
- Power Rule: d/dx x^n = n x^(n-1)
- Sum Rule: (∂) / (∂ x) [f(x) + g(x)] = (∂ f) / (∂ x) + (∂ g) / (∂ x)
- Scaling Rule: (∂) / (∂ x) [c · f(x)] = c (∂ f) / (∂ x)
The gradient vector bundles all individual partial derivatives into a single directional compass pointing in the direction of steepest ascent:
∇ f = [ (∂ f) / (∂ x₁), (∂ f) / (∂ x₂), ṡ, (∂ f) / (∂ xₙ) ]^T
During the backward pass of model training, the gradient ∇_W L tells the optimizer exactly how many units the total loss will rise or fall if any specific weight in the network is nudged by one unit.
Gottfried Wilhelm Leibniz and Isaac Newton co-invented differential calculus in the late 17th century. Leibniz introduced the intuitive dy/dx notation representing infinitesimal quotient ratios.
- Derivative is the wiggle ratio: output change divided by input nudge.
- Freeze all other knobs; measure the response.
How does multiplying intermediate sensitivity ratios like interlocking bicycle gears transmit error gradients backward through deep layers?
Think of a three-gear bicycle transmission. If turning your pedals by 1 rotation spins the middle gear 3 times (du/dx = 3), and each rotation of the middle gear spins the back wheel 4 times (dy/du = 4), how many times does the back wheel turn for every pedal rotation? You simply multiply the gear ratios: 3 × 4 = 12 rotations (dy/dx = 12). Backpropagation is this exact gear-ratio multiplication: each neural layer computes its local gear ratio, and multiplying them transmits the loss signal all the way back to the input.
- $$Compute the derivative of composite functions by multiplying local Jacobians along the computational graph path: dy/dx = (dy/du) * (du/dx).$$
- Learning goalApply the chain rule across chained functional operations to compute exact parameter gradients in a computational graph.
- Helpful foundationsUnivariate differential calculus · Learning Representations by Back-Propagating Errors (Rumelhart, Hinton, Williams, 1986)
- Common trapThinking backpropagation requires re-computing the forward pass; intermediate activations from the forward pass are cached in memory specifically to evaluate local derivatives during the backward pass.
- Next connectionWhat happens when you multiply 100 gear ratios together if each gear is slightly smaller than 1? We investigate the compound interest danger of vanishing gradients.
More context & mechanics
For a composite function y = f(u) where u = g(x):
dy/dx = dy/du · du/dx
In tensor form, for vector-valued functions y = f(u) and u = g(x):
(∂ y) / (∂ x) = (∂ y) / (∂ u) · (∂ u) / (∂ x) = J_f · J_g
During backpropagation, we compute the gradient of a scalar loss L:
(∂ L) / (∂ x) = ( (∂ L) / (∂ u) ) · (∂ u) / (∂ x)
The upstream gradient (∂ L) / (∂ u) is multiplied by the local Jacobian (∂ u) / (∂ x) to generate the downstream gradient.
Every deep learning framework (PyTorch Autograd, JAX, TensorFlow) constructs a dynamic directed acyclic graph (DAG) during the forward pass, and traverses it in reverse during `.backward()` by executing chain rule products at each node.
Paul Werbos first applied the chain rule to train artificial neural networks in his 1974 Harvard Ph.D. dissertation. Rumelhart, Hinton, and Williams popularized backpropagation in Nature in 1986.
- Upstream gradient times local gradient equals downstream gradient.
- Interlocking gears multiply their gear ratios.
How does the tangent line represent the local linear best approximation of a differentiable non-linear function?
Imagine driving on a winding, curved racetrack at night. At any exact second, your car's headlights don't bend around the curve: they shine straight ahead in a perfectly straight beam along your immediate heading. If you drove off the track without turning, you would travel straight along that headlight beam. The tangent line is that headlight beam: it is the straight line that touches the curve at exactly one point, matching the curve's direction and speed at that precise instant.
- $$Define the tangent line y - f(x_0) = f'(x_0)(x - x_0) as the limit of secant lines as delta x -> 0, formalizing local instantaneous rate of change.$$
- Learning goalDerive the tangent line equation for arbitrary differentiable curves and compute local first-order Taylor approximations.
- Helpful foundationsHigh-school point-slope line equation · Limits · Calculus: Early Transcendentals (Stewart, 2015)
- Common trapBelieving a tangent line can only touch a curve at one point globally; a tangent line matches local slope at x₀, but can intersect the curve elsewhere at distant coordinates.
- Next connectionHow does backpropagation compute derivatives for weights and biases in dense layers? We examine the dense layer backward pass.
More context & mechanics
Let f(x) be a smooth differentiable curve.
- Secant Line Slope: Given point x₀ and a step Δ x:
m_(secant) = (f(x₀ + Δ x) - f(x₀)) / (Δ x)
- Tangent Line Slope (The Derivative):
Take the limit as Δ x → 0:
f'(x₀) = lim_(Δ x → 0) (f(x₀ + Δ x) - f(x₀)) / (Δ x)
- Point-Slope Tangent Equation:
The linear first-order Taylor approximation near x₀ is:
y(x) = f(x₀) + f'(x₀)(x - x₀)
In multi-dimensional space, the tangent line generalizes to the tangent hyperplane:
y(x) = f(x₀) + ∇ f(x₀)^T (x - x₀)
Where the gradient vector ∇ f(x₀) defines the normal vector tilt of the tangent plane.
Every step of gradient descent moves parameters along this local tangent hyperplane approximation: x_(t+1) = x_t - η ∇ f(x_t).
Pierre de Fermat discovered the method of tangents around 1636, predating Newton and Leibniz's formal calculus by decades.
- The tangent line is the headlight beam of a curved road.
- Limit of secant lines as delta x shrinks to zero.