Stochastic Gradient Descent (SGD): Hyperparameter Dynamics & Mini-Batching
Master mini-batch SGD optimization dynamics, gradient covariance noise, learning rate scaling laws, and high-dimensional parameter trajectories.
How this lesson unfolds.
Full-batch gradient descent evaluates the entire dataset before taking a single parameter step—an impossible feat on modern datasets with millions of samples. Stochastic Gradient Descent introduces controlled stochasticity by computing gradients on small mini-batches, fundamentally transforming optimization dynamics.
How do learning rate alpha and mini-batch size B jointly govern the diffusion and convergence speed of Stochastic Gradient Descent?
Imagine trying to find the lowest spot in a foggy valley using a flashlight. Full batch gradient descent takes 10 hours to measure every rock with a laser before taking one careful millimeter step. Mini-batch SGD takes a quick glance with a flashlight every second and takes a brisk step. The quick steps have a little wobble (gradient noise), but that wobble helps you kick your way out of small potholes that would trap a slow, cautious walker.
- $$\mathbf{w}_{t+1} = \mathbf{w}_t - \alpha \mathbf{g}_\mathcal{B}, \quad T_{eff} \propto \frac{\alpha}{B}$$
- Learning goalDerive the SGD stochastic noise covariance and apply the linear scaling rule to maintain convergence when increasing batch size.
- Helpful foundationsFull batch vs mini-batch gradient descent · Expected value and covariance matrices · Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (Goyal et al., 2017)
- Common trapBelieving larger batch sizes always train better; excessively large batch sizes eliminate gradient noise, causing networks to settle into sharp, non-generalizing minima.
- Next connectionTransitions to Momentum (Chapter 24) to accelerate SGD through narrow ravines.
More context & mechanics
The mini-batch SGD parameter update is:
w_(t+1) = w_t - α g_B(w_t), g_B(w_t) = 1/B ∑_(i=1)^B ∇ Lᵢ(w_t)
Decomposing the mini-batch gradient into true gradient plus stochastic noise:
g_B(w_t) = ∇ L_(full)(w_t) + ε_t, E[ε_t] = 0
The noise covariance matrix scales inversely with batch size B:
Cov(ε_t) = 1/B Σ(w_t) ⟹ Effective Temperature T_(eff) ≈ (α) / (2B)
This establishes the Linear Scaling Rule: when scaling batch size B → k B, learning rate must scale proportionally α → k α to preserve diffusion dynamics.
Suppose baseline training uses batch size B = 32 and learning rate α = 0.01.
- Ratio: (α) / (B) = (0.01) / (32) = 3.125 × 10^(-4).
- Scaling to Distributed Cluster: Batch size increases to B' = 256 (8× increase).
- Linear Scaling Update: To preserve optimal exploration noise:
α' = 8 × 0.01 = 0.08
- Verification: (α') / (B') = (0.08) / (256) = 3.125 × 10^(-4) (Identical diffusion temperature!).
Standard baseline optimizer used to train ResNet vision models on ImageNet and fine-tune large foundation models where SGD often generalizes better than adaptive optimizers.
Introduced by Herbert Robbins and Sutton Monro (1951) in 'A Stochastic Approximation Method'; scaled to deep networks by LeCun, Bottou, and Goyal et al. (2017).
Noise is not the enemy; noise is exploration: keep the ratio of alpha over batch size constant.
Why does vanilla gradient descent waste computational iterations bouncing between steep canyon walls instead of traveling down the valley?
Imagine trying to roll a marble down a long gutter. You want the marble to roll 50 feet forward down the gentle length of the pipe. But the sides of the pipe are curved and steep. Because the slope up the sides is 100 times steeper than the forward slope, gradient descent gets terrified of the side slope and throws all its update energy into jumping back and forth across the walls: left, right, left, right. It takes 10,000 violent bounces just to drift 1 foot forward.
- Learning goalAnalyze the coordinate-wise convergence dynamics of gradient descent in anisotropic quadratic potentials and trace oscillation trajectories.
- Helpful foundationsUnivariate recurrence relations · Condition number · Optimization for Machine Learning (Sra, Nowozin, Wright, 2011)
- Common trapAssuming more training epochs can easily overcome the oscillation trap; without momentum, convergence time scales with the condition number, requiring millions of redundant forward-backward passes.
- Next connectionWhat is the exact mathematical speed limit before gradient descent explodes out of the canyon entirely? We derive the stability bound.
More context & mechanics
Consider the 2D anisotropic quadratic bowl:
L(w₁, w₂) = 1/2 λ₁ w₁² + 1/2 λ₂ w₂², λ₁ ≫ λ₂
The gradient is:
∇ L = [λ₁ w₁, λ₂ w₂]^T
Gradient descent updates coordinates independently:
w₁^((t+1)) = (1 - η λ₁) w₁^((t))
w₂^((t+1)) = (1 - η λ₂) w₂^((t))
Because λ₁ ≫ λ₂, we must set η < 2 / λ₁ to avoid divergence along w₁.
Under this bounded η:
1 - η λ₂ = 1 - (λ₂) / (λ₁) ≈ 1 - ε ≈ 0.9999
While w₁ rapidly bounces sign (1 - η λ₁ ≈ -0.9), w₂ barely changes by 0.01\% per step. Over 99\% of total FLOPs are wasted in futile perpendicular bounces.
This fatal oscillation is why vanilla SGD without momentum or adaptive learning rates is virtually never used to train modern deep learning models.
Augustin-Louis Cauchy invented Gradient Descent in 1847 to solve astronomical orbits. Even Cauchy noted its sluggish convergence in elongated valley geometries.
- Steep walls steal the gradient; the valley floor starves.
- High condition number causes ping-pong paralysis.
Why does the maximum eigenvalue of the Hessian impose a strict mathematical upper bound on the learning rate?
Imagine adjusting a bathroom shower knob that is insanely sensitive: a quarter-inch turn changes the water from freezing cold to scalding hot. If you react aggressively and turn the knob by a whole inch every time the water feels cold, you will violently overshoot: freezing → scalding → hypothermia → third-degree burns → explosion. To keep the shower stable, your knob adjustment speed must be smaller than the shower's sensitivity. The stability bound η < 2 / λ_(max) is that exact mathematical speed limit.
- $$Prove via spectral radius analysis of linear dynamical systems that the update operator (I - eta * H) diverges if eta >= 2 / lambda_max.$$
- Learning goalDerive the critical learning rate stability threshold eta < 2 / lambda_max using contraction mapping analysis.
- Helpful foundationsDiscrete dynamical systems · Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (Cohen et al., 2021)
- Common trapBelieving a larger learning rate always accelerates learning; exceeding 2/lambda_max causes instant divergence and NaN loss crashes.
- Next connectionWhy do high-dimensional loss landscapes avoid local minimum traps? We revisit Morse theory and saddle points from an optimization perspective.
More context & mechanics
Consider the recurrence relation w_(t+1) = (I - η H) w_t for quadratic error surface L(w) = 1/2 w^T H w.
In the eigenvector coordinate basis uᵢ:
uᵢ^((t+1)) = (1 - η λᵢ) uᵢ^((t))
For the dynamic system to be asymptotically stable (\|w_t\| → 0 as t → ∞), the contraction multiplier must have magnitude strictly less than 1:
|1 - η λᵢ| < 1, ∀ i ∈ 1, ṡ, d
Solving the double inequality:
-1 < 1 - η λᵢ < 1
-2 < -η λᵢ < 0 ⟹ 0 < η < (2) / (λᵢ)
Since this must hold for ALL eigenvalues, it is bounded by the most restrictive (largest) eigenvalue λ_(max):
η_(crit) = (2) / (λ_(max))
If η > (2) / (λ_(max)), updates along eigenvector v_(max) multiply by a factor <-1, triggering exponential oscillatory divergence to ± ∞.
In deep learning theory, this boundary defines the 'Edge of Stability' (Cohen et al., 2021), where training automatically self-stabilizes near the 2 / λ_(max) threshold.
Aleksandr Lyapunov established stability theory for dynamical systems in 1892. Jeremy Cohen et al. (ICLR 2021) demonstrated that deep neural networks operate at the edge of stability throughout training.
- Two over lambda-max is the hard speed limit.
- Step past 2/lambda_max, and the shower explodes into NaN.
How does an end-to-end neural network learn to twist non-linear decision boundaries around the intertwined 3-class spiral dataset?
Imagine three snakes of different colors (red, green, blue) coiled tightly around each other in an interlocking spiral. No straight ruler cut can separate the snakes without chopping through all three bodies! When your 2-layer neural network starts training at epoch 0, its decision boundaries are flat straight lines: its accuracy is 33% (pure random guessing). But as backpropagation updates the weights over 10,000 steps, the hidden neurons fold and bend the rubber coordinates: the boundaries wrap around the snakes like three curving tentacles, reaching 99% accuracy.
- Learning goalTrain a multi-layer neural network from scratch on the 3-class spiral dataset and evaluate loss convergence and decision boundary curvature.
- Helpful foundationsComplete forward-backward graph · Learning rate decay · Neural Networks from Scratch in Python (Kinsley & Kukiela, 2020)
- Common trapAssuming a single hidden layer of 3 neurons can solve the spiral; the spiral's topological winding requires sufficient width (e.g. 64 neurons) to crease space into multi-turn spiral regions.
- Next connectionWe have now established the complete foundation of deep learning from scratch, from single neurons to high-dimensional transformer architectures.
More context & mechanics
The spiral dataset consists of 3 classes swirling outward in 2D space:
X ∈ R^(300 × 2), y ∈ 0, 1, 2^(300)
At epoch 0:
- Initial cross-entropy loss: -ln(1/3) = ln(3) ≈ 1.0986 nats.
- Initial accuracy: ≈ 33.3\% (random guessing).
Training trajectory over 10,000 epochs with learning rate η = 1.0 and learning rate decay:
- Epoch 1,000: Loss drops to ≈ 0.75, accuracy climbs to 60\%. Boundaries bend into soft wedges.
- Epoch 5,000: Loss drops to ≈ 0.35, accuracy climbs to 88\%. The network discovers the spiral arms.
- Epoch 10,000: Loss reaches ≈ 0.15, accuracy exceeds 98\%. The three decision regions form interlocking spiral tentacles that cleanly separate the intertwined points.
This demonstrates the Universal Approximation Theorem in physical, visual reality: two hidden dense layers with ReLU space folding can learn arbitrary non-linear topologies.
The spiral dataset is the definitive benchmark used in Stanford's CS231n (Andrej Karpathy) and Harrison Kinsley's NNFS to prove that a scratch-built neural network autograd engine functions correctly.
Kevin Lang and Michael Witbrock designed the Two-Spirals Problem in 1988 at Carnegie Mellon as a notoriously difficult benchmark for multi-layer perceptron training algorithms.
- Initial loss is ln(num_classes) = ln(3) = 1.098.
- Watch flat lines bend into spiral arms as loss drops to zero.