RMSProp: Exponential Moving Average Gradient Scaling & Leaky Memory
How does replacing AdaGrad's infinite sum with an Exponential Moving Average prevent learning rate suffocation, and how does leaky memory stabilize non-convex optimization?
How this lesson unfolds.
AdaGrad proved that per-parameter learning rate scaling is transformative, but its infinite accumulation permanently froze deep networks. RMSProp solved this limitation by introducing leaky memory through an Exponential Moving Average (EMA).
How does replacing AdaGrad's infinite sum with an Exponential Moving Average prevent learning rate suffocation, and how does leaky memory stabilize non-convex optimization?
Imagine replacing an infinite stone ledger with a whiteboard. AdaGrad etched every mistake in permanent stone, so after years of writing, the stone was completely full and unusable. RMSProp writes on a whiteboard: today's gradients are dark, yesterday's are faded, and gradients from last month are completely erased. The optimizer only cares about the terrain it is walking on right now.
- $$\mathbf{v}_{t+1} = \beta \mathbf{v}_t + (1 - \beta) \mathbf{g}_t^2, \quad \mathbf{w}_{t+1} = \mathbf{w}_t - \frac{\alpha}{\sqrt{\mathbf{v}_{t+1}} + \epsilon} \mathbf{g}_t$$
- $$RMSProp computes an Exponential Moving Average (EMA) of squared gradients v_t = beta * v_{t-1} + (1 - beta) * g_t^2, establishing a finite sliding window of memory that allows learning rates to increase when loss surfaces flatten.$$
- Learning goalDerive RMSProp's exponential moving average update equations and contrast its dynamic recovery mechanism against AdaGrad's monotonic decay.
- Helpful foundationsAdaGrad sum of squares formulation · Exponential moving average mathematics · Neural Networks for Machine Learning, Lecture 6e (Hinton, 2012)
- Common trapThinking RMSProp only decreases step sizes; if gradients become small on a plateau, the denominator shrinks and effective learning rate increases dynamically.
- Next connectionTransitions to Adam (Chapter 27), which unites RMSProp's adaptive scaling with Polyak's momentum.
More context & mechanics
RMSProp replaces AdaGrad's monotonic sum with an Exponential Moving Average (EMA) parameterized by decay factor β (typically 0.9 or 0.99):
v_(t+1) = β v_t + (1 - β) g_t²
w_(t+1) = w_t - (α) / (√(v_(t+1)) + ε) \odot g_t
Because v_(t+1) is a convex combination:
- Effective memory window spans only the most recent N ≈ (1) / (1 - β) steps (e.g. 10 steps for β = 0.9).
- If gradients shrink in a flat plateau, v_(t+1) decreases, automatically increasing effective learning rate to cross the plateau quickly!
Let β = 0.9, initial α = 0.01, and ε = 10^(-8).
- Steep Canyon Phase (g = 10.0 for 5 steps):
v ≈ 100.0 ⟹ α_(eff) = (0.01) / (√(100)) = 0.001 (Step size shrinks by 10× to prevent explosion).
- Flat Plateau Phase (g = 0.1 for subsequent steps):
After 10 steps on the plateau, the EMA washes away the ancient canyon gradients:
v ≈ 0.01 ⟹ α_(eff) = (0.01) / (√(0.01)) = 0.100 (Step size increases by 100× to speed across the flat plain!).
AdaGrad would have remained frozen at 0.001 forever.
Primary optimizer for Recurrent Neural Networks (LSTMs, GRUs) and foundational building block of the Adam optimizer.
Proposed by Geoffrey Hinton in 2012 in Lecture 6e of his Coursera course 'Neural Networks for Machine Learning'. Never published in an academic paper, yet cited over 20,000 times!
Fade the past, adapt to the present: leaky memory lets dead optimizers breathe again.
How does an Exponential Moving Average (EMA) of squared gradients cure AdaGrad's monotonic freezing?
Instead of keeping every receipt you have ever received since kindergarten in a growing warehouse (AdaGrad), you set up a leaky bucket with a small drain hole at the bottom. When you pour fresh water in (recent gradients), the bucket reflects recent rainfall. But older water steadily leaks out over time. If you hit a dry spell, the water level drops; if you hit a storm, the water level rises. The bucket always reflects current weather conditions, never overflowing to infinity.
- $$Replace the unbounded sum with a leaky memory s_t = beta * s_{t-1} + (1 - beta) * g_t^2 to restrict variance tracking to an effective window of 1 / (1 - beta) steps.$$
- Learning goalFormulate the RMSProp leaky memory recurrence and calculate its effective historical time window T_eff = 1 / (1 - beta).
- Helpful foundationsExponential moving averages · Geometric series · Neural Networks for Machine Learning - Lecture 6e (Hinton, 2012)
- Common trapBelieving RMSProp completely eliminates learning rate decay; RMSProp eliminates artificial premature freezing, but standard scheduled decay (e.g. cosine decay) is still applied intentionally.
- Next connectionWhat does dividing by root-mean-square gradients do geometrically to the shape of loss ravines? We look at isotropic sphere mapping.
More context & mechanics
RMSProp replaces the cumulative sum with an Exponential Moving Average (EMA):
s_t = β s_(t-1) + (1 - β) g_t²
Where β ∈ (0, 1), typically chosen as β = 0.99.
Expanding s_t as a geometric power series:
s_t = (1 - β) ∑_(k=0)^t β^k g_(t-k)²
The effective temporal memory window is:
T_(eff) = ∑_(k=0)^∞ β^k = (1) / (1 - β)
For β = 0.99, T_(eff) = (1) / (1 - 0.99) = 100 steps.
The parameter update is:
w_(t+1) = w_t - (η) / (√(s_t) + ε) \odot g_t
Because s_t is a weighted average, it stays bounded by the recent gradient magnitudes:
E[s_t] ≈ E[g²]
The effective learning rate never freezes to zero, allowing continuous adaptation across non-stationary loss surfaces.
RMSProp is built into all deep learning frameworks (`torch.optim.RMSprop`) and powers reinforcement learning (DQN in Atari games) and recurrent neural networks.
Geoffrey Hinton proposed RMSProp in 2012 in Lecture 6e of his Coursera MOOC. Despite never being formally published in an academic conference, it became one of the most cited optimization ideas in AI.
- Leaky bucket forgets the past; tracks the recent storm.
- Effective window is 1 / (1 - beta).
How does diagonal second-moment preconditioning warp elongated elliptical loss contours into isotropic circular bowls?
Imagine looking at an elliptical running track that is 100 meters long but only 1 meter wide: a squashed 100:1 pancake. Running in it is maddening because you constantly bounce off the rails. Now imagine putting on magic distortion glasses that magnify the narrow width by 100× while leaving the length unchanged. Suddenly, the squashed 100:1 pancake looks like a perfect 1:1 round circle! In this round circular room, gradient descent steps point directly straight toward the center bullseye from any angle. RMSProp's division by √(sᵢ) is those exact distortion glasses.
- $$Show that dividing each coordinate by sqrt(s_i) approximates diagonal Hessian preconditioning H^(-1/2), transforming eccentricity kappa >> 1 into kappa_eff ~ 1.$$
- Learning goalProve that coordinate-wise second-moment division neutralizes extreme curvature disparities, reducing effective condition numbers toward unity.
- Helpful foundationsHessian condition number · Diagonal preconditioning · Quasi-Newton Methods for Large-Scale Optimization (Nocedal, 1992)
- Common trapBelieving RMSProp handles off-diagonal Hessian cross-correlations; it only rescales along axis-aligned coordinates (diagonal approximation), which is why coordinate rotation matters.
- Next connectionWhat happens when you marry Polyak momentum (the bowling ball) with RMSProp (the adaptive speedometer)? You create the legendary Adam engine.
More context & mechanics
Consider an ill-conditioned quadratic bowl:
L(w₁, w₂) = 1/2 λ₁ w₁² + 1/2 λ₂ w₂², λ₁ = 10000, λ₂ = 1
The contour eccentricity is \kappa = 10000:1.
The gradient components are g₁ = λ₁ w₁ and g₂ = λ₂ w₂.
The steady-state RMSProp variance estimates are:
s₁ ≈ g₁² = λ₁² w₁², s₂ ≈ g₂² = λ₂² w₂²
Taking the square root:
√(s₁) ≈ λ₁ |w₁|, √(s₂) ≈ λ₂ |w₂|
The normalized update step is:
Δ w₁ = -(η) / (√(s₁)) g₁ = -η (λ₁ w₁) / (λ₁ |w₁|) = -η sgn(w₁)
Δ w₂ = -(η) / (√(s₂)) g₂ = -η (λ₂ w₂) / (λ₂ |w₂|) = -η sgn(w₂)
Both coordinates take steps of identical magnitude η regardless of how steep or flat their individual curvatures are!
This effectively pre-conditions the coordinate space:
\kappa_(effective) ≈ 1
Transforming treacherous ravines into easily traversable isotropic spheres.
This diagonal preconditioning is the fundamental engine that allows modern optimizers to train 100-layer Transformers without manual per-layer learning rate tuning.
Quasi-Newton methods (BFGS, DFP) pioneered curvature preconditioning in the 1960s. RMSProp provided a diagonal O(d) approximation capable of scaling to billions of parameters.
- Divide by sigma to turn stretched ellipses into round circles.
- Preconditioning turns a 10,000:1 ravine into a 1:1 isotropic bullseye.