Adam Optimizer

The evolution of deep learning optimization addresses the ill-conditioned curvature of loss surfaces where the Hessian condition number \kappa = λ_(max) / λ_(min) ≫ 1:

1. Vanilla SGD: θ_(t+1) = θ_t - α g_t. Suffers from orthogonal ping-pong oscillations in ravines and stalls in flat plateaus.
2. Polyak Momentum: v_t = β₁ v_(t-1) + α g_t, θ_(t+1) = θ_t - v_t. Accumulates directional velocity, canceling high-frequency transverse noise and boosting valley-floor progress by an amplification factor of 1 / (1 - β₁) = 10× (for β₁ = 0.9).
3. RMSProp Leaky Memory: s_t = β₂ s_(t-1) + (1 - β₂) g_t², θ_(t+1) = θ_t - (α) / (√(s_t) + ε) \odot g_t. Rescales anisotropic coordinate gradients, warping high-aspect elliptical ravines into isotropic spheres.
4. Adam (Adaptive Moment Estimation): Combines first-moment tracking m_t (momentum) and second-moment tracking v_t (RMSProp) with initialization bias corrections:

m̂_t = (m_t) / (1 - β₁^t), v̂_t = (v_t) / (1 - β₂^t), θ_(t+1) = θ_t - (α) / (√(v̂_t) + ε) m̂_t

Adam and its decoupled weight decay successor AdamW are the default optimization engines powering virtually all modern foundation models, including GPT-4, Llama 3, Stable Diffusion, and Gemini.

The progression spans Robbins & Monro's stochastic approximation (1951), Boris Polyak's heavy-ball method (1964), Geoff Hinton's RMSProp lecture (2012), and Kingma & Ba's Adam paper (ICLR 2015).

• Golden Axiom: "Momentum provides velocity; RMSProp provides per-parameter friction; Adam harmonizes both with bias-corrected clocks."
• m_t tracks gradient direction, v_t scales step size, (1 - β^t) fixes the cold start.

In an ill-conditioned quadratic valley, the Hessian has widely separated eigenvalues λ₁ ≫ λ₂.

Vanilla gradient descent oscillates violently along the steep eigenvector v₁ while creeping along v₂:

w_(t+1) = w_t - η ∇ L(w_t)

Polyak momentum introduces an inertial velocity state v_t ∈ R^d:

v_t = β v_(t-1) + η ∇ L(w_t)

w_(t+1) = w_t - v_t

Where β ∈ [0.9, 0.99] is the momentum coefficient.

Along oscillating directions where ∇ L alternates signs (+g, -g, +g), successive steps cancel out:

v_t^((1)) ≈ 0

Along consistent directions where ∇ L has steady sign (+g, +g, +g), velocity compounds by a factor of (1) / (1 - β):

v_t^((2)) → (η) / (1 - β) g

When β = 0.9, consistent steps accelerate by 10×.

Momentum is a core component of Stochastic Gradient Descent with Momentum (`optim.SGD(lr=0.01, momentum=0.9)`) and forms the first-moment velocity vector m_t inside the Adam and AdamW optimizers.

Boris Polyak introduced the Heavy Ball method in 1964 in the USSR, proving that second-order linear multistep methods accelerate the convergence rate on ill-conditioned convex quadratics from O(1/t) to O(1/t²).

• Sideways zig-zags cancel out; forward velocity compounds.
• Momentum turns a nervous ping-pong ball into a heavy bowling ball.

In neural networks, some weights receive frequent massive gradients, while others receive rare tiny gradients.

RMSProp tracks the uncentered second raw moment of gradients g_t:

s_t = β₂ s_(t-1) + (1 - β₂) g_t²

Where g_t² is the element-wise square, and β₂ ≈ 0.99.

The parameter update scales coordinates inversely by their standard deviation:

w_(t+1) = w_t - (η) / (√(s_t) + ε) \odot g_t

Effects across coordinates:

1. Coordinates with huge gradients (gᵢ² ≫ 1): √(sᵢ) ≈ |gᵢ|, so effective step is (η) / (|gᵢ|) gᵢ = ± η (tamed!).
2. Coordinates with tiny gradients (gᵢ² ≪ 1): √(sᵢ) is tiny, boosting the effective step to prevent stagnation.

RMSProp forms the exact second-moment engine v_t inside the Adam and AdamW optimizers. Without this adaptive coordinate scaling, training modern deep Transformers with mixed-frequency embeddings would be completely impossible.

Geoffrey Hinton proposed RMSProp in 2012 in Lecture 6e of his Coursera online course, solving the catastrophic 'monotonic gradient freezing' flaw of AdaGrad.

• Divide by root-mean-square to level the playing field.
• Hot gradients get braked; cold gradients get boosted.

The Adam algorithm maintains two state vectors m_t, v_t ∈ R^d, initialized to 0:

1. First Moment Vector (Mean / Velocity Momentum):

m_t = β₁ m_(t-1) + (1 - β₁) g_t

1. Second Moment Vector (Uncentered Variance / RMSProp):

v_t = β₂ v_(t-1) + (1 - β₂) g_t²

1. Bias Correction:

m̂_t = (m_t) / (1 - β₁^t), v̂_t = (v_t) / (1 - β₂^t)

1. Parameter Update:

w_(t+1) = w_t - (η) / (√(v̂_t) + ε) \odot m̂_t

Canonical default hyperparameters:

β₁ = 0.9, β₂ = 0.999, ε = 10^(-8), η = 10^(-3) to 10^(-4)

Adam (and its weight-decoupled variant AdamW) is the undisputed universal optimizer of modern generative AI, used to pretrain almost every major frontier LLM (GPT-4, LLaMA-3, Claude, Gemini).

Diederik P. Kingma and Jimmy Lei Ba introduced Adam in their ICLR 2015 paper 'Adam: A Method for Stochastic Optimization', which has accumulated over 150,000 citations.

• m_t is velocity (momentum); v_t is speedometer (RMSProp).
• Divide m_hat by sqrt(v_hat) to step with directional confidence and balanced speed.

At initialization, m₀ = 0 and v₀ = 0.

At step t = 1, given gradient g₁:

m₁ = (1 - β₁) g₁ = (1 - 0.9) g₁ = 0.1 g₁

v₁ = (1 - β₂) g₁² = (1 - 0.999) g₁² = 0.001 g₁²

If we naively compute the uncorrected update step:

Δ w₁ = -η (m₁) / (√(v₁)) = -η (0.1 g₁) / (√(0.001 g₁²)) = -η (0.1) / (√(0.001)) sgn(g₁)

Notice the scaling factor:

(0.1) / (√(0.001)) = (0.1) / (0.03162) ≈ 3.162

Now observe step t = 2 and beyond: as t grows, m_t and v_t scale toward E[g] and E[g²], where their ratio approaches 1/1 = 1.0.

In the second moment alone:

Bias factor at t=1: √(1 - β₂¹) = √(0.001) ≈ (1) / (31.62)

Without correction, the denominator is suppressed by 31.6×, causing parameter updates to wildly overshoot and destroy early weight initialization.

This cold-start catastrophe is why Kingma and Ba introduced analytic bias correction, and why modern training recipes also include a 'learning rate warmup' over the first 2,000 steps.

Diederik Kingma and Jimmy Ba derived this exact bias correction in Section 3 of their 2014 paper, establishing mathematical consistency from step t = 1 to t = ∞.

• Zero initialization biases moments toward zero.
• Dividing by a zero-biased denominator creates a 316% overshoot explosion.

We derive the proof for the second moment v_t (the proof for m_t is identical).

Unroll the recurrence v_t = β₂ v_(t-1) + (1 - β₂) g_t² starting from v₀ = 0:

v_t = (1 - β₂) ∑_(i=1)^t β₂^(t-i) gᵢ²

Take expectations of both sides, assuming the true second moment E[gᵢ²] is stationary:

E[v_t] = E[ (1 - β₂) ∑_(i=1)^t β₂^(t-i) gᵢ² ] = (1 - β₂) ∑_(i=1)^t β₂^(t-i) E[gᵢ²]

Factor out E[g²]:

E[v_t] = E[g²] · (1 - β₂) ∑_(i=1)^t β₂^(t-i)

Let k = t - i. The finite geometric series sums to:

∑_(k=0)^(t-1) β₂^k = (1 - β₂^t) / (1 - β₂)

Substitute back:

E[v_t] = E[g²] · (1 - β₂) ( (1 - β₂^t) / (1 - β₂) ) = E[g²] (1 - β₂^t)

Therefore, v_t is a biased estimator by the factor 1 - β₂^t.

Defining the bias-corrected estimator:

v̂_t = (v_t) / (1 - β₂^t) ⟹ E[v̂_t] = (E[v_t]) / (1 - β₂^t) = E[g²]

As t → ∞, β₂^t → 0, so 1 - β₂^t → 1, and the correction gracefully fades away.

This mathematical derivation is implemented in every autograd framework (`1.0 - beta1**step` in PyTorch's `adam.cpp`).

Diederik Kingma and Jimmy Ba published this formal proof in 2014, showing how finite geometric series summation solves the initialization bias of recursive exponential smoothing.

• Sum of beta^k from 0 to t-1 equals (1 - beta^t) / (1 - beta).
• Divide by 1 - beta^t to restore unbiased statistical expectation.

The Rosenbrock Banana Function is defined as:

f(x, y) = (1 - x)² + 100(y - x²)²

Global minimum: (x^, y^) = (1, 1) where f(1, 1) = 0.

Optimization challenges:

1. Extreme Anisotropy: The parabolic valley y = x² has steepness scaling by 100, while the valley floor gradient is gentle.
2. Non-Linear Ravine: Unlike a straight quadratic valley, the valley curves parabolically, requiring continuous simultaneous adjustments in both velocity direction and coordinate scale.

Performance breakdown across optimizers:

• Vanilla SGD: Gets trapped in cross-ravine oscillations; requires > 100,000 steps to round the bend.
• Polyak Momentum: Rapidly reaches the valley floor, but overshoots wildly on the curved parabolic bend due to excessive momentum inertia.
• RMSProp: Rescales the steep canyon walls effectively, making steady progress without oscillation, but moves slowly along the flat valley floor.
• Adam: Combines momentum speed along the floor with RMSProp dampening on the walls, rounding the parabolic bend and reaching (1, 1) in the fewest total iterations.

The Rosenbrock function and the Rastrigin function are standard benchmark test suites used by optimization researchers to evaluate new optimizers (e.g. Lion, Sophia, Muon) before training billion-parameter neural networks.

Howard H. Rosenbrock introduced this function in 1960 ('An automatic method for finding the greatest or least value of a function'), cementing it as the ultimate stress test for optimization algorithms.

• Rosenbrock is the ultimate bobsled test track.
• Adam wins the Grand Prix by combining bowling-ball speed with sports-car traction.

A master optimization algorithm card in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.

Top section: Title in navy serif: "The Adam Optimizer: King of Deep Learning". Subtitle: "Kingma & Ba (2014): Combining Momentum (First Moment) with RMSProp (Second Moment)."

Middle section: Step-by-step algorithmic breakdown across 4 numbered panels:

1. First Moment (Directional Momentum):

m_t = β₁ m_(t-1) + (1 - β₁) g_t (β₁ = 0.9, decaying average of gradients).

1. Second Moment (Uncentered Variance):

v_t = β₂ v_(t-1) + (1 - β₂) g_t² (β₂ = 0.999, decaying average of squared gradients).

1. Bias Correction (Overcoming Zero-Initialization Bias):

m̂_t = (m_t) / (1 - β₁^t), v̂_t = (v_t) / (1 - β₂^t).

1. Adaptive Parameter Step:

θ_(t+1) = θ_t - (η) / (√(v̂_t) + ε) m̂_t.

Bottom section: Adaptive scaling: "Parameters with large historical gradients receive smaller learning rates; parameters with infrequent gradients receive larger learning rates."

Clean 4-panel layout, prominent bias-correction callout, balanced square formatting.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Kingma & Ba (2014), Adam: A Method for Stochastic Optimization, ICLR 2015 ↗

"Adam Optimizer: Adaptive Moment Estimation: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."