Regularization Geometry: L1 Diamond vs L2 Circle Constrained Optimization
Why does the geometric intersection of loss contours with constraint norms force weights to exact zero under L1 while shrinking them smoothly under L2?
How this lesson unfolds.
Regularization restricts the capacity of over-parameterized neural networks. By examining the geometry of constraint balls in parameter space, we uncover why L1 induces sparse feature selection while L2 uniformly decays weights.
Why does the geometric intersection of loss contours with constraint norms force weights to exact zero under L1 while shrinking them smoothly under L2?
Imagine blowing up a round balloon (elliptical loss contours) inside a box. If the box is round (L2 circle), the balloon touches the wall at a smooth arbitrary point where both coordinates are non-zero. If the box is a diamond with sharp pointy corners pointing along the North, South, East, West axes (L1 diamond), the expanding balloon almost always hits one of the sharp corner tips first. Touching a tip means one coordinate is exactly zero!
- $$\nabla \mathcal{L}_0(\mathbf{w}) = -\lambda \partial \|\mathbf{w}\|_p$$
- Learning goalIllustrate the geometric tangency between loss contours and L1/L2 constraint balls to explain why L1 induces sparsity.
- Helpful foundationsL1 and L2 penalty loss formulations · Lagrange multipliers and KKT conditions · Regression Shrinkage and Selection via the Lasso (Tibshirani, 1996)
- Common trapBelieving L1 sparsity is an accidental numerical artifact; sparsity is a rigorous mathematical consequence of the non-differentiable vertices of the Manhattan norm ball.
- Next connectionTransition to stochastic regularization in Chapter 31 to explore neuron Dropout.
More context & mechanics
By the Karush-Kuhn-Tucker (KKT) conditions, regularization is equivalent to constrained optimization:
min_w L₀(w) subject to \|w\|_p ≤ C
- L1 Norm Ball (p=1):
\|w\|₁ = |w₁| + |w₂| ≤ C
Forms a diamond polytope in R² (cross-polytope in R^D) with non-differentiable vertices positioned directly on the coordinate axes: (± C, 0) and (0, ± C).
- L2 Norm Ball (p=2):
\|w\|₂² = w₁² + w₂² ≤ C²
Forms a smooth Euclidean hypersphere with continuous tangent hyperplanes everywhere.
The unconstrained loss contours expand outward from unregularized minimum w^*. The first point of tangency between the expanding ellipse and the constraint ball defines the regularized solution w_(reg).
Consider 2D parameter space with unregularized minimum at w^* = (3.0, 1.0):
- Loss contours are ellipses: L = (w₁ - 3)² + 4(w₂ - 1)².
- Under L1 constraint |w₁| + |w₂| ≤ 2.0:
- The ellipse expands from (3, 1) toward the diamond.
- The closest diamond point is the sharp vertex (2.0, 0.0).
- Solution: w_(L1) = (2.0, 0.0)! Coordinate w₂ is snapped to absolute zero, pruning feature 2 completely!
- Under L2 constraint w₁² + w₂² ≤ 2.0² = 4.0:
- Tangency occurs where gradient vectors align: ∇ L = -λ w.
- Solution: w_(L2) ≈ (1.88, 0.68). Both parameters shrink, but neither touches zero.
Fundamental theoretical justification taught across all statistical learning and deep learning curricula.
First visualized geometrically by Robert Tibshirani (1996) in his landmark Lasso paper.
Diamonds have pointy tips that catch coordinates at zero; circles are smooth and never force a corner.
What is the mathematical foundation and geometric intuition of L2 Regularization: From Loss Penalty to Weight Decay?
Understand L2 Regularization: From Loss Penalty to Weight Decay visually: How adding 1/2λ ||w||² to the loss multiplies weights by (1 - η λ) on every step.. Every transformation preserves core structural invariants while mapping coordinate spaces.
- $$How adding \frac{1}{2}\lambda ||w||^2 to the loss multiplies weights by (1 - \eta \lambda) on every step.$$
- Learning goalMaster the algebraic and geometric derivation of L2 Regularization: From Loss Penalty to Weight Decay.
- Helpful foundationsLinear Algebra Fundamentals · Matrix Transformations · Vector Spaces
- Common trapAssuming L2 Regularization: From Loss Penalty to Weight Decay is merely a computational trick rather than an invariant spatial law.
- Next connectionConnects to upstream spectral theory and downstream tensor operations.
More context & mechanics
An algebraic derivation diagram on clean ivory linen (#f7f6ee) with an 8px border.
Top section: Title in technical grotesque: "L2 Regularization: From Loss Penalty to Weight Decay". Subtitle: "How adding 1/2λ ||w||² to the loss multiplies weights by (1 - η λ) on every step."
Middle section: A step-by-step calculus derivation flow:
- Regularized Objective Function: L_(total)(w) = L₀(w) + 1/2λ ||w||₂² = L₀(w) + 1/2λ ∑ wᵢ².
- Gradient Computation: ∇_w L_(total) = ∇_w L₀(w) + λ w.
- SGD Parameter Update Step:
w_(t+1) = w_t - η ∇_w L_(total) = w_t - η (∇_w L₀ + λ w_t).
- Regrouping terms:
w_(t+1) = (1 - η λ) w_t - η ∇_w L₀.
Annotate the multiplicative shrinkage factor (1 - η λ) < 1.0: "Weights undergo geometric exponential decay toward zero on every iteration!"
Bottom section: Card: "L2 regularization acts as a Bayesian Gaussian prior p(w) ~ N(0, 1/λ) over network parameters, penalizing complex overfitted models."
Spacious equations, clean boxed containers, generous margins.
This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.
"L2 Regularization: From Loss Penalty to Weight Decay: Invariance and structure determine stability under spatial and numerical transformations."
How does L₂ Regularization: From Loss Penalty to Weight Decay operate mathematically at scale?
Step-by-step calculus derivation:
- $$Subtitle: "How adding \frac{1}{2}\lambda ||w||^2 to the loss multiplies weights by (1 - \eta \lambda) on every step."$$
- Learning goalMaster the algebraic and geometric derivation of L₂ Regularization: From Loss Penalty to Weight Decay.
- Helpful foundationsFoundations of Neural Networks From Scratch · Coordinate Invariance & Geometric Projections · Goodfellow, Bengio, and Courville (2016), Deep Learning, MIT Press
- Common trapAssuming L₂ Regularization: From Loss Penalty to Weight Decay is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
An analytical mathematical equivalence card in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.
Top section: Title in navy serif: "L2 Regularization: From Loss Penalty to Weight Decay". Subtitle: "How adding 1/2λ ||w||² to the loss multiplies weights by (1 - η λ) on every step."
Middle section: Step-by-step calculus derivation:
- Regularized Objective Function:
L_(total)(w) = L₀(w) + 1/2 λ ||w||₂² = L₀(w) + 1/2 λ ∑ⱼ wⱼ².
- Differentiate with respect to weight w:
∇_w L_(total) = ∇_w L₀ + λ w.
- Apply Gradient Descent update rule (step size η):
w_(t+1) = w_t - η ∇_w L_(total) = w_t - η (∇_w L₀ + λ w_t).
- Rearrange terms:
w_(t+1) = (1 - η λ) w_t - η ∇_w L₀.
Bottom section: Golden insight: "The factor (1 - η λ) < 1 slightly shrinks the weight vector toward zero before applying the data gradient step. This is the exact definition of Weight Decay."
Clear highlighted algebra steps, distinct shrinkage coefficient callout, balanced square styling.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
Goodfellow, Bengio, and Courville (2016), Deep Learning, MIT Press
"L₂ Regularization: From Loss Penalty to Weight Decay: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."