Weight Regularization: L1 Sparsity, L2 Ridge & Weight Decay Calculus

Regularization augments the unconstrained objective L₀(w) with a norm penalty:

1. L1 Regularization (Lasso):

L(w) = L₀(w) + λ ∑_(i) |wᵢ| = L₀(w) + λ \|w\|₁

Subgradient is constant: (∂) / (∂ wᵢ) |wᵢ| = sign(wᵢ), pulling weights toward zero with constant force regardless of magnitude, driving small weights to absolute zero (feature selection/sparsity).

1. L2 Regularization (Ridge):

L(w) = L₀(w) + 1/2 λ ∑_(i) wᵢ² = L₀(w) + 1/2 λ \|w\|₂²

Gradient is proportional to weight size: (∂) / (∂ wᵢ) 1/2 wᵢ² = wᵢ, pulling large weights strongly and small weights gently, keeping values diffuse and non-zero.

L1 is used when sparse feature selection and model compression are required. L2 (weight decay) is used universally across all modern deep learning to prevent extreme weight magnitudes.

Lasso was introduced by Robert Tibshirani (1996) in statistics; Ridge regression was formulated by Hoerl and Kennard (1970) for ill-conditioned inverse problems.

L1 has sharp diamond corners that clip weights to exact zero; L2 has smooth circular boundaries that shrink weights in harmony.

L2 regularization penalizes complex models by imposing a quadratic constraint on the parameter vector:

1. The Regularized Loss Objective:

L_(total)(W) = L₀(W) + (λ) / (2) \|W\|₂² = L₀(W) + (λ) / (2) ∑_(i) wᵢ²

Where L₀(W) is the unregularized data loss (e.g., cross-entropy), \|W\|₂ is the Euclidean norm, and λ > 0 is the regularization strength.

1. The Geometric Lagrangian Perspective:

Minimizing L_(total) is mathematically dual to minimizing L₀(W) subject to a hard spherical constraint:

min_W L₀(W) subject to \|W\|₂² ≤ C

In 2D parameter space, \|W\|₂² = w₁² + w₂² ≤ C defines a disk (a hypersphere in R^D).

1. The Equilibrium Condition:

The optimal solution occurs where the elliptical contours of the unregularized loss L₀ become tangent to the spherical boundary of the L2 ball. At this contact point:

∇ L₀(W^) = -λ W^

The negative gradient of the data loss exactly counters the elastic restorative force of the L2 penalty, smoothly shrinking all coordinates toward zero.

Standard across linear models (Ridge regression), convolutional neural nets, and support vector machines (SVMs) to prevent weights from exploding and to control model capacity.

The mathematical foundation of L2 regularization was established by Andrey Tikhonov in 1943 (Tikhonov Regularization) for solving ill-posed inverse problems. In statistics, it was introduced by Arthur Hoerl and Robert Kennard in 1970 as Ridge Regression before being universally adopted in neural network training.

• Bungee cord to the origin: The farther weights roam from zero, the harder the penalty pulls them back.
• Smooth circular shrinkage: Unlike L1 which clips weights to sharp axes, L2 shrinks all weights uniformly in proportion to their size.

#### 1. The Analytical Gradient Derivation

Consider the regularized objective function:

L_(total)(W) = L₀(W) + (λ) / (2) ∑_(i,j) w_(i,j)²

Taking the partial derivative with respect to an arbitrary weight w_(i,j):

(∂ L_(total)) / (∂ w_(i,j)) = (∂ L₀) / (∂ w_(i,j)) + (∂) / (∂ w_(i,j)) ( (λ) / (2) ∑_(k,l) w_(k,l)² )

Evaluating the derivative of the quadratic sum:

(∂) / (∂ w_(i,j)) ( (λ) / (2) w_(i,j)² ) = (λ) / (2) · 2 w_(i,j) = λ w_(i,j)

In vector notation:

∇_W L_(total) = ∇_W L₀ + λ W

Applying standard gradient descent with learning rate η:

W_(t+1) = W_t - η ∇_W L_(total)

Substituting the expanded gradient:

W_(t+1) = W_t - η ( ∇_W L₀(W_t) + λ W_t )

Factoring out the current weight matrix W_t:

W_(t+1) = (1 - η λ) W_t - η ∇_W L₀(W_t)

The term (1 - η λ) is the weight decay contraction coefficient.

• Since η > 0 and λ > 0, (1 - η λ) < 1.0.
• In the absence of data gradients (∇_W L₀ = 0), weights decay geometrically toward zero: W_t = (1 - η λ)^t W₀.
• For typical hyperparameters η = 0.01 and λ = 0.001:

(1 - η λ) = 1 - (0.01 × 0.001) = 1 - 0.00001 = 0.99999

At every single step, weights are scaled down by 0.001% of their current value.

While mathematically equivalent in vanilla SGD, L2 regularization and true weight decay decouple completely in adaptive optimizers like Adam. This led to Loshchilov & Hutter's AdamW formulation, which forms the core training engine of modern large language models.

In 1991, John Moody investigated the relationship between weight decay and model complexity in neural networks. In 2017, Ilya Loshchilov and Frank Hutter demonstrated that standard L2 regularization fails in Adam due to second-moment normalization, establishing decoupled weight decay as the modern gold standard.

• Decay first, update second: W_(t+1) = (1 - η λ) W_t - η g_t.
• The lambda-eta contraction: Weight decay simply shaves a sliver off the weight vector at every iteration.

The L1 regularization objective adds the Manhattan norm of parameters to the unregularized loss L₀:

L(w) = L₀(w) + λ ∑_(i=1)^D |wᵢ|

Because the absolute value function |w| has a sharp corner at w = 0, it is non-differentiable in the classical sense. We employ the subgradient set ∂ |w|:

∂ |w| = +1 & if w > 0 \ [-1, 1] & if w = 0 \ -1 & if w < 0

The parameter update with learning rate α applies a constant thresholding force α λ · sign(w_t). When |w_t| ≤ α λ, the subgradient contains 0, arresting the parameter at exactly zero.

Let learning rate α = 0.01, penalty λ = 0.5, meaning threshold τ = α λ = 0.005. Suppose the unregularized data gradient is small, say g₀ = 0.001.

1. Case A: Large Weight (w = 0.500):

w_(next) = 0.500 - 0.01 × (0.001 + 0.5 × sign(0.500)) = 0.500 - 0.01(0.501) = 0.495

1. Case B: Tiny Weight (w = 0.003):

Without soft-thresholding, naive update would flip sign: 0.003 - 0.005 = -0.002. The proximal operator (Lasso) snaps it directly to:

w_(next) = sign(0.003) · max(0, |0.003| - 0.005) = 0.000

The feature is completely pruned from the model with zero floating-point residue!

Used in model pruning, sparse linear regression, compressed sensing, and feature selection on high-dimensional genomic and financial datasets.

Formalized by Robert Tibshirani (1996) in 'Regression Shrinkage and Selection via the Lasso', connecting optimization geometry with automatic variable selection.

L2 rounds the edges; L1 points the corners: constant slope cuts straight to absolute zero.

In standard Stochastic Gradient Descent (SGD), adding an L2 regularization penalty to the objective function produces an update mathematically identical to weight decay:

∇_(W) (L + 1/2 λ \|W\|²) = ∇ L + λ W

W_(t+1) = W_t - α (∇ L + λ W_t) = (1 - α λ) W_t - α ∇ L

However, in adaptive gradient algorithms such as Adam, the update step scales inversely with the root of the uncentered second moment v_t = β₂ v_(t-1) + (1 - β₂) g_t²:

W_(t+1) = W_t - α (m̂_t) / (√(v̂_t) + ε)

If the L2 penalty is baked into the gradient g_t = ∇ L + λ W_t, the regularization term is divided by √(v̂_t). Parameters associated with large or frequent gradients receive virtually no weight decay, while parameters with tiny historical gradients are over-regularized. Loshchilov & Hutter (2019) resolved this breakdown by decoupling the decay step entirely:

W_(t+1) = W_t - α λ W_t - α (m̂_t) / (√(v̂_t) + ε) = (1 - α λ) W_t - α (m̂_t) / (√(v̂_t) + ε)

Here, λ represents the true decoupled weight decay coefficient and α is the learning rate. Every parameter undergoes proportional shrinkage (1 - α λ) regardless of gradient variance.

Concrete Numeric Walkthrough:

Let parameter w = 2.0, learning rate α = 0.01, and weight decay λ = 0.1. Suppose the gradient step Δ_(opt) = (m̂) / (√(v̂) + ε) = 0.05.

1. Decoupled Shrinkage: Decay factor is 1 - α λ = 1 - (0.01 × 0.1) = 1 - 0.001 = 0.999.
2. Shrunk Weight: w_(shrunk) = 0.999 × 2.0 = 1.998.
3. Final Update: w_(next) = 1.998 - (0.01 × 0.05) = 1.998 - 0.0005 = 1.9975.

Notice that weight decay reduced the weight by exactly 0.002 (0.1\% of its magnitude) completely independent of gradient magnitude.

In frontier deep learning frameworks (PyTorch, Hugging Face `transformers`), `torch.optim.AdamW` is the default optimizer for training foundation models (GPT-4, Llama 3, Mistral, ViT). In practice, weight decay is strictly applied only to multi-dimensional weight matrices W, while 1D normalization parameters (LayerNorm γ, β) and bias vectors b are excluded from weight decay to maintain network expressivity.

First proposed by Ilya Loshchilov and Frank Hutter in their seminal ICLR 2019 paper "Decoupled Weight Decay Regularization", demonstrating that Adam had underperformed SGD with momentum on computer vision benchmarks for years solely because of coupled L2 regularization.

• Golden Axiom: "In SGD, L2 and weight decay are identical twins; in Adam, L2 breaks down and decoupled weight decay wins."
• Remember: Decoupled weight decay applies shrinkage directly to W_t, untangled from the adaptive denominator √(v̂_t) + ε.

In standard L2 regularization with objective L_(reg) = L(w) + (λ) / (2) \|w\|²:

The gradient passed to the optimizer is:

g_t = ∇ L(w_t) + λ w_t

In Adam, this modified gradient enters both m_t and v_t:

Δ w_t = -(η) / (√(v̂_t) + ε) m̂_t ≈ -(η) / (√(v̂_t)) (∇ L + λ w_t)

The effective weight decay applied to coordinate i is:

Effective Decay = -(η λ) / (√(v̂_(t, i))) w_(t, i)

Notice the catastrophic inverse scaling by √(v̂_(t, i)):

1. Weights with large gradients (vᵢ ≫ 1) receive negligible weight decay!
2. Weights with small gradients (vᵢ ≪ 1) receive massive, over-penalizing weight decay!

This completely breaks the purpose of regularization.

Decoupled Weight Decay (AdamW):

Subtract weight decay directly from the weights outside the adaptive moment update:

w_(t+1) = w_t - η λ w_t - (η) / (√(v̂_t) + ε) m̂_t

Now, every weight decays strictly proportional to its magnitude, independent of its gradient variance.

AdamW (`torch.optim.AdamW`) completely replaced Adam in modern deep learning frameworks. Every single modern LLM (GPT-4, LLaMA-3, Mistral) is pretrained using AdamW.

Ilya Loshchilov and Frank Hutter identified this mathematical corruption and introduced AdamW in their ICLR 2019 paper 'Decoupled Weight Decay Regularization'.

• L2 inside Adam penalizes the wrong weights.
• Decouple weight decay: subtract lambda * w directly from the parameter.