Gelu Activation
How does the Gaussian Error Linear Unit (GELU) introduce probabilistic gating compared to deterministic ReLU?
How this lesson unfolds.
Begin with How does the Gaussian Error Linear Unit (GELU) introduce probabilistic gating compared to deterministic ReLU?. Follow the visual diagrams to build an intuitive mental model.
Why does the smooth probabilistic gating of GELU outperform the hard zero-clamping threshold of ReLU in language modeling?
Imagine a security guard stationed at the door of an exclusive VIP lounge. A ReLU guard is a ruthless, rigid bouncer: if your guest pass is even one cent under \0, you are instantly ejected into the freezing street and your microphone is permanently muted. If you are even slightly positive, you pass through unaltered. A GELU guard is a suave concierge: if your ticket value is slightly below zero, they do not banish you immediately; they acknowledge there is still a small, probabilistic chance you belong inside. Only deeply negative, hopeless tickets are fully suppressed. This gentle grace allows borderline signals to recover and learn.
- Learning goalContrast the mathematical and optimization properties of GELU against ReLU, explaining how probabilistic gating prevents the dying neuron defect.
- Helpful foundationsCumulative distribution function of the Gaussian distribution · Non-linear activation functions (ReLU, Sigmoid)
- Common trapAssuming that GELU is monotonic like Sigmoid or ReLU. GELU is non-monotonic: it dips below zero for x ∈ (-1.5, 0), reaching a minimum of ≈ -0.17 before smoothly approaching zero.
- Next connectionNext, derive the two-layer 4× feed-forward expansion and fast Tanh approximation calculus in Part B.
More context & mechanics
The transition from ReLU to GELU represents a major paradigm shift in non-linear activation design:
- The ReLU Discontinuity & Dying Neuron Problem:
ReLU(x) = max(0, x)
- Derivative: f'(x) = 1 for x > 0, but f'(x) = 0 for x < 0.
- The Dying ReLU Trap: If a large gradient knocks a neuron's pre-activation into strongly negative territory, its derivative becomes permanently 0.0. The neuron stops learning forever.
- The GELU Probabilistic Formulation (Hendrycks & Gimpel, 2016):
GELU weights an input by the probability that a standard Gaussian variable X ~ N(0, 1) is less than x:
GELU(x) = x · Φ(x) = x · P(X ≤ x)
- Smooth Curvature & Negative Dips:
Unlike ReLU, GELU is non-monotonic and infinitely differentiable (C^∞). It dips smoothly to a minimum of ≈ -0.17 at x ≈ -0.75, allowing small negative values to propagate useful curvature information.
GELU is the universal standard activation function across modern LLMs, including GPT-2, GPT-3, BERT, and RoBERTa. In SwiGLU variants (used in Llama-3), GELU's smooth gating principle is extended into gated linear unit formulations.
Dan Hendrycks and Kevin Gimpel introduced the Gaussian Error Linear Unit (GELU) in 2016. They sought to combine the stochastic regularizing benefits of Dropout with the deterministic non-linearity of Zoneout and ReLU.
- ReLU chops; GELU gates: ReLU is a hard knife; GELU is a smooth probabilistic dimmer switch.
- Never kill a neuron permanently: Small negative inputs deserve a voice; smooth curves keep gradients alive.
How is the two-layer 4× feed-forward network structured, and how does the fast Tanh approximation compute GELU efficiently?
Think of an accordion player. To create a rich, resonant musical phrase, the player pulls the accordion wide open to four times its resting width (4× expansion). In this expanded chamber, air vibrates across dozens of specialized reeds (GELU activation). Then, the player squeezes the accordion back to its standard compact profile to fit the master tempo of the band. The FFN expands to think, activates to filter, and contracts to transmit.
- Learning goalFormulate the two-layer 4× FFN architecture, calculate its parameter census, and derive the fast Tanh GELU approximation formula.
- Helpful foundationsError function erf(x) and hyperbolic tangent \tanh(x) · Matrix multiplication dimensions and parameter counting
- Common trapAssuming that attention consumes the majority of a transformer's parameters. The feed-forward network contains roughly two-thirds (66.7%) of all trainable parameters in each transformer block.
- Next connectionNext, discover how residual shortcut connections provide a gradient highway around these layers in Chapter 22 Screen 1.
More context & mechanics
#### 1. Two-Layer 4× Expansion Architecture
The position-wise Feed-Forward Network consists of two linear transformations separated by a non-linear activation:
FFN(x) = GELU(x W₁ + b₁) W₂ + b₂
- Layer 1 (Expansion): W₁ ∈ R^d_(model) × 4d_(model) projects the hidden state into a wide 4x feature space (768 → 3072 in GPT-2).
- Activation: GELU is applied element-wise across the 4d_(model) dimensions.
- Layer 2 (Contraction): W₂ ∈ R^4d_(model) × d_(model) compresses the wide state back to d_(model) (3072 → 768).
The exact cumulative normal distribution involves the error function:
Φ(x) = 1/2 [ 1 + erf( (x) / (√(2)) ) ]
Because erf(x) requires expensive numerical integration, Hendrycks and Gimpel derived a fast analytical approximation based on hyperbolic tangent:
GELU(x) ≈ 0.5 x ( 1 + \tanh( √((2) / (π)) ( x + 0.044715 x³ ) ) )
Where √(2/π) ≈ 0.79788456. This approximation matches the exact error function with maximum error < 10^(-4).
For d_(model) = 768:
- W₁: 768 × 3072 = 2,359,296 weights (+ 3,072 biases)
- W₂: 3072 × 768 = 2,359,296 weights (+ 768 biases)
- Total FFN parameters: 4,722,432 parameters (accounts for roughly 66.7% of all parameters in each transformer block!).
In PyTorch and Hugging Face, `nn.GELU(approximate='tanh')` invokes this fast mathematical formula. On NVIDIA GPUs, tensor cores compute this polynomial via fast fused multiply-add (FMA) instructions, executing 3× faster than standard `torch.erf`.
The 4× expansion factor (4d_(model)) was introduced empirically by Vaswani et al. in 2017 and has endured as the golden standard for almost a decade. Modern architectures like Llama use a 8/3× expansion ratio in combination with SwiGLU to maintain equivalent parameter counts.
- 1 → 4 → 1: Expand fourfold to process; contract back to fit the residual bus.
- Two-thirds of the block: The FFN contains twice as many parameters as the attention layer.
What is the analytical formulation of GELU and its fast hyperbolic tangent approximation used in GPT-2?
Calculating the exact Gaussian integral requires calculating the error function erf(x), which requires numerical series expansion and slows down GPU tensor cores. In GPT-2, OpenAI replaced erf with a clever polynomial inside tanh: sqrt(2/pi) * (x + 0.044715 x³). To the naked eye and to gradient descent, the two curves are virtually indistinguishable (differing by less than 0.0004), but tanh executes substantially faster in CUDA floating-point units.
- Learning goalDerive the analytical formulation GELU(x) = 0.5x(1 + erf(x/sqrt(2))) and compute values using the fast tanh approximation.
- Helpful foundationsGELU Activation (Hendrycks & Gimpel, 2016) · Gaussian Error Function (erf) · Hyperbolic Tangent (tanh) & Polynomial Approximations
- Common trapAssuming the tanh version is an unrelated activation. It is a Taylor-series-inspired algebraic approximation to the Gaussian error function: erf(x / sqrt(2)) approx tanh(sqrt(2/pi) * (x + 0.044715 x³)).
- Next connectionWith the GELU activation function in hand, we explore how it powers the non-linear expansion and contraction inside the Transformer Feed-Forward Network.
More context & mechanics
1. Exact Analytical Formulation:
Using the relationship between the standard Gaussian CDF Φ(x) and the error function erf(z) = (2) / (√(π)) ∫₀^z e^(−t²) dt:
Φ(x) = 1/2[1 + erf((x) / (√(2)))]
Multiplying by x gives the exact analytical formula:
GELU(x) = x/2 [1 + erf((x) / (√(2)))]
2. Fast Tanh Approximation (GPT-2):
To avoid computing `erf` on older GPU architectures lacking hardware-accelerated error functions, Hendrycks & Gimpel provided the tanh approximation:
GELU(x) ≈ 0.5 x (1 + \tanh(√((2) / (π)) (x + 0.044715 x³)))
where √(2/π) ≈ 0.79788456.
The maximum absolute error between this approximation and the true CDF is less than 4 × 10^(-4) for all x ∈ R.
OpenAI chose this exact tanh formulation for GPT-2 and GPT-3 (`gelu_new` or `gelu_fast` in Hugging Face Transformers). Modern PyTorch supports both `nn.GELU(approximate='none')` and `nn.GELU(approximate='tanh')`.
The cubic coefficient 0.044715 was derived via minimax polynomial approximation by Dan Hendrycks in 2016 to minimize uniform error against the exact normal CDF across the interval [-3, 3].
- Golden Axiom: "Half x times one plus tanh."
- GELU(x) ≈ 0.5x(1 + \tanh(ṡ)): captures 99.96% of the exact Gaussian curve at a fraction of the hardware cost.
How does the two-layer Feed-Forward Network expand and project token representations with GELU activation?
Imagine an accordion. When playing a note, you pull the accordion wide open to let air fill its chambers (expanding the token dimension from 768 to 3072). In this expanded space, specialized acoustic reeds (GELU neurons) resonate with specific factual patterns. Then you press the accordion back together (projecting from 3072 back to 768), condensing the enriched sound into a single chord. The 4x expansion gives the network room to untangle complex non-linear combinations of facts.
- Learning goalTrace the tensor transformations of the Feed-Forward Network FFN(x) = GELU(x W₁ + b₁) W₂ + b₂ through shapes (B, T, d) -> (B, T, 4d) -> (B, T, d).
- Helpful foundationsAttention Is All You Need (Vaswani et al., 2017) · Multi-Layer Perceptrons & Non-Linear Embeddings · Associative Memory in Neural Networks
- Common trapAssuming the Feed-Forward Network mixes information across different tokens. The FFN is strictly position-wise: each token is processed by W₁ and W₂ completely independently, with zero cross-token parameter sharing during that step.
- Next connectionBecause deep transformer blocks stack multiple Attention and FFN layers, training would collapse without residual shortcut connections preserving gradient flow.
More context & mechanics
The Position-wise Feed-Forward Network consists of two linear transformations with a GELU non-linearity in between:
FFN(x) = GELU(x W₁ + b₁) W₂ + b₂
Tensor Dimensions:
- Input: x ∈ R^B × T × d_(model) (e.g. 768)
- First Linear Layer (W₁ ∈ R^d_(model) × 4 d_(model), b₁ ∈ R^4 d_(model)):
h₁ = x W₁ + b₁ ∈ R^(B × T × 3072)
- Non-linear Activation:
a₁ = GELU(h₁) ∈ R^(B × T × 3072)
- Second Linear Layer (W₂ ∈ R^4 d_(model) × d_(model), b₂ ∈ R^d_(model)):
x_(out) = a₁ W₂ + b₂ ∈ R^(B × T × 768)
Memory Interpretation:
Recent mechanistic interpretability research (Geva et al., 2021) shows that W₁ acts as a bank of pattern detectors (Keys), and W₂ acts as a bank of associated facts (Values). When GELU activates a specific neuron in the 3072 intermediate dimension, its corresponding column in W₂ writes a factual update into the token representation.
The FFN accounts for approximately two-thirds of the total parameters in a Transformer model (excluding embedding matrices), serving as the primary repository of learned factual knowledge.
Vaswani et al. (2017) established the 4 × d_(model) ratio as the optimal empirical balance between parameter capacity and compute throughput. Modern architectures (LLaMA, Mistral) adapt this into SwiGLU with an expansion ratio of 8/3 ≈ 2.67 × d_(model).
- Golden Axiom: "Expand 4x, activate with GELU, project back to d."
- Dimension flow: d → 4d → d. The FFN is where knowledge is stored.
How does element-wise multiplicative gating with Swish outperform standard non-linear activations in Transformer feed-forward networks?
Think of a river lock system with two gates. Instead of a single rigid weir that lets water spill through if it rises above a threshold, SwiGLU uses one gate to measure the current and a second motorized valve that opens or constricts based on that measurement, dynamically throttling token information.
- $$SwiGLU replaces single-path activation with a bilinear gated unit SwiGLU(x) = (Swish(x W_1) * (x W_3)) W_2, providing dynamic information routing that improves perplexity across large pretraining runs.$$
- Learning goalFormulate the bilinear tensor transformations in a SwiGLU MLP block, calculate parameter expansion ratios (8/3 * d_model), and implement the forward pass in PyTorch.
- Helpful foundationsGLU Variants Improve Transformer (Noam Shazeer, Google Brain, 2020) · Language Modeling with Gated Convolutional Networks (Dauphin et al., Facebook AI, 2016) · Searching for Activation Functions / Swish (Ramachandran, Zoph & Le, Google Brain, 2017)
- Common trapAssuming SwiGLU adds 50% more parameters without trade-offs. To keep total parameter count identical to a standard 4 d_model FFN (2 d_model 4 d_model = 8 d_model²), SwiGLU scales hidden dimension to (8/3) d_model. Because it uses 3 weight matrices (W₁, W₂, W₃), the total parameter count is 3 d_model (8/3) d_model = 8 d_model², matching standard GPT-2 exactly.
- Next connectionWith the non-linear feed-forward component upgraded to gated units, Chapter 22 explores how residual shortcut connections prevent gradient vanishing across deep stacks of these blocks.
More context & mechanics
In standard GPT-2, the Feed-Forward Network (FFN) applies:
FFN(x) = GELU(x W₁ + b₁) W₂ + b₂
where W₁ ∈ R^d_(model) × 4 d_(model) and W₂ ∈ R^4 d_(model) × d_(model).
Gated Linear Units (GLU) split the intermediate projection into two linear streams and compute their element-wise Hadamard product \odot. In SwiGLU, the gating stream uses the Swish (SiLU) activation Swish(z) = z · σ(z):
SwiGLU(x) = ( Swish(x W₁) \odot (x W₃) ) W₂
where:
- W₁ ∈ R^d_(model) × d_(ff) (gate projection)
- W₃ ∈ R^d_(model) × d_(ff) (value/up projection)
- W₂ ∈ R^d_(ff) × d_(model) (down projection)
Toy numerical trace:
Let x_(proj1) = 2.0 (gate input) and x_(proj3) = 3.0 (value input):
- σ(2.0) = (1) / (1 + e^(−2.0)) ≈ 0.8808.
- Swish(2.0) = 2.0 × 0.8808 = 1.7616.
- Gated product: Swish(2.0) × x_(proj3) = 1.7616 × 3.0 = 5.2848.
Compare to an inhibited token where gate input is negative, e.g., x_(proj1) = -3.0:
σ(-3.0) ≈ 0.0474 ⟹ Swish(-3.0) ≈ -0.142, suppressing the value stream to -0.142 × 3.0 = -0.427.
SwiGLU is the engine of virtually every frontier open LLM developed since 2022: Meta's Llama 1/2/3, Mistral/Mixtral, Google's PaLM and Gemma, and DeepSeek-V3.
Yann Dauphin et al. introduced Gated Linear Units in 2016 for convolutional language models. In 2020, Noam Shazeer (Google Brain) published 'GLU Variants Improve Transformer', systematically benchmarking activations and finding that SwiGLU consistently achieved the lowest validation loss across language tasks.
- Golden Axiom: "Two projections multiply: one decides what to remember, the other decides how loud to say it."
- Remember the parameter conservation rule: hidden dimension drops from 4d to 8/3d so 3 matrices cost the exact same FLOPs and weights as 2 matrices.
How does GELU Activation: Gaussian Error Linear Units operate mathematically at scale?
A clean 2D coordinate plot comparing two curves:
- Learning goalMaster the algebraic and geometric derivation of GELU Activation: Gaussian Error Linear Units.
- Helpful foundationsFoundations of Build Llm From Scratch · Coordinate Invariance & Geometric Projections · Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
- Common trapAssuming GELU Activation: Gaussian Error Linear Units is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
A mathematical activation function comparison plot in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.
Top section: Title in navy serif: "GELU: Probabilistic Smooth Non-Linearity". Subtitle: "Hendrycks & Gimpel (2016): Weighting inputs by their percentile in a standard normal distribution."
Middle section: A clean 2D coordinate plot comparing two curves:
- Dashed Grey Curve (ReLU): Hard corner hinge at x = 0. Zero gradient for all negative inputs (Dying ReLU problem).
- Smooth Navy Curve (GELU): Curving gently below zero with a smooth local minimum at x ≈ -0.75, then asymptotically approaching x for positive inputs.
Mathematical Definition:
"GELU(x) = x · Φ(x) = x · P(X ≤ x), X ~ N(0, 1)".
Fast Approximation:
"GELU(x) ≈ 0.5 x (1 + \tanh(√((2) / (π)) (x + 0.044715 x³)))".
Bottom section: Theoretical insight: "Combines deterministic scaling with stochastic dropout: larger inputs are passed with higher probability, while small negative inputs are gently suppressed."
Pristine mathematical curves, clearly labeled inflection points, balanced square frame.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
"GELU Activation: Gaussian Error Linear Units: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."