Activation Functions: Non-Linearity, ReLU & Space Bending

The evolution of activation functions reflects the historical struggle to enable deep backpropagation:

1. The Heaviside Step Function (Perceptron 1957):

f(z) = 1 & if z ≥ 0 \ 0 & if z < 0

• Fatal Flaw: The derivative is 0.0 everywhere except at z=0, where it is undefined (δ(z)). Because (∂ f) / (∂ z) = 0, backpropagation cannot calculate gradients.

1. The Sigmoid Function (Backpropagation 1986):

σ(z) = (1) / (1 + e^(−z))

• Breakthrough: Smooth, non-linear, and infinitely differentiable (C^∞). Output is bounded in (0, 1), ideal for probabilities.
• Fatal Flaw (Vanishing Gradients): Maximum derivative is σ'(0) = 0.25. As |z| > 4, derivative collapses to 0.0. Cascading through 5 layers shrinks gradients by 0.25⁵ ≈ 0.00097.

1. The ReLU Revolution (Nair & Hinton, 2010):

ReLU(z) = max(0, z)

• The Miracle of Constant Derivative: For all z > 0, the derivative is identically 1.0! Gradients propagate through dozens of layers without shrinking.

ReLU remains the standard activation function across computer vision architectures (ResNet, ConvNeXt, YOLO). In language models, smooth variants like GELU and SwiGLU build directly upon ReLU's non-saturating foundation.

In 2010, Vinod Nair and Geoffrey Hinton introduced the Rectified Linear Unit (ReLU) for Restricted Boltzmann Machines, and in 2012, Alex Krizhevsky (AlexNet) used ReLU to train the first deep convolutional network that won ImageNet, launching the modern deep learning boom.

• Step has no gradient; Sigmoid vanishes; ReLU survives: The three epochs of activation history.
• Derivative = 1: The simple property that saved deep neural networks from extinction.

#### 1. Mathematical Derivations

1. Sigmoid Derivative:

σ(z) = (1) / (1 + e^(−z)) = (1 + e^(−z))^(-1)

(dσ) / (dz) = -(1 + e^(−z))^(-2) · (-e^(−z)) = (e^(−z)) / ((1 + e^(−z))²) = (1) / (1 + e^(−z)) · (e^(−z)) / (1 + e^(−z)) = σ(z) ( 1 - σ(z) )

• Maximum at z = 0: σ(0) = 0.5 ⟹ σ'(0) = 0.5 × 0.5 = 0.25.

1. Tanh Derivative:

\tanh(z) = (e^z - e^(−z)) / (e^z + e^(−z)) ⟹ (d\tanh) / (dz) = 1 - \tanh²(z)

• Maximum at z = 0: \tanh(0) = 0 ⟹ \tanh'(0) = 1 - 0 = 1.0.

1. ReLU Derivative:

f(z) = max(0, z) ⟹ df/dz = 1 & if z > 0 \ 0 & if z < 0

Evaluate the Sigmoid derivative at pre-activation z = 5.0:

σ(5.0) = (1) / (1 + e^(−5.0)) = (1) / (1 + 0.006738) ≈ (1) / (1.006738) ≈ 0.993307

Now compute its derivative:

σ'(5.0) = σ(5.0) ( 1 - σ(5.0) ) = 0.993307 × (1 - 0.993307) = 0.993307 × 0.006693 ≈ 0.006648

The gradient has shrunk by 99.34\%! A learning signal passing through two consecutive saturated sigmoid neurons shrinks by:

0.006648 × 0.006648 ≈ 0.000044 ⟹ Backpropagation completely halts!

For ReLU at z = 5.0:

ReLU'(5.0) = 1.000000 ⟹ Zero shrinkage!

Understanding saturation limits is vital for proper weight initialization. Xavier (Glorot) initialization scales weights specifically to keep initial pre-activations near z ∈ [-1, 1] where Sigmoid/Tanh derivatives are non-zero. For ReLU, He (Kaiming) initialization scales variance by √(2/N) to account for the half of inputs that fall below zero.

The algebraic identity σ'(z) = σ(z)(1-σ(z)) was a primary mathematical reason David Rumelhart, Geoffrey Hinton, and Ronald Williams chose the logistic function for their seminal 1986 Nature paper introducing Backpropagation.

• σ' = σ(1 - σ): The derivative of Sigmoid is expressed purely in terms of its own output.
• Max derivative is 0.25: Sigmoid cuts your gradient by at least 4× at every single layer.

Consider a 3-layer linear network with weight matrices W₁, W₂, W₃:

y = (((x W₁ + b₁) W₂ + b₂) W₃ + b₃) = x (W₁ W₂ W₃) + b_(combined) = x W_(net) + b_(net)

Because matrix multiplication is associative, the product of linear transformations is strictly linear. The network can only draw straight decision planes and cannot solve XOR.

Demonstrates why every deep architecture must interleave linear affine transformations with non-linear activation functions.

Proved by Marvin Minsky and Seymour Papert (1969) in 'Perceptrons', exposing the fundamental limitation of single-layer linear perceptrons.

Linear plus linear equals linear: depth without non-linearity is a single flat matrix.

Consider a 2-layer network without activation functions:

h = W₁ x + b₁

y = W₂ h + b₂

Substitute h directly into y:

y = W₂ (W₁ x + b₁) + b₂ = (W₂ W₁) x + (W₂ b₁ + b₂)

Define equivalent parameters:

W_(eq) = W₂ W₁ ∈ R^(m × n), b_(eq) = W₂ b₁ + b₂ ∈ R^m

y = W_(eq) x + b_(eq)

No matter how many linear layers L are composed:

y = W_L ṡ W₂ W₁ x + b_(net) = W_(effective) x + b_(net)

The expressiveness remains strictly restricted to linear hyperplanes, unable to solve even simple XOR non-linear problems.

This mathematical reality is why non-linearities (ReLU, GELU, SwiGLU) are placed strictly between every linear projection in deep neural networks.

Marvin Minsky and Seymour Papert demonstrated in their famous 1969 book 'Perceptrons' that single-layer perceptrons could not compute the simple XOR function, causing the first 'AI Winter' until multi-layer non-linear networks emerged.

• Matrix times matrix is just another matrix.
• Without non-linearities, depth is an expensive illusion.

The Rectified Linear Unit (ReLU) is defined as:

f(z) = max(0, z) = z & if z > 0 \ 0 & if z ≤ 0

Its derivative is the step function:

f'(z) = 1 & if z > 0 \ 0 & if z < 0

Geometrically:

1. When z > 0, space passes through completely unchanged (identity transformation).
2. When z ≤ 0, the entire negative half-space is projected directly onto the origin/axis hyperplane (dimension collapse).

Stacking layers of ReLUs creates a continuous piecewise linear (CPWL) manifold, partitioning R^n into millions of distinct convex polyhedral regions.

ReLU revolutionized computer vision in AlexNet (2012). Modern variants like GELU (Gaussian Error Linear Unit) in GPT models and SwiGLU in LLaMA smooth out the sharp corner at z = 0, improving gradient flow.

Vinod Nair and Geoffrey Hinton introduced ReLU to neural networks in 2010, showing it trained deep networks six times faster than traditional sigmoid or tanh functions because its gradient does not saturate when z > 0.

• Pass the positives; zero the negatives.
• A crease in space turns curves into straight piecewise cuts.

An affine subspace A ⊂ R^n is defined as A = x₀ + V, where V is a linear subspace.

Under an affine transformation T(x) = W x + b:

T(A) = T(x₀ + V) = (W x₀ + b) + WV

Because the image of a subspace under a linear map WV is itself a linear subspace:

1. Lines map to lines (or points).
2. Planes map to planes (or lines/points).
3. Flat hyperplanes x | w^T x + b = 0 remain strictly flat hyperplanes.

Therefore, composing L affine transformations:

T_(composite) = T_L ∘ T_(L-1) ∘ ṡ ∘ T₁

Is strictly an affine transformation whose decision boundaries are flat hyperplanes of degree 1. It cannot construct quadratic, polynomial, or topological boundaries (e.g. circles, spheres, or XOR clusters).

This geometric proof explains why even a 100-layer neural network with 10 billion parameters cannot classify concentric circles without activation functions.

The algebraic closure of affine transformations under composition was established in early 19th-century projective geometry by August Ferdinand Möbius and Michel Chasles.

• Affine composed with affine is always affine.
• You cannot create an origami swan with a flat rolling pin.

In the classic XOR problem:

• Class 0: (0, 0) and (1, 1)
• Class 1: (1, 0) and (0, 1)

Consider a hidden layer with 2 neurons using ReLU:

h₁ = max(0, x₁ + x₂ - 0.5)

h₂ = max(0, x₁ + x₂ - 1.5)

Mapping the four XOR points into hidden feature space (h₁, h₂):

• (0, 0) → (max(0, -0.5), max(0, -1.5)) = (0, 0) [Class 0]
• (1, 0) → (max(0, 0.5), max(0, -0.5)) = (0.5, 0) [Class 1]
• (0, 1) → (max(0, 0.5), max(0, -0.5)) = (0.5, 0) [Class 1]
• (1, 1) → (max(0, 1.5), max(0, 0.5)) = (1.5, 0.5) [Class 0]

In the (h₁, h₂) plane:

• Class 1 points merge at (0.5, 0).
• Class 0 points sit at (0, 0) and (1.5, 0.5).

A simple linear output neuron y = h₁ - 2 h₂ evaluates:

• Class 0 at (0, 0): 0 - 0 = 0
• Class 1 at (0.5, 0): 0.5 - 0 = 0.5 (Separated!)
• Class 0 at (1.5, 0.5): 1.5 - 2(0.5) = 0.5 - 0.5 = 0.5 (Threshold at 0.25 cleanly separates Class 1 from Class 0!).

This 2-neuron folding is the minimal working proof of deep learning expressiveness, taught as the fundamental triumph over Minsky's 1969 XOR critique.

Rumelhart, Hinton, and Williams (1986) demonstrated this exact hidden representation mapping using backpropagation, unlocking the modern multi-layer connectionist era.

• ReLU creases the paper so a straight ruler can make the cut.
• Hidden layers warp coordinates until tangled classes uncouple.

The Softmax function σ: R^K → Δ^(K-1) executes two sequential transformations:

1. Stage 1: Exponential Positivity Mapping (f: R → R^+):

uᵢ = e^(zᵢ)

Since e^z > 0 for all real z ∈ (-∞, +∞), negative logits are smoothly mapped to small positive decimals (e^(−10) ≈ 4.5 × 10^(-5)), preserving monotonicity:

z_a > z_b ⟺ e^(z_a) > e^(z_b)

1. Stage 2: Partition Function Normalization (g: R^K → Δ^(K-1)):

Compute the partition function (normalizer):

Z = ∑_(j=1)^K uⱼ = ∑_(j=1)^K e^(zⱼ)

Compute probability coordinates:

qᵢ = uᵢ/Z = (e^(zᵢ)) / (∑_(j=1)^K e^(zⱼ))

Summing across all coordinates:

∑_(i=1)^K qᵢ = (∑_(i=1)^K e^(zᵢ)) / (∑_(j=1)^K e^(zⱼ)) = 1.0

The output vector q is strictly constrained to the interior of the standard (K-1)-simplex.

Softmax is the universal output head for multi-class classification and autoregressive token generation across all modern neural architectures.

Ludwig Boltzmann derived this exponential partition function in 1868. John Bridle adapted it as 'Softmax' for neural network classification in 1989.

• Step 1: Exponentiate to make positive.
• Step 2: Divide by the sum to make it 100%.

Mathematical proof of exact shift invariance:

softmax(z - c)ᵢ = (e^(zᵢ - c)) / (∑_(j=1)^K e^(zⱼ - c)) = (e^(−c) e^(zᵢ)) / (e^(−c) ∑_(j=1)^K e^(zⱼ)) = (e^(zᵢ)) / (∑_(j=1)^K e^(zⱼ)) = softmax(z)ᵢ

Let c = maxⱼ zⱼ. Then for every index i:

zᵢ - c ≤ 0

Evaluating the exponential bounds:

e^(zᵢ - c) ∈ (0, 1.0]

The maximal exponent is e⁰ = 1.0, completely eliminating overflow to +∞ (`inf`).

Evaluating the partition sum bounds:

1.0 ≤ ∑_(j=1)^K e^(zⱼ - c) ≤ K

Because the sum is at least 1.0 (from the maximal element e⁰ = 1), division by zero is strictly impossible.

This identity is written into every production deep learning system:

```python

def safe_softmax(z):

z_shifted = z - np.max(z, axis=-1, keepdims=True)

exp_z = np.exp(z_shifted)

return exp_z / np.sum(exp_z, axis=-1, keepdims=True)

```

The max-shift identity became canonical in numerical computing with the release of BLAS and LAPACK in the 1970s and was standard practice in early machine learning toolkits.

• Subtract the max before taking the exp.
• Top exponent is e⁰ = 1; zero overflow, zero crashes.

Consider a batch of logits Z ∈ R^(3 × 4) (N = 3 samples, C = 4 classes).

We wish to compute softmax by dividing each row by its row-sum:

1. The Bug (`keepdims=False`):

```python

row_sums = np.sum(Z, axis=1) # Shape: (3,) -- 1D vector!

probs = Z / row_sums # BROKEN! Triggers unexpected broadcasting!

```

By NumPy broadcasting rules:

• `Z` has shape `(3, 4)`
• `row_sums` has shape `(3,)` → expanded from trailing dimension to `(1, 3)`
• Shapes `(3, 4)` and `(1, 3)` CANNOT broadcast together ⟹ `ValueError: operands could not be broadcast together with shapes (3,4) (3,)`!
• If N = C (e.g. batch size equals class count), it broadcasts across the WRONG axis silently, creating completely corrupted probabilities without throwing an error!

1. The Correct Implementation (`keepdims=True`):

```python

row_sums = np.sum(Z, axis=1, keepdims=True) # Shape: (3, 1) -- 2D column!

probs = Z / row_sums # PERFECT! (3, 4) / (3, 1) -> (3, 4)

```

The column dimension of length 1 broadcasts cleanly across all 4 class columns for each sample independently.

This is one of the top 3 most common bugs encountered by deep learning engineers when implementing custom layers or autograd functions in NumPy and PyTorch.

Travis Oliphant designed NumPy's broadcasting semantics in 2005 based on the original Numeric and C-API conventions, introducing `keepdims` to preserve rank consistency.

• Always keepdims when reducing across an axis.
• Shape (N, 1) divides rows; shape (N,) causes broadcasting chaos.

A geometric function approximation diagram on clean ivory linen (#f7f6ee) with a hairline border.

Top section: Title in technical grotesque: "Universal Approximation: Stacking ReLU Hinges". Subtitle: "How staggered linear ramps combine to approximate arbitrary non-linear curves."

Middle section: A multi-curve synthesis plot on a 2D Cartesian plane:

Top subplot: Three separate individual ReLU neurons with staggered bias thresholds:

• Neuron 1: h₁ = max(0, x - 1) in blue.
• Neuron 2: h₂ = -2 max(0, x - 2) in red.
• Neuron 3: h₃ = 1.5 max(0, x - 3) in green.

Bottom subplot: The linear sum y = w₁ h₁ + w₂ h₂ + w₃ h₃ forms a smooth continuous piecewise linear curve that tightly approximates a target parabolic curve y = f(x).

Annotate: Every added neuron adds an inflection hinge, allowing deep networks to approximate any continuous function to arbitrary precision.

Bottom section: Card: "Cybenko (1989) & Hornik (1991): A single hidden layer with non-linear activations is a universal continuous function approximator."

Clean coordinate axes, high legibility, generous whitespace.

This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.

"Universal Approximation: Stacking ReLU Hinges: Invariance and structure determine stability under spatial and numerical transformations."

A function synthesis geometric diagram in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.

Top section: Title in navy serif: "Universal Approximation: Stacking ReLU Hinges". Subtitle: "How staggered linear ramps combine to approximate any continuous non-linear function."

Middle section: Visual geometric synthesis in three steps:

1. Individual ReLU Neurons: Show three shifted hinge functions:

h₁(x) = max(0, x - 1)

h₂(x) = max(0, x - 2)

h₃(x) = max(0, x - 3).

1. Linear Combination: Weighted sum f(x) = 2 h₁(x) - 3 h₂(x) + 1.5 h₃(x).

Show how subtraction of a steeper hinge creates a localized bump or plateau.

1. Continuous Piecewise Linear (CPWL) Curve: As number of neurons n -> ∞, the CPWL jagged segments become infinitely fine, matching an arbitrary smooth curve (e.g., sine wave or Gaussian peak) with zero error.

Bottom section: Cybenko & Hornik Theorem: "A neural network with just one hidden layer of non-linear activations can approximate any continuous function on a compact domain to arbitrary precision."

Crisp mathematical coordinate plots, distinct colored hinge curves, high educational clarity.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Goodfellow, Bengio, and Courville (2016), Deep Learning, MIT Press ↗

"Universal Function Approximation: Stacking ReLU Hinges: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."