Simplified Attention
What is the simplest mathematical formulation of self-attention without trainable weights?
How this lesson unfolds.
Begin with What is the simplest mathematical formulation of self-attention without trainable weights?. Follow the visual diagrams to build an intuitive mental model.
What is the simplest mathematical formulation of self-attention without trainable weights?
Imagine each word vector as an arrow pointing into space. The dot product is the shadow or projection of one arrow onto another. When two arrows point in nearly the same direction, like [2, 1] and [2, 1], their dot product is large and positive: 2(2) + 1(1) = 5. When perpendicular, it is 0; when pointing in opposite directions, it is negative. Without any trainable matrices, the dot product matrix Omega = X X^T serves as a raw geometric similarity scoreboard between every token pair.
- Learning goalCompute raw pairwise dot product similarity scores omega_ij = xᵢ^T xⱼ for a sequence matrix X in R^(T x d).
- Helpful foundationsAttention Is All You Need (Vaswani et al., 2017) · Linear Algebra 18.06: Dot Products & Projections (Strang, MIT) · Euclidean Vector Spaces & Inner Products
- Common trapThinking dot product similarity is bounded between 0 and 1. Dot products can take any real value in (-infinity, +infinity) depending on vector magnitudes; they are unnormalized geometric projections, not probabilities.
- Next connectionBecause raw dot products are unbounded and do not sum to one, we must pass each row through the Softmax function to obtain valid attention weights.
More context & mechanics
Given an input sequence of T token embedding vectors arranged as rows of a matrix X ∈ R^(T × d):
X = - & x₁^\top & - \ - & x₂^\top & - \ & … & \ - & x_T^\top & -
The simplified attention mechanism computes raw compatibility scores directly via the matrix product:
Ω = X X^\top ∈ R^(T × T)
Each individual entry ω_(ij) = xᵢ^\top xⱼ = ∑_(m=1)^d x_(im) x_(jm) = \|xᵢ\| \|xⱼ\| cos(θ) measures the geometric alignment between token i and token j.
For example, with T=3 tokens in d=2:
x₁ = [1, 0], x₂ = [0, 1], x₃ = [1, 1]
The unnormalized similarity score between token 1 and token 3 is ω_(13) = 1(1) + 0(1) = 1, whereas between token 1 and token 2 it is ω_(12) = 1(0) + 0(1) = 0.
Simplified attention reveals the essential inner-product engine at the heart of Transformers before introducing Query, Key, and Value projections. It illustrates why dense embeddings naturally encode semantic similarity through angular proximity.
The dot product as an algebraic measure of vector projection was formulated by Josiah Willard Gibbs and Oliver Heaviside in the 1880s. In modern deep learning, Sukhbaatar et al. (2015) in End-to-End Memory Networks demonstrated that inner products could retrieve relevant associative memories without recurrence.
- Golden Axiom: "Dot product is directional alignment."
- Pointing together yields large positive scores; pointing orthogonally yields zero; pointing apart yields negative scores.
Why do raw dot products need row-wise Softmax normalization before computing the context vector?
Imagine blending fruit smoothies. If you combine ingredients using raw scores like 10, -3, and 25, what recipe do you follow? You cannot add negative strawberries or an unbounded bucket of bananas. Softmax exponentiates the scores to make them all positive (e^(omegaⱼ) > 0), then divides by their sum. The resulting percentages (e.g. 0.05, 0.00, 0.95) specify an exact, stable recipe for mixing the vector representations into a coherent new flavor.
- Learning goalApply row-wise Softmax normalization to the raw score matrix Omega and compute the resulting context matrix Z = softmax(Omega) X.
- Helpful foundationsAttention Is All You Need (Vaswani et al., 2017) · The Softmax Activation Function · Convex Combinations in Vector Spaces
- Common trapNormalizing across the entire T x T matrix instead of row-by-row. Attention weights must sum to 1 along each individual query token's row (sumⱼ alpha_(ij) = 1), not across all tokens globally.
- Next connectionWhile simplified attention shows how dot products and Softmax work, raw input vectors cannot differentiate between searching for information and providing information—motivating trainable Q, K, V projection matrices.
More context & mechanics
Raw dot products ω_(ij) range over (-∞, +∞) and depend on vector magnitudes. To convert them into blending coefficients, we apply the Softmax function row-by-row:
α_(ij) = (exp(ω_(ij))) / (∑_(k=1)^T exp(ω_(ik)))
This normalization guarantees two vital mathematical properties:
- Positivity: α_(ij) > 0 for all i, j.
- Conservation of Mass: ∑_(j=1)^T α_(ij) = 1 for every row i.
The contextualized representation for token i is then computed as the convex combination:
zᵢ = ∑_(j=1)^T α_(ij) xⱼ
In compact matrix form, where A = softmax(Ω, dim=-1) ∈ R^(T × T):
Z = A X ∈ R^(T × d)
This row-wise normalization guarantees that activation vectors maintain a stable scale throughout deep transformer layers, preventing numerical explosion as sequences pass through multiple consecutive attention blocks.
The Softmax function originates in Ludwig Boltzmann's 1868 statistical mechanics, where the Boltzmann distribution gave the probability that a physical system occupies a particular energetic microstate. John Bridle introduced Softmax into artificial neural networks in 1989 as a differentiable winner-take-all mechanism.
- Golden Axiom: "Softmax turns raw energy into an attention budget."
- Every row receives exactly 100% of an attention budget to divide among the sequence tokens.
Why does raw exponentiation crash floating-point registers, and how does shifting by the maximum score eliminate numerical overflow?
Think of an altimeter on Mount Everest. If your instruments max out at 10,000 meters, measuring an 8,848-meter peak causes digital redlining. By measuring every altitude relative to the peak summit (0 m at the crest, negative below), no readout ever exceeds 0, and relative height differences remain perfectly preserved.
- $$Subtracting the maximum logit m = max(z) shifts all exponents into (-inf, 0], guaranteeing exp(z_i - m) <= 1.0 and eliminating IEEE-754 overflow while preserving exact probability ratios.$$
- Learning goalProve the algebraic identity Softmax(z) = Softmax(z - max(z)), implement the safe logit subtraction in PyTorch, and verify zero numerical overflow in FP16/FP32 precision.
- Helpful foundationsDeep Learning Book: Numerical Computation (Goodfellow, Bengio & Courville, MIT Press, 2016) · FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022) · Attention Is All You Need (Vaswani et al., 2017)
- Common trapConfusing logit shifting with probability distortion. Subtracting constant m scales both numerator and denominator by identical factor e^(-m), leaving output softmax probabilities mathematically identical to infinite-precision arithmetic.
- Next connectionNow that raw dot products can be normalized into stable weights, Chapter 15 introduces trainable Query, Key, and Value matrices to give self-attention dynamic semantic roles.
More context & mechanics
In IEEE-754 32-bit floating-point arithmetic (FP32), numbers overflow to `+inf` when x > ln(3.4028 × 10^(38)) ≈ 88.72. In 16-bit half precision (FP16), the ceiling is even lower: x > ln(65504) ≈ 11.09. If unnormalized attention scores ωᵢ ≥ 89, calculating e^(ωᵢ) produces `inf`, and evaluating inf/inf yields `NaN` (Not a Number), poisoning the entire computational graph.
The algebraic proof of numerical equivalence demonstrates that shifting logits by any scalar c preserves exact probabilities:
Softmax(z)ᵢ = (e^(zᵢ)) / (∑_(j=1)^N e^(zⱼ)) = (e^(zᵢ - c) · e^c) / (∑_(j=1)^N (e^(zⱼ - c) · e^c)) = (e^(zᵢ - c) · e^c) / (e^c · ∑_(j=1)^N e^(zⱼ - c)) = (e^(zᵢ - c)) / (∑_(j=1)^N e^(zⱼ - c))
Setting c = m = max_(k=1)^N zₖ guarantees that the maximum shifted value is m - m = 0, so e⁰ = 1.0. All other shifted values are strictly non-positive (zᵢ - m ≤ 0), bounding every exponential term in the interval (0, 1.0]. Overflow is mathematically impossible.
Worked toy numerical trace:
Let raw logits z = [100.0, 102.0, 105.0].
- Unshifted exponentiation: e^(100) ≈ 2.69 × 10^(43), e^(105) ≈ 4.00 × 10^(45) (instantly overflows in FP16).
- Max logit extraction: m = max(z) = 105.0.
- Shifted vector: z - m = [100 - 105, 102 - 105, 105 - 105] = [-5.0, -3.0, 0.0].
- Safe exponentials: e^(−5) ≈ 0.006738, e^(−3) ≈ 0.049787, e⁰ = 1.000000.
- Denominator sum: S = 0.006738 + 0.049787 + 1.000000 = 1.056525.
- Final probabilities: p = [0.00638, 0.04712, 0.94650], summing exactly to 1.0000.
Every deep learning framework applies this transformation internally inside `torch.softmax` and `torch.nn.functional.cross_entropy`. In frontier low-latency attention kernels such as FlashAttention-2 (Tri Dao, 2023), the max trick is tracked online across tiles within on-chip GPU SRAM without writing unnormalized score matrices back to high-bandwidth VRAM.
John S. Bridle introduced the softmax function to neural classification in 1989/1990 ('Probabilistic Interpretation of Feedforward Classification Network Outputs'), adapting Ludwig Boltzmann's 1868 kinetic gas energy distributions. The numerical max subtraction technique emerged from early numerical linear algebra libraries (BLAS/LAPACK) to stabilize floating-point partition function calculations.
- Golden Axiom: "Subtract the peak before exponentiating: relative differences determine probabilities, while negative exponents guarantee finite numbers."
- e⁰ = 1.0 is your safe ceiling: no shifted exponent can ever exceed unity.
What is the mathematical foundation and geometric intuition of Self-Attention: Dynamic Message Passing on Complete Graphs?
Understand Self-Attention: Dynamic Message Passing on Complete Graphs visually: Transformers are Graph Neural Networks where edge weights are computed dynamically from data.. Every transformation preserves core structural invariants while mapping coordinate spaces.
- Learning goalMaster the algebraic and geometric derivation of Self-Attention: Dynamic Message Passing on Complete Graphs.
- Helpful foundationsLinear Algebra Fundamentals · Matrix Transformations · Vector Spaces
- Common trapAssuming Self-Attention: Dynamic Message Passing on Complete Graphs is merely a computational trick rather than an invariant spatial law.
- Next connectionConnects to upstream spectral theory and downstream tensor operations.
More context & mechanics
A graph neural network perspective diagram on natural warm linen (#f5f4e9) with a 1px slate frame.
Top section: Title in bold sans-serif: "Self-Attention: Dynamic Message Passing on Complete Graphs". Subtitle: "Transformers are Graph Neural Networks where edge weights are computed dynamically from data."
Middle section: A clean 4-node complete directed graph with circular nodes representing tokens: ["The", "bank", "of", "river"].
Draw directed edges connecting every token to every other token.
Edge thickness indicates attention weight α_(ij) = softmax(qᵢ · kⱼ / √(dₖ)).
Highlight the heavy edge connecting "bank" to "river" (α = 0.82), pulling the contextual representation of "river" into "bank" to resolve polysemy (river bank vs financial bank).
Bottom section: Formula card: "xᵢ^((l+1)) = ∑_(j=1)^T α_(ij) (xⱼ^((l)) W_V) ⟹ Attention is an anisotropic message-passing operator over a dynamic complete graph."
Clear circular nodes, variable-width edge arrows, ample breathing room.
This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.
"Self-Attention: Dynamic Message Passing on Complete Graphs: Invariance and structure determine stability under spatial and numerical transformations."
How does Self-Attention as a Complete Directed Graph operate mathematically at scale?
A 4-node complete directed graph representing tokens [The, cat, sat, down]:
- Learning goalMaster the algebraic and geometric derivation of Self-Attention as a Complete Directed Graph.
- Helpful foundationsFoundations of Build Llm From Scratch · Coordinate Invariance & Geometric Projections · Vaswani et al. (2017), Attention Is All You Need, NeurIPS
- Common trapAssuming Self-Attention as a Complete Directed Graph is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
A graph neural network concept diagram in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.
Top section: Title in navy serif: "Self-Attention: Dynamic Message Passing on Complete Graphs". Subtitle: "Transformers as Graph Neural Networks where attention weights define edge affinities."
Middle section: A 4-node complete directed graph representing tokens [The, cat, sat, down]:
Every node has directed edges pointing to every other node (and a self-loop to itself).
Annotate edge weights dynamically computed via dot product:
"A_(ij) = Softmax((qᵢ · kⱼ) / (√(dₖ)))".
Contrast with standard Graph Neural Networks:
- Standard GNN: Static, hardcoded edge adjacency matrix A.
- Transformer Self-Attention: Fully connected complete graph where edge weights are continuously computed on-the-fly from token content!
Bottom section: Causal masking: "In decoder LLMs, causal masking removes forward edges, turning the complete graph into a lower-triangular Directed Acyclic Graph (DAG)."
Clean network graph layout, glowing directed edges with probability labels, balanced square composition.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
"Self-Attention as a Complete Directed Graph: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."