Positional Embedding
Why does standard self-attention treat 'dog bites man' and 'man bites dog' identically without positional encodings?
How this lesson unfolds.
Begin with Why does standard self-attention treat 'dog bites man' and 'man bites dog' identically without positional encodings?. Follow the visual diagrams to build an intuitive mental model.
Why does standard self-attention treat 'dog bites man' and 'man bites dog' identically without positional encodings?
Imagine throwing the words 'dog', 'bites', and 'man' into a kitchen blender and pressing pulse. Self-attention is fundamentally a blender: it views input tokens as an unordered set of floating particles in space, not as an ordered parade. Because dot products (qᵢ . kⱼ) only measure mutual similarity between vector pairs without knowing who stood in front of whom, 'dog bites man' and 'man bites dog' produce mathematically identical representations unless we stamp each particle with a timestamp.
- $$Prove the permutation equivariance of self-attention: Attn(P X) = P Attn(X) for any permutation matrix P.$$
- Learning goalProve that unmasked self-attention is permutation equivariant Attention(P X) = P Attention(X) and explain why sequence order must be injected explicitly.
- Helpful foundationsPermutation matrices and group theory properties · Attention Is All You Need (Vaswani et al., 2017) · Matrix multiplication under row permutations
- Common trapAssuming that causal masking alone fully solves the position problem. While causal masking prevents attending to future tokens, it still does not distinguish whether a past token occurred 1 step ago or 500 steps ago; explicit positional encodings are required.
- Next connectionTo break this permutation symmetry, we must examine how positional embeddings inject sequence order into token vectors.
More context & mechanics
Mathematical proof of permutation equivariance in self-attention:
Let P ∈ R^(T × T) be an arbitrary permutation matrix (P^T P = I).
Let inputs be permuted: X̃ = P X.
- Linear projections permute identically:
Q̃ = P X W_Q = P Q, K̃ = P X W_K = P K, Ṽ = P X W_V = P V
- The attention logits matrix:
Q̃ K̃^T = (P Q)(P K)^T = P (Q K^T) P^T
- Softmax applied row-wise commutes with row permutations:
softmax((P (Q K^T) P^T) / (√(dₖ))) = P softmax((QK^T) / (√(dₖ))) P^T
- Multiplying by permuted values Ṽ = P V:
Attention(Q̃, K̃, Ṽ) = (P softmax((QK^T) / (√(dₖ))) P^T) (P V) = P (softmax((QK^T) / (√(dₖ))) V) = P Attention(Q, K, V)
Thus, self-attention is strictly permutation equivariant: permuting the input rows simply permutes the output rows with zero awareness of spatial order. Without positional signals, syntax and grammar are completely invisible.
This set-invariance property is why transformers can be repurposed for point clouds, molecular graphs, and particle physics without modifications: simply omit positional embeddings, and the network operates as a universal set processor.
Zaheer et al. (2017) explored permutation invariant architectures in 'Deep Sets'. Vaswani et al. recognized that language, unlike point clouds, strictly requires sequential ordering, introducing positional encodings as the solution.
- Attention sees sets, not sequences.
- Without position, 'dog bites man' and 'man bites dog' are the exact same soup.
How does adding a learned or sinusoidal positional tensor to token embeddings break permutation symmetry?
Imagine a line of identical runners wearing blank white jerseys. If the race judge looks at photos, they cannot tell who was running in Lane 1, Lane 2, or Lane 3. Additive positional embedding is pinning a giant fluorescent race number onto each runner's jersey: runner 1 gets +1, runner 2 gets +2, runner 3 gets +3. Now, even though the runners move through the exact same track checkpoints, the race judge's camera can instantly identify their lane and order from the jersey badge.
- $$Demonstrate additive positional injection: $X_{\text{input}} = X_{\text{token}} + W_{\text{pos}}$ and its geometric impact.$$
- Learning goalImplement additive positional embedding addition X_(in) = X_(token) + W_P and evaluate learned vs sinusoidal encodings.
- Helpful foundationsTensor broadcasting rules ((B, T, d) + (1, T, d)) · Attention Is All You Need / Sinusoidal Encodings (Vaswani et al., 2017)
- Common trapWorrying that adding positional vectors directly to token vectors corrupts or destroys the word's semantic meaning. In high-dimensional spaces (d=768 or d=4096), vectors are practically orthogonal; semantic information and positional signals occupy nearly orthogonal sub-spaces without destructive interference.
- Next connectionWhile learned absolute embeddings work well for fixed lengths, how do modern LLMs handle infinitely long context windows? We compare Absolute vs Rotary Position Embeddings.
More context & mechanics
Additive positional encoding creates unique combined coordinates:
X_(in) = X_(tok) + W_P
where X_(tok) ∈ R^(B × T × d) and W_P ∈ R^(T × d) is broadcast across the batch dimension.
Two primary absolute schemes:
- Sinusoidal Positional Encoding (Vaswani et al., 2017):
Deterministic trigonometric formulas:
PE_((pos, 2i)) = sin((pos) / (10000^(2i/d))), PE_((pos, 2i+1)) = cos((pos) / (10000^(2i/d)))
Allows the model to learn relative position because for any fixed offset k, PE_(pos+k) is a linear function of PE_(pos).
- Learned Absolute Positional Embeddings (GPT-2):
A trainable parameter tensor:
`self.pos_emb = nn.Embedding(max_length, d_model)`
W_P ∈ R^(1024 × 768) initialized with N(0, 0.02) and updated via backpropagation.
Why Addition Works (High-Dimensional Orthogonality):
In R^(768), random vectors are almost strictly orthogonal (u · v ≈ 0). When X_(tok) and W_P are summed:
(X_(tok) + W_P) W_Q = X_(tok) W_Q + W_P W_Q
Attention dot products decompose into four terms:
Q K^T = (X_(tok) W_Q + W_P W_Q)(X_(tok) W_K + W_P W_K)^T = X W_Q W_K^T X^T (token-token) + X W_Q W_K^T W_P^T (token-pos) + W_P W_Q W_K^T X^T (pos-token) + W_P W_Q W_K^T W_P^T (pos-pos)
Both semantic and spatial relationships are preserved simultaneously.
Learned absolute position embeddings are used in GPT-2, GPT-3, and BERT. They are simple to implement, fast to execute, and highly effective within a fixed context window length T ≤ 2048.
Gehring et al. (Facebook AI, 2017) used learned positional embeddings in convolutional seq2seq models. Vaswani (2017) introduced sinusoids, and Radford et al. adopted learned position embeddings in GPT-2 (2019).
- Token says WHO; Position says WHERE.
- Addition works because high-dimensional spaces have room for both.
Why do absolute additive position embeddings fail to generalize beyond trained sequence lengths, and how does rotary position geometry solve this?
Imagine numbering runners in a marathon by painting rigid mile-marker numbers on their shirts (Runner at Mile 1 gets #1, Mile 2 gets #2). If the race is extended to 50 miles, runners beyond mile 42 have no shirts, and the numbers tell you nothing about how far apart two runners are unless you do global subtraction. Now imagine instead that each runner carries a pocket watch whose hands advance by an angle proportional to their progress. To see how far apart two runners are, you simply compare the angle between their watch hands. The relative angle between 2 o'clock and 5 o'clock is always 3 hours, whether the race is 5 miles or 500 miles long!
- Learning goalContrast the mathematical and architectural failure modes of absolute additive position embeddings against the relative geometric invariance of RoPE.
- Helpful foundations2D rotation matrices and trigonometry · RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)
- Common trapBelieving that RoPE adds extra learnable parameters to the model. RoPE has zero learnable parameters; it is a deterministic, orthogonal geometric transformation applied directly to Query and Key tensors.
- Next connectionNext, derive the exact 2D rotation matrix, complex number formulation, and dot-product invariance proof in Part B.
More context & mechanics
Positional representation in transformers has evolved through two distinct geometric paradigms:
- The Absolute Additive Paradigm (Vaswani 2017 / GPT-2):
A learned or sinusoidal position vector p_t is directly added to token embedding x_t:
x̃_t = x_t + p_t
- Failure Mode 1 (Semantic Distortion): Addition alters the magnitude and direction of x_t, partially corrupting pure word semantics.
- Failure Mode 2 (Length Extrapolation Failure): If trained on context window T_(max) = 2048, position t = 2049 has no learned vector, causing severe out-of-distribution performance collapse.
- The Rotary Geometric Paradigm (Su et al., 2021 - RoPE):
Instead of modifying token vectors additively, RoPE partitions the vector into 2D orthogonal pairs and rotates each pair by an angle proportional to sequence index m.
- Inner Product Invariance: The dot product between a query at position m and a key at position n depends strictly on the relative displacement m - n.
Virtually all modern open-weight and proprietary frontier foundation models—including Llama-3, Mistral, Gemma, and Qwen-2—have completely replaced absolute positional embeddings with RoPE. RoPE allows these models to natively extrapolate from 8k training contexts to 128k+ production context windows via simple frequency scaling.
Jianlin Su introduced Rotary Position Embedding in 2021 in the seminal paper "RoFormer: Enhanced Transformer with Rotary Position Embedding". Su combined complex number geometry with Givens rotation matrices to solve the longstanding relative position challenge in transformers.
- Do not shift coordinates; rotate dials: Addition corrupts meaning; rotation preserves vector lengths while encoding time.
- Relative displacement is angle difference: The gap between clock hands is invariant to the time of day.
How do we prove that the inner product of RoPE-rotated query and key vectors depends strictly on their relative token separation m - n?
Think of two dancers spinning on ice. Dancer A spins at speed θ and makes m turns; Dancer B spins at the same speed and makes n turns. If you measure their relative alignment, what matters is not how many total revolutions they made from the center of the rink, but the difference in their angles: (m - n)θ. By treating pairs of coordinates as points on a spinning wheel, the dot product between queries and keys depends only on how many beats separate them.
- Learning goalDerive the 2D RoPE orthogonal rotation matrix and prove algebraically that query-key inner products reduce to a function of relative distance m - n.
- Helpful foundationsComplex numbers and Euler's formula · Orthogonal matrices and inner product invariance
- Common trapConfusing RoPE with absolute sinusoidal embeddings. Sinusoids are added (x + p); RoPE is multiplicative and orthogonal (Rx), preserving vector magnitude and enabling seamless length extrapolation.
- Next connectionNext, explore the full multi-channel RoPE frequency spectrum and Givens rotation geometry in Chapter 11 Screen 4.
More context & mechanics
#### 1. 2D Coordinate Rotation Formulation
Let x = (x₁, x₂) ∈ R². RoPE treats (x₁, x₂) as a complex number z = x₁ + i x₂. At position m, the rotated vector is:
R_(\Theta, m) x = cos(mθ) & -sin(mθ) \ sin(mθ) & cos(mθ) x₁ \ x₂
Because R_(\Theta, m) is an orthogonal matrix (R^T R = I), vector norm is perfectly preserved:
\|R_(\Theta, m) x\|₂ = \|x\|₂
Let query q reside at position m and key k reside at position n. Their rotated representations are qₘ = Rₘ q and kₙ = Rₙ k.
The attention score is the inner product:
⟨ qₘ, kₙ ⟩ = (Rₘ q)^T (Rₙ k) = q^T ( Rₘ^T Rₙ ) k
Using the standard trigonometric product identities:
Rₘ^T Rₙ = cos(mθ) & sin(mθ) \ -sin(mθ) & cos(mθ) cos(nθ) & -sin(nθ) \ sin(nθ) & cos(nθ) = cos((n - m)θ) & -sin((n - m)θ) \ sin((n - m)θ) & cos((n - m)θ) = R_(n - m)
Therefore:
⟨ qₘ, kₙ ⟩ = q^T R_(n - m) k = g(q, k, n - m)
The attention compatibility depends strictly on the relative distance (n - m), achieving exact shift invariance! \blacksquare
Let θ = (π) / (4), m = 3, n = 1. The relative displacement is m - n = 2.
Relative angle Δ = (1 - 3)(π) / (4) = -(π) / (2).
R_(1 - 3) = cos(-π/2) & -sin(-π/2) \ sin(-π/2) & cos(-π/2) = 0 & 1 \ -1 & 0
For q = [1, 0]^T and k = [1, 0]^T:
⟨ q₃, k₁ ⟩ = [1, 0] 0 & 1 \ -1 & 0 1 \ 0 = [1, 0] 0 \ -1 = 0
The inner product accurately reflects the 90-degree relative rotation induced by the 2-token separation.
In high-throughput inference engines like vLLM and TensorRT-LLM, RoPE rotation is implemented in a single fused CUDA kernel. To avoid full matrix multiplications, coordinates are updated via element-wise vector operations:
q_(rot) = q \odot cos(mθ) + q̃ \odot sin(mθ)
where q̃ = [-q₂, q₁, -q₄, q₃, ṡ].
The mathematical foundation of RoPE relies on Euler's formula e^(iθ) = cosθ + isinθ (1748). Jianlin Su's key insight in 2021 was recognizing that multiplying query and key by complex exponentials e^(imθ) and e^(−inθ) yields e^(i(m-n)θ), directly mapping relative position into attention dot products.
- Rₘ^T Rₙ = R_(n-m): Transposing the query rotation turns absolute positions into relative displacement.
- Inner product preserves distance: Lengths never change; angles measure token separation.
How does Rotary Position Embedding (RoPE) encode relative word distances into self-attention dot products through 2D Givens rotation matrices?
Imagine a grand clock tower with 384 pairs of clock hands. When the first word of a sentence arrives at position m=1, each clock hand ticks forward by a small angle. When word two arrives at position m=2, its hands tick forward twice as far. Words further down the page spin the hands even further. When Word 5 (Query) compares itself with Word 3 (Key), the angle between their clock hands is simply the difference between their ticks: 5 - 3 = 2. Regardless of whether the phrase appears at the top of page 1 or the bottom of page 500, the angular distance between words two steps apart is identical.
- Learning goalCalculate 2D Givens rotational matrices and prove analytically that the RoPE inner product preserves relative position invariance.
- Helpful foundationsRoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021) · Llama 2: Open Foundation and Fine-Tuned Chat Models (Touvron et al., Meta, 2023) · LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (Ding et al., Microsoft, 2024)
- Common trapAssuming RoPE adds vectors like classical positional embeddings. RoPE performs multiplicative orthogonal rotations (Rₘ x), preserving vector norm ||Rₘ x|| = ||x|| without distorting semantic vector lengths.
- Next connectionWith token identities and relative positions encoded into geometry, Chapter 12 summarizes the complete preprocessing pipeline before passing tensors into the Transformer Attention layers.
More context & mechanics
RoPE groups the d-dimensional query vector q ∈ R^d into d/2 two-dimensional subvectors (q_(2i-1), q_(2i)). At position m, each 2D subvector is multiplied by an orthogonal 2D Givens rotation matrix:
R_(\Theta, m)^((i)) = cos(m θᵢ) & -sin(m θᵢ) \ sin(m θᵢ) & cos(m θᵢ) , θᵢ = 10000^(-2(i-1)/d)
When computing the dot product between query q at position m and key k at position n:
⟨ Rₘ q, Rₙ k ⟩ = (Rₘ q)^T (Rₙ k) = q^T Rₘ^T Rₙ k = q^T R_(n-m) k
Because Rₘ^T Rₙ = R_(n-m), the scalar attention score depends strictly on the relative distance m - n, completely independent of absolute position!
Toy Numerical Trace: Let d = 2 with θ = (π) / (4) radians (45^∘).
Position m = 3, position n = 1. Relative distance m - n = 2.
Rotated query angle: 3 × (π) / (4) = (3π) / (4) (135^∘).
Rotated key angle: 1 × (π) / (4) = (π) / (4) (45^∘).
Angular difference: (3π) / (4) - (π) / (4) = (2π) / (4) = (π) / (2) (90^∘).
Let q = [1, 0]^T and k = [1, 0]^T:
R₃ q = cos(135^∘) & -sin(135^∘) \ sin(135^∘) & cos(135^∘) 1 \ 0 = -(√(2)) / (2) \ (√(2)) / (2)
R₁ k = cos(45^∘) & -sin(45^∘) \ sin(45^∘) & cos(45^∘) 1 \ 0 = (√(2)) / (2) \ (√(2)) / (2)
⟨ R₃ q, R₁ k ⟩ = (-(√(2)) / (2))((√(2)) / (2)) + ((√(2)) / (2))((√(2)) / (2)) = -1/2 + 1/2 = 0
Since cos(π/2) = 0, the dot product reflects exact orthogonality at relative distance 2.
RoPE is the standard positional encoding adopted by virtually all modern open-weight LLMs, including Meta's Llama 2 and Llama 3, Mistral, Qwen, Gemma, and DeepSeek. Its relative formulation enables context-window extension techniques like YaRN and LongRoPE.
Invented in 2021 by Jianlin Su in 'RoFormer: Enhanced Rotary Position Embedding', resolving the rigid context limitations of Vaswani's 2017 sinusoidal encodings.
- RoPE rotates 2D pairs on clock dials; attention scores measure the relative angle between the hands.
- Absolute vectors rotate; relative dot products endure.
How do Vaswani sinusoidal positional encodings encode sequence positions as geometric harmonics, and what are their context extrapolation limits?
Imagine an old-fashioned grandfather clock with multiple hands: a microsecond hand that spins like a propeller, a second hand that ticks once a second, a minute hand that turns slowly, and an hour hand that barely inches forward. At any exact moment in time, the combined angles of all four hands form a completely unique geometric signature. The fast hands tell you whether two moments are 1 second apart, while the slow hands tell you what part of the day you are in. Vaswani sinusoidal embeddings equip each token with 384 pairs of clock hands: Channel 0 ticks so fast it completes a cycle every 6 tokens, while Channel 383 turns so slowly its wavelength spans 61,342 tokens. Together, these 768 harmonic channels encode an exact positional timestamp for every token in the sequence.
- $$Vaswani sinusoidal encodings interleave sine and cosine waves across geometric frequency bands $10000^{-2i/d}$, creating unique deterministic positional fingerprints across channels.$$
- Learning goalState the Vaswani sinusoidal positional encoding equations and calculate channel frequencies, wavelengths, and extrapolation boundaries.
- Helpful foundationsTrigonometric functions (sine and cosine) · Frequency and wavelength relationships
- Common trapBelieving sinusoidal encodings allow transformers to extrapolate to infinite context lengths out-of-the-box. While the encoding itself never overflows mathematically, the neural network's learned attention weights fail to generalize to unseen phase combinations beyond the training window.
- Next connectionNext, examine Rotary Position Embeddings (RoPE) and how 2D complex plane rotations enforce relative distance in Screen 11-003B.
More context & mechanics
#### 1. The Vaswani Sinusoidal Formulation (Vaswani et al., 2017)
For position pos ∈ N and channel index i ∈ [0, d_(model)/2 - 1]:
PE_((pos, 2i)) = sin( (pos) / (10000^2i / d_(model)) )
PE_((pos, 2i+1)) = cos( (pos) / (10000^2i / d_(model)) )
Where d_(model) = 768, yielding 384 distinct frequency bands.
The angular frequency ωᵢ and wavelength λᵢ are:
ωᵢ = 10000^(-2i / 768), λᵢ = (2π) / (ωᵢ) = 2π · 10000^(2i / 768)
- Channel i = 0 (Highest Frequency):
ω₀ = 1.0, λ₀ = 2π ≈ 6.28 tokens
Oscillates rapidly to distinguish immediate adjacent tokens.
- Channel i = 32 (Moderate Frequency):
ω_(32) ≈ 0.464, λ_(32) ≈ 13.54 tokens
- Channel i = 128 (Low Frequency):
ω_(128) ≈ 0.0464, λ_(128) ≈ 135.37 tokens
- Channel i = 383 (Lowest Frequency):
ω_(383) ≈ 0.0001024, λ_(383) ≈ 61,342.74 tokens
Nearly constant over standard context windows, tracking macro-level position.
While PE(pos) is mathematically defined for all pos > 0 to infinity, empirical model performance degrades sharply when evaluated beyond the training context length (e.g., L > 1024). Because downstream attention projection weights never observed the high-magnitude phase alignments of unseen positions during training, learned attention distributions collapse.
Used in the original Transformer (Attention Is All You Need), BERT, and early encoder-decoder models (T5). Modern generative models largely replace it with Rotary Position Embeddings (RoPE).
Introduced by Vaswani et al. in 2017 to eliminate learned positional embedding tables and theoretically allow models to generalize to sequence lengths longer than those observed during training.
- Sine for even, cosine for odd: Interleaved harmonic pairs.
- Geometric decay of frequency: Low channels tick fast; high channels move like glaciers.
How does RoPE enforce relative positional distance through 2D Givens rotations, and why does their dot product depend exclusively on (m - n)?
Imagine you and a friend are dancing on a giant spinning carousel. If you both dance near the center of the carousel, your absolute positions in the room change continuously as the turntable spins. But if your friend is standing 5 feet in front of you facing your direction, your relative distance and angle to each other remain completely identical regardless of how fast the room revolves. RoPE does not add static positional stamps to word vectors. Instead, it rotates the Query vector by angle mθ and rotates the Key vector by angle nθ. When you compute their dot product, the absolute carousel rotation cancels out completely, leaving only the relative angle (m - n)θ. A 5-word gap means the exact same relative rotation whether it occurs at token 10 or token 30,000.
- Learning goalDerive the mathematical proof showing why the dot product of RoPE-transformed query and key vectors depends strictly on relative token distance (n - m).
- Helpful foundations2D rotation matrices and orthogonality · Complex numbers and Euler's formula
- Common trapAssuming RoPE is applied to the input token embeddings before the transformer layers. RoPE is applied directly to Query and Key projections inside every self-attention layer right before computing attention scores.
- Next connectionNext, transition to Chapter 13 to discover why dynamic Attention mechanisms overcome the limitations of static word embeddings.
More context & mechanics
#### 1. Complex Coordinate Pair Partition
For a query vector q ∈ R^(dₖ) (where dₖ is head dimension), RoPE partitions coordinates into dₖ / 2 two-dimensional pairs, treating each pair as a complex number:
zᵢ = q_(2i) + j · q_(2i+1) ∈ C (i ∈ [0, dₖ/2 - 1])
At token position m, RoPE applies a 2D rotation matrix R_(\Theta, m)^((i)) to each pair:
R_(\Theta, m)^((i)) = cos(mθᵢ) & -sin(mθᵢ) \ sin(mθᵢ) & cos(mθᵢ)
Where base frequencies follow: θᵢ = 10000^(-2i/dₖ).
The rotated query and key vectors become:
q̃ₘ = R_(\Theta, m) qₘ, k̃ₙ = R_(\Theta, n) kₙ
Computing the attention score between query at position m and key at position n:
⟨ q̃ₘ, k̃ₙ ⟩ = q̃ₘ^T k̃ₙ = (R_(\Theta, m) q)^T (R_(\Theta, n) k) = q^T R_(\Theta, m)^T R_(\Theta, n) k
Because rotation matrices are orthogonal, R_(\Theta, m)^T = R_(\Theta, -m):
R_(\Theta, m)^T R_(\Theta, n) = R_(\Theta, n - m)
Substituting this identity:
⟨ q̃ₘ, k̃ₙ ⟩ = q^T R_(\Theta, n - m) k = g(q, k, n - m)
The attention dot product depends exclusively on relative distance (n - m), completely decoupling attention scores from absolute positions m and n!
RoPE is the universal positional embedding architecture across modern open and frontier LLMs: LLaMA-1/2/3, Mistral, Gemma, Falcon, and Qwen.
Jianlin Su introduced Rotary Position Embedding in 2021 (RoFormer: Enhanced Transformer with Rotary Position Embedding). It solved the context length limitation of absolute embeddings, enabling context extension from 2K to 32K, 128K, and 1M tokens via frequency scaling (YaRN, LongLoRA).
- Rotate the queries and keys, cancel the absolute angles: Rₘ^T Rₙ = R_(n-m).
- Relative distance via 2D rotation: Pure rotational harmony in the complex plane.
How do block-diagonal Givens rotation matrices decompose high-dimensional vectors into independent planar rotating trajectories?
Imagine a high-dimensional ballroom where 768 dancers stand in line. Instead of trying to spin the entire line as one rigid rod, you pair the dancers up into 384 couples. Couple 1 spins fast (one full spin every 2 steps). Couple 2 spins slightly slower. Couple 384 spins like a giant grandfather clock (one spin every 10,000 steps). Every couple spins exclusively inside their own private 2D floor tile without bumping into neighboring pairs. This block-diagonal rotation is a Givens decomposition.
- Learning goalConstruct block-diagonal Givens rotation matrices and prove that vector Euclidean norms are strictly invariant under RoPE.
- Helpful foundationsOrthogonal matrices · Computation of Plane Unitary Rotations (Givens, 1958)
- Common trapThinking RoPE rotates the entire vector in a single d-dimensional space; it breaks the vector into d/2 independent 2D planes, rotating each plane at a distinct geometric frequency.
- Next connectionHow does attention measure how much two vectors in these rotated spaces point toward each other? We analyze dot product geometric projection.
More context & mechanics
A Givens rotation acts on an isolated 2D coordinate plane (2i - 1, 2i) of R^d:
R_(θᵢ, m) = cos(mθᵢ) & -sin(mθᵢ) \ sin(mθᵢ) & cos(mθᵢ)
The full rotary matrix Rₘ ∈ R^(d × d) is the direct sum (block-diagonal):
Rₘ = oplus_(i=1)^(d/2) R_(θᵢ, m) = R_(θ₁, m) & 0 & ṡ & 0 \ 0 & R_(θ₂, m) & ṡ & 0 \ … & … & ⋱ & … \ 0 & 0 & ṡ & R_θ_(d/2), m
Key mathematical properties:
- Orthogonality: Rₘ^T Rₘ = I_d (preserves vector norm \|Rₘ x\| = \|x\|).
- Determinant: det(Rₘ) = 1 (pure rotation with zero reflection or volume distortion).
- Group homomorphism: Rₘ Rₙ = R_(m+n).
RoPE is executed in Triton and CUDA kernels using vectorized complex arithmetic:
x_(rotated) = x \odot cos(mθ) + rotate\_half(x) \odot sin(mθ)
Running at memory-bandwidth speed on NVIDIA Tensor Cores.
Wallace Givens introduced planar Givens rotations in 1958 at Oak Ridge National Laboratory for numerical matrix diagonalization. Jianlin Su adapted them in 2021 for Transformer positional encoding.
- Block-diagonal Givens: 384 independent 2D spinning clocks.
- Norm is conserved; distance is encoded as relative phase difference.
How does Learned vs. Sinusoidal Absolute Positional Embeddings operate mathematically at scale?
Two contrasting positional encoding panels:
- Learning goalMaster the algebraic and geometric derivation of Learned vs. Sinusoidal Absolute Positional Embeddings.
- Helpful foundationsFoundations of Build Llm From Scratch · Coordinate Invariance & Geometric Projections · Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
- Common trapAssuming Learned vs. Sinusoidal Absolute Positional Embeddings is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
A comparative mathematical signal diagram in square 1:1 format on warm cream parchment (#fcf0db) with a sepia frame.
Top section: Title in navy serif: "Positional Embeddings: Injecting Sequence Order". Subtitle: "Why Transformers are permutation-invariant and how positional vectors restore order."
Middle section: Two contrasting positional encoding panels:
Left Panel (Sinusoidal - Vaswani et al. 2017):
Formula: PE_((pos, 2i)) = sin(pos / 10000^(2i/d)), PE_((pos, 2i+1)) = cos(pos / 10000^(2i/d)).
Show continuous multi-frequency wave harmonics oscillating across hidden dimensions. Fixed, deterministic, enables relative distance extrapolation.
Right Panel (Learned Positional Embeddings - GPT-2 / GPT-3):
Trainable parameter matrix W_(pos) ∈ R^L_(max) × D. Each position index 0, 1, ..., L-1 learns a dedicated dense vector.
Bottom section: Fusion formula: "Input Representation = Token Embedding + Positional Embedding: x_t = W_(token)[id_t] + W_(pos)[t]."
Smooth trigonometric wave visualizations, clear matrix strip layout, elegant square frame.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
"Learned vs. Sinusoidal Absolute Positional Embeddings: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."