Causal Attention
Why must generative LLMs conceal future token positions, and how does causal masking enforce autoregression?
How this lesson unfolds.
Generative language models predict the next token given only the past. Follow this visual module to see how causal masking prevents information leakage during parallel training using a lower-triangular mask of negative infinity.
Why must generative LLMs conceal future token positions during parallel training?
Imagine taking an open-book exam where the questions ask you to predict what happens next in a mystery story. If the textbook contains the whole story and you can flip ahead to the final chapter, you will simply copy the murderer's name without ever learning deductive reasoning. To teach a generative model to anticipate what comes next, you must place an opaque blindfold over every word that occurs after the current cursor position.
- Learning goalFormulate the autoregressive sequence likelihood factorization P(w₁, ..., w_T) = prod P(w_t | w_<t) and prove why bidirectional attention causes label leakage.
- Helpful foundationsAttention Is All You Need (Vaswani et al., 2017) · Language Models are Unsupervised Multitask Learners / GPT-2 (Radford et al., 2019) · The Chain Rule of Probability & Autoregressive Modeling
- Common trapConfusing generative autoregressive LLMs (like GPT) with masked language models (like BERT). BERT sees both left and right context simultaneously for classification; GPT must strictly predict the future from the past.
- Next connectionTo computationally enforce this past-only constraint within standard parallel matrix multiplication, we inject a lower-triangular masking matrix into the raw attention scores.
More context & mechanics
By the chain rule of probability, the joint probability of any token sequence (w₁, w₂, ṡ, w_T) factorizes strictly causally:
P(w₁, w₂, ṡ, w_T) = ∏_(t=1)^T P(w_t | w₁, w₂, ṡ, w_(t-1)) = ∏_(t=1)^T P(w_t | w_(<t))
In an autoregressive decoder, the training objective is to maximize the log-likelihood of each next token w_t given only the history w_(<t):
L(θ) = -∑_(t=1)^T log P_θ(w_t | w_(<t))
If the attention layer allowed token t to compute dot products with keys k_(t+1), k_(t+2), ṡ, the network would find a degenerate shortcut: simply copying the identity of w_(t+1) directly into the output representation z_t. The training loss would drop to zero, but the model would learn zero general linguistic predictive capacity.
Causal masking is the fundamental defining constraint of all autoregressive foundation models, including OpenAI's GPT series, Anthropic's Claude, Meta's LLaMA, and Mistral.
Claude Shannon (1948) established n-gram autoregressive modeling in A Mathematical Theory of Communication. In 2018, Radford et al. at OpenAI realized that applying a causal mask to Vaswani's Transformer decoder produced a generative model (GPT-1) capable of zero-shot task transfer.
- Golden Axiom: "The future cannot teach the past how to predict the future."
- In causal autoregression, token t may only look left (j ≤ t), never right (j > t).
Why must autoregressive models strictly enforce causal masking, and how does lower-triangular masking protect temporal order?
Imagine taking an open-book history exam where the goal is to predict what happens next in an unfolding crisis. If the textbook publisher accidentally prints the answers for tomorrow's events right in the margin of today's chapter, you will simply copy the future outcome without ever learning the underlying historical causes. When you are later placed in the real world where tomorrow's newspaper has not been printed yet, you are completely helpless. Causal masking is a strict cardboard blindfold: it covers all future chapters so you are forced to genuinely deduce what comes next using only what has occurred so far.
- Learning goalExplain the autoregressive factorization of sequence probabilities and formulate the lower-triangular causal attention mask to prevent future information leakage.
- Helpful foundationsAutoregressive generation paradigm · Chain rule of probability
- Common trapBelieving that autoregressive models must be trained one token at a time sequentially. Causal masking enables fully parallel O(1) training sweeps across all T positions simultaneously while mathematically preserving sequential causality.
- Next connectionNext, derive the exact pre-softmax additive mask calculus and row-normalization mechanics in Part B.
More context & mechanics
Autoregressive language models predict the joint probability of a sequence using the chain rule of probability:
P(x₁, x₂, ṡ, x_T) = ∏_(t=1)^T P(x_t | x₁, x₂, ṡ, x_(t-1))
- The Bidirectional Leakage Trap:
Standard self-attention allows all tokens to attend to all other tokens. If token t=2 ("sat") can attend to token t=5 ("mat"), the network achieves trivial near-zero training loss by simply memorizing the future token, learning zero predictive reasoning.
- The Causal Lower-Triangular Contract:
To train an autoregressive model in parallel on full sequences, we apply a lower-triangular mask matrix:
- Token 1 attends only to Token 1.
- Token 2 attends to Tokens 1 and 2.
- Token T attends to Tokens 1, 2, ṡ, T.
Every row i is strictly forbidden from looking at columns j > i.
All decoder-only generative models (GPT-4, Llama-3, Claude, Gemini) use causal masking during pretraining. This enables parallel training across 8,192+ tokens simultaneously while ensuring that during autoregressive generation (token-by-token inference), the model's environment perfectly matches its training regime.
The autoregressive property in probability theory traces back to Andrey Markov (1906). The lower-triangular causal mask was first adapted to transformer architectures in the original 2017 paper for the decoder stack, and subsequently adopted as the sole architectural engine in OpenAI's GPT-1 (Radford et al., 2018).
- Past can see past; future is forbidden: Time flows forward; information must never flow backward.
- Lower-triangular means lawful: Zeros on and below the diagonal; negative infinity strictly above.
How does adding -∞ to the upper-triangle of pre-softmax attention logits enforce exact zero weights after exponentiation?
Think of voting in an election where some candidates are disqualified for being underage. Rather than designing a complex custom voting machine with special gears that physically blocks those levers, the election board simply programs the digital voting machine to subtract one billion votes from disqualified names before tallying percentages. When percentages are calculated, the disqualified candidates get exactly 0.0000\% of the vote, without breaking the standard counting software.
- $$Setting upper-triangular logits to $-\infty$ before softmax guarantees that $\exp(-\infty) = 0$, mathematically eradicating future attention weights while maintaining gradient differentiability.$$
- Learning goalProve that adding -∞ to upper-triangular logits guarantees exact zero future attention weights, and trace numerical row-wise softmax calculations.
- Helpful foundationsCausal masking concept · Softmax limits as x → -∞
- Common trapAttempting to zero out future attention weights after softmax. If you manually set weights to zero after softmax, the remaining weights no longer sum to 1.0, destroying the probability simplex contract.
- Next connectionNext, explore how multi-head attention splits the embedding dimension into parallel subspaces in Chapter 17 Screen 2.
More context & mechanics
#### 1. Mathematical Formulation of the Causal Mask
Let S = (Q K^T) / (√(dₖ)) ∈ R^(T × T) be the raw attention score matrix.
We define the additive causal mask matrix M ∈ R^(T × T) as:
M_(i, j) = 0 & if j ≤ i \ -∞ & if j > i
The masked attention logits are computed by simple matrix addition:
S̃ = S + M
Applying softmax across rows:
A_(i, j) = (exp(S̃_(i, j))) / (∑_(k=1)^T exp(S̃_(i, k)))
For any future position where j > i:
S̃_(i, j) = S_(i, j) + (-∞) = -∞
Taking the natural exponential:
exp(S̃_(i, j)) = lim_(x → -∞) e^x = 0
Therefore, the numerator is identically zero:
A_(i, j) = (0) / (∑_(k=1)^i exp(S_(i, k)) + ∑_(k=i+1)^T 0) = (0) / (∑_(k=1)^i exp(S_(i, k))) = 0.0
The attention weight for future tokens is strictly and provably zero. \blacksquare
Given a 3 × 3 raw score matrix S:
S = 2.0 & 1.5 & 3.0 \ 1.0 & 2.5 & 0.5 \ 0.5 & 1.0 & 2.0
Add upper-triangular mask M:
S̃ = S + M = 2.0 & -∞ & -∞ \ 1.0 & 2.5 & -∞ \ 0.5 & 1.0 & 2.0
Compute row-wise softmax:
- Row 1: Exponents: [e^(2.0), 0, 0] = [7.389, 0, 0]. Sum = 7.389.
A_(1, :) = [1.0, 0.0, 0.0]
- Row 2: Exponents: [e^(1.0), e^(2.5), 0] = [2.718, 12.182, 0]. Sum = 14.900.
A_(2, :) = [2.718/14.90, 12.182/14.90, 0.0] = [0.182, 0.818, 0.0]
- Row 3: Exponents: [e^(0.5), e^(1.0), e^(2.0)] = [1.649, 2.718, 7.389]. Sum = 11.756.
A_(3, :) = [0.140, 0.231, 0.629]
Every row forms a valid probability distribution on the simplex, with zero weight assigned to future columns!
In PyTorch, floating-point -∞ is implemented using `-torch.finfo(dtype).max` (e.g. `-1e4` for FP16, `-1e9` for FP32) to prevent NaN errors in mixed-precision arithmetic. In FlashAttention, causal masking is implemented without materializing any mask matrix: the GPU kernel simply skips memory reads and computations for blocks where j > i.
The mathematical trick of adding -∞ before softmax was popularized in early neural machine translation systems (Bahdanau 2014) to mask padding tokens, and adopted universally in 2017 for autoregressive transformer decoding.
- exp(-∞) = 0: Negative infinity before softmax is zero after softmax.
- Add before exponentiating: Never zero out probabilities after softmax; add -∞ before to keep row sums equal to 1.
How does causal masking enable O(1) full-sequence parallel training while inference generates tokens sequentially?
Imagine an orchestra rehearsal versus a live performance. In rehearsal with a musical score, every musician has all sheet music open simultaneously (parallel training), but a sliding cardboard shield prevents them from playing ahead of the conductor's beat (causal masking). In a live improvisation (inference), there is no sheet music: each note must be played, heard, and fed into memory before deciding what note to play next.
- Learning goalAnalyze the computational difference between training under teacher forcing (O(1) forward pass) and sequential autoregressive decoding (O(T) forward passes with KV caching).
- Helpful foundationsAttention Is All You Need (Vaswani et al., 2017) · Teacher Forcing in Sequence Models · KV Cache Architecture for LLM Inference
- Common trapAssuming LLMs generate full paragraphs in parallel during chat inference because they train in parallel. Inference is strictly sequential: generating 100 new tokens requires 100 forward passes through the model.
- Next connectionNow that we understand single-head causal attention, we expand capacity by splitting representations across multiple specialized attention heads.
More context & mechanics
1. Training Phase (Teacher Forcing & Full Parallelism):
During training, the ground truth sequence w₁, w₂, ṡ, w_T is entirely known. Because the causal mask guarantees that z_t depends only on w_(≤ t), all T positions can be computed simultaneously in a single forward pass:
Z = softmax((Q K^\top) / (√(dₖ)) + M)V
All T next-token cross-entropy losses are computed in parallel:
L = 1/T∑_(t=1)^(T-1) CrossEntropy(logits_t, w_(t+1))
Complexity in GPU time steps: O(1) sequential operations.
2. Inference Phase (Autoregressive Generation):
At test time, future tokens do not yet exist. The model generates one token at a time:
w_(T+1) ~ softmax(logits_T)
To generate N tokens, the model must execute N sequential forward passes: O(N) sequential operations. To avoid recomputing past Keys and Values at every step, production engines store previously computed Key-Value vectors in a KV Cache.
This distinction drives the split between model training (throughput-optimized on thousands of GPUs using large batch parallelism) and model serving (latency-optimized using KV caches, vLLM PagedAttention, and speculative decoding).
Williams and Zipser introduced Teacher Forcing in 1989 for training recurrent neural networks. Vaswani et al. combined teacher forcing with causal matrix masking, allowing GPUs to saturate their tensor cores across all sequence positions simultaneously.
- Golden Axiom: "Train in parallel via matrix mask; generate in sequence token by token."
- Training: All tokens at once thanks to the mask.
- Inference: One token at a time because the future is unknown.
How does setting upper-triangular attention logits to negative infinity enforce autoregressive temporal causality?
Imagine reading a detective novel where all the upcoming paragraphs are redacted with thick black marker until you have finished reading the current sentence. If you were allowed to look ahead to page 200, predicting page 2 would be trivial cheating: you wouldn't learn how to deduce clues, you'd just copy the solution. Causal masking is that digital black marker: it sets future attention paths to -∞. When the softmax machine sees -∞, it computes e^(−∞) = 0, guaranteeing zero information flows backward in time.
- Learning goalConstruct causal attention masks and prove that additive minus-infinity entries yield strictly zero attention weight to future tokens.
- Helpful foundationsMatrix upper/lower triangles · Improving Language Understanding by Generative Pre-Training (Radford et al., 2018)
- Common trapThinking causal masks require running a sequential Python loop during training; the entire sequence is processed in a single parallel GEMM by masking the T x T attention matrix simultaneously.
- Next connectionWhy should the model maintain separate input embeddings and output projection matrices when they both map between tokens and vectors? We examine weight tying.
More context & mechanics
In autoregressive modeling, the output at step t must depend strictly on tokens 1, ṡ, t.
The raw attention logits matrix S = Q K^T / √(dₖ) ∈ R^(T × T) is masked with additive matrix M:
M_(i, j) = 0 & if i ≥ j \ -∞ & if i < j
Computing softmax on the sum:
A_(i, j) = (e^S_(i, j) + M_(i, j)) / (∑_(k=1)^T e^S_(i, k) + M_(i, k))
For any future index j > i:
e^S_(i, j) + (-∞) = e^(−∞) = 0 ⟹ A_(i, j) = 0
The resulting attention weight matrix A is strictly lower-triangular, preserving the causal autoregressive factorization:
P(x₁, ṡ, x_T) = ∏_(t=1)^T P(x_t | x₁, ṡ, x_(t-1))
Every decoder-only LLM (GPT, LLaMA, Falcon, Mistral) uses causal masking during pretraining. This enables parallel training across entire text sequences in a single forward pass while preserving sequential autoregressive inference behavior.
Peter Brown et al. established statistical autoregressive language modeling at IBM in 1990. Radford et al. combined causal masking with deep transformers in GPT-1 (2018).
- Upper triangle gets minus infinity; future weights vanish to zero.
- Look backward to learn; block forward to prevent cheating.
How does Causal Masking: The Upper-Triangular -∞ Barrier operate mathematically at scale?
Three progressive matrix states for sequence length L = 4:
- Learning goalMaster the algebraic and geometric derivation of Causal Masking: The Upper-Triangular -∞ Barrier.
- Helpful foundationsFoundations of Build Llm From Scratch · Coordinate Invariance & Geometric Projections · Vaswani et al. (2017), Attention Is All You Need, NeurIPS
- Common trapAssuming Causal Masking: The Upper-Triangular -∞ Barrier is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
A matrix masking operational diagram in square 1:1 format on warm cream parchment (#fcf0db) with a hairline border.
Top section: Title in navy serif: "Causal Attention Masking: The -∞ Barrier". Subtitle: "Enforcing autoregressive causality during parallel training forward passes."
Middle section: Three progressive matrix states for sequence length L = 4:
- Raw Score Matrix S = Q K^\top / √(dₖ) (Full 4x4 matrix with bidirectional affinities).
- Add Additive Causal Mask M:
[[0, -∞, -∞, -∞],
[0, 0, -∞, -∞],
[0, 0, 0, -∞],
[0, 0, 0, 0]].
- Softmax Output P = Softmax(S + M):
Because e^(−∞) = 0, all upper-triangular entries become exactly 0.00!
Row 0 attends only to Token 0.
Row 3 attends to Tokens 0, 1, 2, 3.
Bottom section: Autoregressive invariant: "Tokens can never attend to future positions during pretraining, matching the exact conditions of test-time sequential generation."
High-contrast matrix grid cells, clear red barrier over upper triangle, balanced square layout.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
"Causal Masking: The Upper-Triangular -∞ Barrier: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."