Basics Of Transformers
What core breakthrough did Attention Is All You Need introduce?
How this lesson unfolds.
Begin with What core breakthrough did Attention Is All You Need introduce?. Follow the visual diagrams to build an intuitive mental model of basics of transformers.
What mathematical breakthrough enabled the 2017 Transformer to replace recurrence and convolutions with pure self-attention?
Think of an old telephone switchboard where operators plug physical cables between callers one by one, creating a massive tangle as calls increase. Recurrent networks were like that switchboard operator, passing a single message down a chain of 1,000 operators who inevitably garbled the words. Vaswani et al. in 2017 tore out the entire switchboard and replaced it with an open broadcast hall: every speaker projects a Query, every listener offers a Key, and whenever frequencies align, Values stream across the room simultaneously in O(1) operations with zero degradation.
- Learning goalDerive and implement scaled dot-product attention Attention(Q, K, V) = softmax(Q K^T / sqrt(dₖ)) V and multi-head projection splitting.
- Helpful foundationsMatrix multiplication and transpose operations · Attention Is All You Need (Vaswani et al., 2017) · Linear projections in Euclidean space
- Common trapAssuming that multi-head attention increases total computational operations by h times. Multi-head attention splits the existing hidden dimension d_model into h smaller sub-spaces of dimension dₖ = d_model / h (e.g. 768 / 12 = 64); the total computational FLOPs remain identical to a single large head.
- Next connectionBefore tokens can form Queries, Keys, and Values, discrete words must first be mapped into continuous geometric space: vector embeddings.
More context & mechanics
Scaled Dot-Product Attention operates on query, key, and value matrices:
Attention(Q, K, V) = softmax((QK^T) / (√(dₖ)))V
where Q, K, V ∈ R^(T × dₖ).
In Multi-Head Attention with h heads:
MultiHead(Q, K, V) = Concat(head₁, ṡ, head_h) W_O
where headᵢ = Attention(Q Wᵢ^Q, K Wᵢ^K, V Wᵢ^V)
with projection matrices Wᵢ^Q, Wᵢ^K, Wᵢ^V ∈ R^d_(model) × dₖ and output projection W_O ∈ R^h dₖ × d_(model).
Each head independently attends to information at different subspace representations (e.g., one head attends to syntax, another to coreference, another to semantic relationships).
This mathematical kernel forms the backbone of GPT-4, Claude, Gemini, Llama, and modern vision transformers. High-performance implementations fuse the attention computation into hardware-optimized kernels using FlashAttention-2 or FlashAttention-3.
In June 2017, Ashish Vaswani and seven coauthors at Google Brain and Google Research released 'Attention Is All You Need'. Originally created to beat the state-of-the-art on English-to-German machine translation, its title became the defining motto of modern machine learning.
- Query times Key determines who listens; Value carries what is said.
- Divide by √(dₖ) so the gradients do not die in softmax saturation.
How does an embedding matrix map discrete vocabulary symbols into continuous semantic geometry in R^d?
If you give a computer the numbers 1 for 'cat', 2 for 'kitten', and 3 for 'bulldozer', the computer thinks 2 is closer to 3 than to 1, which is semantically absurd. An embedding space fixes this by lifting words into a 768-dimensional sphere. In this high-dimensional room, 'cat' and 'kitten' sit right next to each other at a distance of 0.05, while 'bulldozer' is parked thousands of leagues away. Furthermore, directions mean concepts: moving in the direction of the royalty vector transforms the vector for 'man' into 'king' and 'woman' into 'queen'.
- Learning goalConstruct continuous vector spaces where semantic similarity maps to inner product cosine angles cos(theta) = (u . v) / (||u|| ||v||).
- Helpful foundationsVector dot products and Euclidean norms · Efficient Estimation of Word Representations in Vector Space / Word2Vec (Mikolov et al., 2013) · GloVe: Global Vectors for Word Representation (Pennington et al., 2014)
- Common trapConfusing static word embeddings (Word2Vec, GloVe) with contextual embeddings in Transformers. Static embeddings assign one fixed vector to 'bank' regardless of whether it means a riverbank or a financial vault. Transformers compute contextual embeddings where the vector for 'bank' shifts dynamically based on surrounding tokens.
- Next connectionWith continuous token geometries defined, how are transformer blocks composed into Encoders and Decoders?
More context & mechanics
An embedding matrix W_E ∈ R^|V| × d_(model) serves as a continuous lookup table. For token ID i ∈ 0, ṡ, |V|-1:
eᵢ = W_E[i, :] ∈ R^d_(model)
Semantic relationships emerge geometrically:
- Cosine Similarity:
sim(u, v) = (u · v) / (\|u\| \|v\|) ∈ [-1, 1]
Tokens appearing in similar contexts cluster into high-density neighborhoods.
- Vector Arithmetic & Linear Analogies:
vec(v)_(king) - vec(v)_(man) + vec(v)_(woman) ≈ vec(v)_(queen)
Linear substructures in latent space represent relational properties like tense, gender, and capital cities.
Vector databases (Pinecone, Milvus, Qdrant) use these continuous geometric embeddings to perform approximate nearest neighbor (ANN) search for RAG systems, retrieving the most semantically relevant text chunks in milliseconds across millions of documents.
Tomas Mikolov et al. at Google published Word2Vec in 2013, proving that simple neural language modeling tasks (Skip-gram and CBOW) naturally forced vector spaces to organize linear semantic relationships.
- Words that appear together live together in embedding space.
- Semantics is geometry; meaning is distance and direction.
What is the structural and mechanical difference between bidirectional Transformer Encoders and autoregressive Decoders?
Imagine taking an open-book reading comprehension exam versus writing a spontaneous speech live on television. In the Encoder, you are taking the reading exam: you have the entire paragraph right in front of you, and your eyes can dart forward and backward between word 1 and word 100 freely to understand context (bidirectional attention). In the Decoder, you are speaking live: you are forbidden from looking into the future because tomorrow hasn't happened yet. You can only look back at what you have already said (causal masked attention).
- Learning goalConstruct attention mask matrices: full bidirectional masks vs. lower-triangular causal masks (M_(i,j) = -inf for j > i) and formulate cross-attention.
- Helpful foundationsLower-triangular masking matrices · Attention Is All You Need (Vaswani et al., 2017) · Matrix broadcasting and additive masks
- Common trapBelieving that Encoders and Decoders use different internal neural layers. Both contain identical Multi-Head Attention and MLP layers; the sole difference lies in the causal mask applied to the attention logits prior to the softmax operation.
- Next connectionThis architectural divergence led the AI community to split into two competing evolutionary branches: BERT versus GPT.
More context & mechanics
The mathematical distinction lies in the attention mask matrix M ∈ R^(T × T):
Attention(Q, K, V) = softmax((QK^T) / (√(dₖ)) + M)V
- Encoder (Bidirectional):
- Mask: M_(i, j) = 0 for all i, j.
- Every token attends to every other token: past, present, and future.
- Ideal for understanding, classification, and feature extraction.
- Decoder (Causal / Autoregressive):
- Mask: Lower-triangular:
M_(i, j) = 0 & if j ≤ i \ -∞ & if j > i
- Since exp(-∞) = 0, attention weights for future tokens j > i are strictly zero.
- Preserves causality: token i cannot cheat by looking at future token i+1.
- Cross-Attention (Encoder-Decoder Bridge):
Queries come from the decoder (Q = H_(dec) W_Q), while Keys and Values come from the encoder (K = H_(enc) W_K, V = H_(enc) W_V).
Encoder-only models (BERT, RoBERTa) power search ranking and embeddings. Decoder-only models (GPT-4, Llama 3) power generative chat and code generation. Encoder-decoder models (T5, Whisper) power audio transcription and machine translation.
The original 2017 Vaswani paper was an encoder-decoder machine translation system. Within 18 months, the community split: Google isolated the encoder as BERT (2018), while OpenAI isolated the decoder as GPT (2018).
- Encoders see the whole sentence; Decoders only see the past.
- Causal mask: zero on and below the diagonal, negative infinity above.
Why did the NLP revolution split into encoder-only BERT and decoder-only GPT, and why did decoder-only GPT win the generative crown?
BERT was a forensic detective; GPT was a prolific novelist. BERT looked at a completed crime scene where 15% of the clues were blacked out with marker ([MASK]) and used clues from both the left and right to deduce what was missing. It was unbeatable at understanding existing text, but terrible at writing new stories because it couldn't generate long coherent sequences. GPT, on the other hand, was trained purely to write the next word. Because modern AI value exploded around dialogue, coding, and creative generation, the novelist (GPT) conquered the world while the detective (BERT) was relegated to classification pipelines.
- Learning goalCompare the pretraining objectives of Masked Language Modeling P(xᵢ | x_(-i)) and Causal Language Modeling P(xᵢ | x_(<i)) and evaluate their generative capabilities.
- Helpful foundationsMasked language modeling objective · BERT (Devlin et al., 2018) · Improving Language Understanding by Generative Pre-Training / GPT-1 (Radford et al., 2018)
- Common trapBelieving that BERT's bidirectional attention makes it fundamentally superior to GPT for all tasks. While bidirectional attention gives BERT an advantage in discriminative classification, it cannot be trained autoregressively to generate open-ended text without combinatorial sampling slowdowns.
- Next connectionTo understand why both BERT and GPT permanently displaced Recurrent Neural Networks, we analyze the computational downfall of RNNs.
More context & mechanics
Comparing the training objectives:
- BERT (Masked Language Modeling - MLM):
- 15% of input tokens are corrupted: 80% replaced with `[MASK]`, 10% random token, 10% unchanged.
- Objective: Predict the corrupted tokens using full bidirectional context:
L_(MLM)(θ) = -∑_(i ∈ Masked) log P_θ(xᵢ | x_(\setminus i))
- Strength: Unconstrained bidirectional representation learning.
- Limitation: Cannot generate variable-length autoregressive text naturally.
- GPT (Causal Language Modeling - CLM):
- Objective: Predict token x_t given only previous tokens x_(<t):
L_(CLM)(θ) = -∑_(t=1)^T log P_θ(x_t | x_(<t))
- Strength: Natural sequence generation; every token in the sequence provides a training signal.
Why Decoder-Only Won:
Autoregressive models scale seamlessly to in-context learning, zero-shot prompting, and few-shot task adaptation. Generating text is a superset of understanding text: an autoregressive model can classify, summarize, translate, and reason through conversational completion.
While BERT embeddings still power dense retrieval systems in Google Search, virtually all modern foundation and reasoning models (GPT-4, Llama 3, Mistral, DeepSeek) are decoder-only autoregressive transformers.
In late 2018, BERT shattered 11 NLP benchmarks, leading most researchers to declare encoder architectures the definitive future of NLP. OpenAI persisted with decoder-only autoregressive models (GPT-2 in 2019, GPT-3 in 2020), eventually proving that generative scaling unlocked capabilities BERT could never achieve.
- BERT fills in the blanks; GPT writes the future.
- Generation subsumes comprehension: if you can predict what comes next, you must understand what came before.
What fundamental mathematical and hardware constraints made Recurrent Neural Networks obsolete for large-scale language modeling?
Imagine trying to assemble a 1,000-piece puzzle where the factory rule states: you may only look at piece #500 after piece #499 has been glued down. You have 10,000 workers (GPU CUDA cores), but 9,999 workers must sit with their arms crossed waiting for worker #1 to finish piece #1, then worker #2 for piece #2. That was the tragedy of RNNs: strict sequential time dependency. The Transformer fired the rulebook, handed all 1,000 puzzle pieces to all 10,000 workers simultaneously, and let them compute interactions in parallel. Training speed surged by 100x.
- Learning goalFormulate the sequential dependency graph of recurrent hidden states and prove why self-attention enables O(1) sequential operations during training.
- Helpful foundationsComputational complexity of matrix operations · Gradient backpropagation through time (BPTT) · GPU SIMD / SIMT parallel architecture
- Common trapAssuming LSTMs solved the vanishing gradient problem permanently. LSTMs introduced additive cell state gates that alleviated vanishing gradients across dozens of steps, but still suffered from information compression bottlenecks and gradient decay across sequences of 1,000+ tokens.
- Next connectionHaving understood the core transformer foundations, Lesson 5 explores how next-token prediction powers modern ChatGPT systems.
More context & mechanics
Comparison of sequential complexity and path lengths:
| Property | Recurrent Network (RNN/LSTM) | Transformer Self-Attention |
| :--- | :--- | :--- |
| Sequential Operations during Training | O(T) | O(1) |
| Maximum Information Path Length | O(T) | O(1) |
| Computational Complexity per Layer | O(T · d²) | O(T² · d) |
| GPU Parallelization Capability | Poor (strictly sequential loop) | Optimal (massive batched GEMM) |
- Information Bottleneck in RNNs:
Hidden state h_T ∈ R^d must compress all information from x₁ through x_T into a single vector of fixed capacity d.
- Direct Interaction in Attention:
Any token xᵢ attends directly to xⱼ regardless of distance |i - j|:
Grad Path: (∂ yᵢ) / (∂ xⱼ) = A_(i, j) W_V
Gradient signals flow across a direct single-hop connection rather than decaying across T recurrent transitions.
Modern massive clusters (e.g. 16,000 NVIDIA H100s) execute matrix multiplications (GEMMs) at trillions of floating-point operations per second. Self-attention maps directly to parallel GEMMs, allowing modern LLMs to scale to trillions of tokens.
LSTMs were invented by Hochreiter & Schmidhuber in 1997 and ruled NLP until 2017. When Vaswani et al. published the Transformer, researchers initially resisted, but within two years, RNNs were completely eliminated from competitive NLP benchmarks.
- RNNs are chains; Transformers are webs.
- An O(1) path length keeps gradients alive; parallel GEMMs keep GPUs fed.