Intro to LLM
What core mathematical principle governs from task specific nlp to general purpose llms?
How this lesson unfolds.
Begin with What core mathematical principle governs from task specific nlp to general purpose llms?. Follow the visual diagrams to build an intuitive mental model of intro to llm.
How did natural language processing transition from isolated, task-specific pipelines to unified general-purpose foundation models?
In 2015, building an NLP product required hiring five separate engineering teams: one built a sentiment analysis SVM, another trained a Named Entity Recognition (NER) conditional random field, a third trained an English-to-French LSTM, a fourth tuned a question-answering BiDAF model, and a fifth built a dependency parser. None of these five models could perform each other's jobs. Today, a single 7-billion parameter transformer handles all five tasks simultaneously—simply by conditioning its latent state on a natural language text prompt.
- Learning goalTrace the architectural evolution from task-specific feature pipelines to unified autoregressive sequence models trained across broad distributions.
- Helpful foundationsClassical NLP pipelines (tokenization, POS tagging, NER) · Language Models are Few-Shot Learners (Brown et al., 2020) · BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018)
- Common trapAssuming modern LLMs contain distinct internal neural sub-networks dedicated to translation, summarization, and math. The exact same weight matrices W_Q, W_K, W_V, W_O process every task; the apparent specialization emerges entirely from dynamic attention routing conditioned on the input tokens.
- Next connectionThis unification leads directly to the question: how can a single architecture natively formulate every language task as next-token prediction?
More context & mechanics
Historically, NLP systems solved tasks via specialized discriminative objectives:
- Sentiment: P(y ∈ ± 1 | x) using logistic regression over TF-IDF.
- Translation: argmax_y P(y₁, ṡ, yₘ | x₁, ṡ, xₙ) using bilingual phrase tables or seq2seq LSTMs.
- Parsing: Transition-based shift-reduce dependency parsers.
The Foundation Model paradigm unifies all tasks into a single conditional distribution:
P(Y | X) ≡ P(Completion | Prompt)
By scaling parameters (N ≥ 10⁹) and pretraining on trillions of tokens, the network learns an internal world model and semantic manifold capable of zero-shot task transfer without architectural modifications.
Modern enterprise software consolidates dozen of legacy ML microservices into a single foundation model API call. Instead of maintaining separate servers for ticket categorization, sentiment tracking, and entity extraction, a unified LLM prompt performs all three in a single pass.
The term 'Foundation Models' was coined in 2021 by the Stanford Institute for Human-Centered Artificial Intelligence (HAI) in Bommasani et al.'s landmark report, formalizing the historic shift from fractured task-specific models to shared, highly adaptable base models.
- Old NLP had one model per task; modern AI has one model for all tasks.
- Prompts replace architectures; general foundations replace fragile silos.
How does casting diverse natural language tasks into a unified text-to-text sequence format enable multi-task execution?
Think of an old Swiss Army knife with 10 different folding blades versus a laser cutter that can slice wood, metal, plastic, and glass using one single adjustable beam of light. In older NLP, a sentiment classifier output a scalar probability y ∈ [0, 1] while a translation model output target vocabulary indices. In modern LLMs, both tasks are converted into identical token streams: 'Classify sentiment: Great movie -> Positive' and 'Translate to French: Hello -> Bonjour'. When all tasks speak the common currency of tokens, one single autoregressive engine satisfies them all.
- Learning goalFormulate heterogeneous NLP objectives (classification, translation, extraction) as autoregressive text-to-text sequence generation problems.
- Helpful foundationsSequence-to-sequence modeling fundamentals · Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer / T5 (Raffel et al., 2019) · Language Models are Unsupervised Multitask Learners / GPT-2 (Radford et al., 2019)
- Common trapBelieving that text classification with an LLM requires replacing the final layer with a classification head. While classification heads are common in BERT, an autoregressive LLM can classify simply by measuring whether the logit for the token 'Positive' is higher than the logit for 'Negative'.
- Next connectionHaving seen that all tasks can be framed as text, we examine how the model physically steers its internal activations to solve them.
More context & mechanics
In the text-to-text framework, the input string X contains both task instructions and input data:
X = [Task Description] ⊕ [Input Context]
The target Y is generated token-by-token:
P(Y | X) = ∏_(j=1)^(|Y|) P(yⱼ | X, y₁, ṡ, y_(j-1))
Examples:
- Summarization: `X = "Summarize: [Article]"` → `Y = "[Summary]"`
- Arithmetic: `X = "Evaluate: 47 * 12 ="` → `Y = "564"`
- Code: `X = "Python function for quicksort:"` → `Y = "def quicksort(arr): ..."`
Because every token prediction uses the exact same vocabulary V and unembedding projection W_U, no task-specific classification layers are needed.
Google's T5 (Text-to-Text Transfer Transformer) proved that benchmark tasks across GLUE, SuperGLUE, SQuAD, and WMT translation could all achieve state-of-the-art results using a unified text-to-text interface, forming the blueprint for modern chat systems.
Radford et al. demonstrated in GPT-2 (2019) that language models trained without explicit supervision on web text naturally learned to perform translation and summarization when prompted, proving that multi-task learning is an emergent property of scale.
- Text in, text out: the universal format of intelligence.
- When all tasks speak tokens, one transformer rules them all.
How does in-context conditioning modulate a frozen transformer's latent representations without updating parameter weights?
Picture a grand piano whose 88 keys and internal strings are permanently tuned and fixed in place. The piano itself does not change, yet it can play a melancholy Chopin nocturne, an upbeat ragtime tune, or a thunderous Beethoven sonata depending entirely on which sequence of sheet music notes prompts the pianist's fingers. The prompt tokens in an LLM act as the sheet music: they bias the hidden state trajectory through high-dimensional space without altering a single physical weight matrix in the model.
- Learning goalModel in-context task steering as dynamic conditioning of hidden state representations through transformer self-attention over prompt prefix tokens.
- Helpful foundationsHidden state representations in neural networks · Transformers are Secretly Meta-Learners (von Oswald et al., 2023) · Causal self-attention conditioning
- Common trapThinking that when an LLM 'learns' a new task from a prompt, its weights are being fine-tuned via backpropagation in real time. Model weights remain completely frozen during inference; task adaptation occurs strictly through forward-pass activation routing inside the context window.
- Next connectionTo see where this latent routing takes place, we must inspect the physical architecture of the transformer stack.
More context & mechanics
During inference, parameter matrix \Theta is frozen: Δ θ = 0.
The prompt tokens X_(prompt) = (x₁, ṡ, x_p) are mapped to Key and Value vectors:
K_(prompt) = X_(prompt) W_K, V_(prompt) = X_(prompt) W_V
When generating subsequent tokens t > p, the query q_t = x_t W_Q computes attention weights over the prompt:
a_(t, j) = (exp(q_t^T kⱼ / √(dₖ))) / (∑_(i=1)^t exp(q_t^T kᵢ / √(dₖ)))
The output representation h_t = ∑_(j=1)^t a_(t, j) vⱼ is directly modulated by the prompt's key-value vectors. The prompt serves as a dynamic memory bank that alters activation trajectories without modifying weights.
In-context learning allows developers to construct few-shot prompts (e.g. providing 3 examples of SQL queries) to instantly adapt an LLM to generate enterprise-specific schemas without expensive model retraining or deployment downtime.
OpenAI's GPT-3 paper ('Language Models are Few-Shot Learners', 2020) demonstrated that a 175-billion parameter model could match or exceed fine-tuned task-specific models simply by feeding demonstration examples in the prompt, revolutionizing prompt engineering.
- Frozen weights, fluid activations.
- Prompts steer the ship; weights are the ocean currents.
How does stacking identical Transformer blocks enable deep hierarchical feature composition?
Think of an assembly line with 12 identical robotic workstations (as in GPT-2 small with L=12 layers). The raw chassis (token embeddings) enters workstation 1. Each station performs the exact same two-step operation: first, it checks how all the parts relate to one another across the whole car (Multi-Head Self-Attention); second, it tightens and refines individual components independently (Feed-Forward Network). Then it hands the upgraded chassis to station 2. By station 12, raw subwords have been assembled into profound semantic representations.
- Learning goalDerive the mathematical composition of a standard transformer layer block, including Pre-LayerNorm, Multi-Head Attention, residual connections, and Feed-Forward Networks.
- Helpful foundationsResidual connections (ResNet skip connections) · Layer Normalization mechanics · Linear matrix projections
- Common trapAssuming that each layer of a transformer has a completely different mathematical architecture. Every block in a transformer stack is identical in algebraic structure and parameter count; they differ solely in the specific floating-point weights learned during training.
- Next connectionTo understand where this architecture fits in the broader landscape of computer science, we position it across two fundamental axes of AI.
More context & mechanics
In a modern Pre-LN Transformer, each block l ∈ 1, ṡ, L computes:
- Self-Attention Sublayer:
x_(mid) = x^((l-1)) + MultiHead(LayerNorm(x^((l-1))))
- Feed-Forward Sublayer (MLP):
x^((l)) = x_(mid) + MLP(LayerNorm(x_(mid)))
where the MLP expands and contracts the hidden dimension:
MLP(h) = GELU(h W₁ + b₁) W₂ + b₂, W₁ ∈ R^(d × 4d), W₂ ∈ R^(4d × d)
The residual connection (+ x) creates an unimpeded gradient highway:
(∂ x^((L))) / (∂ x^((0))) = I + ∑_(l=1)^L (∂ Sublayer_l) / (∂ x^((l-1)))
preventing vanishing gradients even when stacking 32, 64, or 128 layers.
GPT-2 small stacks L=12 blocks (d=768); GPT-3 stacks L=96 blocks (d=12288); Llama 3 70B stacks L=80 blocks (d=8192). The modularity means scaling an LLM in PyTorch simply involves changing the loop `nn.ModuleList([TransformerBlock() for _ in range(n_layers)])`.
Post-LN (Vaswani et al., 2017) placed normalization after the residual addition (x = LayerNorm(x + Sublayer(x))), which suffered from training instability and required delicate learning rate warmups. Pre-LN (Radford et al., 2019 in GPT-2) placed LayerNorm on the residual branches, dramatically improving gradient flow.
- Attention routes tokens; MLP stores facts; residual links save gradients.
- Identical blocks stacked high build deep hierarchical representations.
How do the Discriminative vs. Generative and Rule-Based vs. Machine-Learned axes map the landscape of AI systems?
Imagine a 2D Cartesian coordinate plane. The horizontal X-axis represents 'How is the system built?' (from rigid human-written rules on the left to statistical deep learning from data on the right). The vertical Y-axis represents 'What is the system's goal?' (from classifying inputs P(Y | X) at the bottom to synthesizing new distributions P(X) at the top). ELIZA sits in the top-left (rule-based generative); a spam classifier sits in the bottom-right (learned discriminative); an LLM sits in the top-right quadrant (deep learned generative).
- Learning goalSituate AI algorithms onto a two-dimensional taxonomy defined by representation learning methodology and probabilistic generation objectives.
- Helpful foundationsDiscriminative vs generative models · Rule-based vs statistical systems
- Common trapConfusing discriminative classifiers with generative models. Discriminative models estimate class boundaries P(Y | X); generative models model the joint probability P(X, Y) or data distribution P(X), allowing them to sample realistic synthetic data.
- Next connectionWith the generative learning quadrant identified, we survey the wide spectrum of practical language applications powered by modern LLMs.
More context & mechanics
The two orthogonal axes of AI categorization:
- Axis 1: Knowledge Acquisition Mechanism
- Symbolic / Handcrafted Rules: Knowledge encoded by human programmers via logical IF-THEN rules, grammars, and heuristics.
- Learned Representations: Statistical models learning parameter weights θ from empirical data D via optimization.
- Axis 2: Probabilistic Modeling Objective
- Discriminative: Models conditional distribution P(Y | X). Separates classes, finds boundaries, outputs labels.
- Generative: Models data distribution P(X) or joint distribution P(X, Y). Capable of generating novel samples X̃ ~ P_θ(X).
| Quadrant | Paradigm | Classic Example |
| :--- | :--- | :--- |
| Top-Left | Rule-Based Generative | ELIZA (1966), Context-Free Grammars |
| Bottom-Left | Rule-Based Discriminative | Expert Systems (MYCIN), Regex filters |
| Bottom-Right| Learned Discriminative | SVMs, ResNet-50, BERT classifiers |
| Top-Right | Learned Generative | GPT-4, Llama 3, Stable Diffusion |
System architects use this matrix to choose appropriate technologies: fraud transaction blocking belongs in Learned Discriminative (XGBoost), whereas autonomous code authoring requires Learned Generative (LLMs).
AI research swung between these quadrants over decades. The 'Good Old-Fashioned AI' (GOFAI) era (1950s–1980s) dominated the left half; statistical machine learning (1990s–2000s) conquered bottom-right; modern LLMs conquered top-right.
- X-axis is How: Rules vs Data. Y-axis is What: Labels vs Creation.
- LLMs reign in the top-right: learned from data, creating new worlds.
How does the single generative engine of an LLM power diverse commercial applications across industries?
Think of electricity. Electricity is just the physical movement of electrons through copper wire. Yet that single physical phenomenon powers refrigerators to keep food cold, ovens to bake bread, supercomputers to calculate orbits, and MRI machines to scan brains. The autoregressive prediction of tokens is the electricity of the cognitive age: conditioned on code syntax it builds software; conditioned on medical literature it assists clinical diagnoses; conditioned on customer chat it provides 24/7 technical support.
- Learning goalEvaluate the deployment modalities of LLMs across software engineering, information extraction, reasoning, and autonomous agent orchestration.
- Helpful foundationsAPI integration patterns · Prompt templates and few-shot formatting
- Common trapBelieving that an LLM needs external specialized software to perform reasoning. Prompting techniques like Chain-of-Thought ('Let's think step by step') allow the base transformer to dedicate additional computational tokens to decompose multi-step logic internally.
- Next connectionTo deploy an LLM effectively, we must understand what it can actually do at a fundamental computational level—and what it cannot do.
More context & mechanics
A single LLM backbone powers multiple application topologies:
- Zero-Shot / Few-Shot Generation: Direct prompt completion for drafting, translation, and summarization.
- Structured Extraction: Constraining output logits via grammar masks (JSON schema enforcement) to extract entities from messy unstructured documents.
- Reasoning Chains (Chain-of-Thought): Allocating test-time compute by prompting intermediate reasoning steps:
P(Answer | Prompt) = ∑_(Steps) P(Answer | Prompt, Steps) P(Steps | Prompt)
- Tool Use & Function Calling: Generating structured API call invocations (e.g., `"tool": "calculator", "args": "expr": "45 * 22"`) executed by external environments.
Production platforms like GitHub Copilot (code generation), Harvey AI (legal document analysis), and Cursor (codebase refactoring) run on the same underlying foundation model backbones, customized via system prompts and contextual retrieval.
Wei et al. (2022) published 'Chain-of-Thought Prompting Elicits Reasoning in Large Language Models', revealing that simply encouraging a model to output intermediate reasoning steps dramatically boosted arithmetic, symbolic, and commonsense reasoning benchmarks.
- One generative engine, infinite downstream manifestations.
- Condition the prompt, constrain the tokens, unlock the application.
What are the fundamental computational and theoretical capabilities and limitations of autoregressive transformers?
If you ask an LLM to multiply two 20-digit prime numbers in its head with zero scratchpad tokens, it will likely hallucinate an answer. Why? Because a standard transformer layer allocates a fixed number of floating-point operations per token (depth L). It cannot dynamically loop 5,000 times to perform carry-over arithmetic unless you allow it to output intermediate reasoning tokens. What an LLM can do is match complex statistical patterns, synthesize styles, and interpolate across high-dimensional semantic spaces; what it cannot do without external tools is guarantee deterministic arithmetic or factual infallibility.
- Learning goalIdentify the computational complexity limits (circuit depth, fixed compute per token) of autoregressive transformers and explain why scratchpad generation enables complex reasoning.
- Helpful foundationsComputational complexity (TC0 circuits) · Autoregressive decoding mechanics
- Common trapAssuming LLMs maintain an internal simulation of truth or objective reality. LLMs model the linguistic distribution of their training corpus; they optimize cross-entropy loss, not epistemological truth.
- Next connectionNow that we know what these models compute, how do engineers decide their physical dimensions: layers, heads, and embedding widths?
More context & mechanics
Theoretical properties of autoregressive transformers:
- Constant Computational Depth per Token:
Generating each token requires exactly O(L) matrix multiplications. An L-layer transformer is computationally equivalent to a circuit of constant depth (TC⁰), limiting single-token computation to non-iterative logic.
- Scratchpad Equivalence to Turing Completeness:
When allowed to emit autoregressive tokens into its context window (scratchpad / Chain-of-Thought), the sequence length T acts as the tape of a Turing Machine, enabling arbitrary iterative algorithms.
- Probability Estimation vs Fact Storage:
The model produces:
P(w_t | w_(<t)) = softmax(W_U h_t)
High probability does not imply ground-truth veracity; it merely reflects high conditional likelihood under the pretraining distribution.
Understanding these boundaries dictates system design: engineers delegate deterministic calculations to Python calculators via tool-calling, use RAG for accurate factual recall, and rely on the LLM for linguistic synthesis and logic orchestration.
Theoretical computer scientists (such as Merrill & Sabharwal, 2023) proved that Transformers without scratchpads cannot solve graph reachability or parity problems for arbitrary lengths, formally delineating the boundaries of single-token attention.
- Fixed depth per token; infinite depth across the sequence.
- Never ask a probability model to do a calculator's job in its head.
What architectural hyperparameters govern the capacity, shape, and parameter count of a Transformer model?
Designing a skyscraper requires choosing the foundation footprint (embedding dimension d_(model) = 768), the number of stories (layers L = 12), the number of elevator shafts per floor (attention heads h = 12), and the width of the common halls (MLP intermediate dimension d_(ff) = 4 × 768 = 3072). In GPT-2 small, these choices produce 124 million parameters. In Llama 3 8B, scaling d_(model) = 4096, L = 32, and h = 32 produces 8 billion parameters. Every hyperparameter directly scales compute and memory requirements.
- Learning goalCalculate the exact parameter count and tensor shapes of a transformer model given its vocabulary size, layer depth, hidden dimension, and attention heads.
- Helpful foundationsMatrix dimension compatibility · Linear layer parameter counting (d_(in) × d_(out) + d_(out))
- Common trapAssuming that attention heads can be sized arbitrarily. The hidden dimension d_(model) must be evenly divisible by the number of attention heads h, yielding the head dimension dₖ = d_(model) / h (e.g. 768 / 12 = 64).
- Next connectionParameter count is the most common metric of model size, but true model scale involves far more than just counting weights.
More context & mechanics
A standard decoder-only transformer parameter tally:
- Embeddings:
- Token Embeddings: |V| × d_(model)
- Position Embeddings: T_(ctx) × d_(model)
- Per Transformer Block (repeated L times):
- Multi-Head Attention:
- W_Q, W_K, W_V: 3 × (d_(model) × d_(model))
- W_O: d_(model) × d_(model)
- Subtotal Attention: 4 d_(model)²
- Feed-Forward MLP:
- W₁: d_(model) × 4 d_(model)
- W₂: 4 d_(model) × d_(model)
- Subtotal MLP: 8 d_(model)²
- LayerNorms: 2 × (2 d_(model)) (scale γ and shift β)
- Total per block ≈ 12 d_(model)²
- Total Parameters:
N ≈ |V| d_(model) + T_(ctx) d_(model) + 12 L d_(model)²
For GPT-2 Small (|V|=50257, d=768, L=12):
N ≈ (50257 × 768) + (1024 × 768) + 12 × 12 × (768)² ≈ 38.6M + 0.8M + 84.9M ≈ 124.4M
Model designers balance aspect ratios: making a model wider (d_(model) \uparrow) increases parallel tensor core throughput, while making it deeper (L \uparrow) increases compositional reasoning capacity.
OpenAI established the classic GPT-2 hyperparameter ladder in 2019: Small (124M, L=12, d=768), Medium (355M, L=24, d=1024), Large (774M, L=36, d=1280), and Extra Large (1.5B, L=48, d=1600), creating standard benchmarks that guided the entire field.
- 12 d² parameters per layer: 4 d² for attention, 8 d² for MLP.
- Every layer is a 12-fold square of its hidden dimension.
Why do memory bandwidth, KV cache footprint, and activation memory dominate the practical scaling of LLMs beyond raw parameter count?
If you buy an 8-passenger van, you might think you only need parking space for the vehicle itself. But when moving a family on vacation, you also need space for suitcases, strollers, food coolers, and a roof rack. In an LLM, the parameters are just the empty van (e.g., 14 GB for a 7B model in FP16). During training, optimizer states (AdamW) consume 8 bytes per parameter (56 GB!), gradients consume 4 bytes (28 GB!), and activation tensors consume tens of gigabytes. During inference, the Key-Value (KV) cache grows linearly with every user and token.
- Learning goalQuantify the memory requirements of an LLM across training (parameters, gradients, optimizer states, activations) and inference (weights and KV cache).
- Helpful foundationsFloating-point precision (FP32, FP16, BF16) · Adam optimizer state tracking (momentum and variance)
- Common trapBelieving that a GPU with 16 GB of VRAM can train a 7B parameter model in FP16 because 7 × 2 = 14 GB. Training a 7B model requires at least 80–120 GB of VRAM without offloading or LoRA due to optimizer states and activations.
- Next connectionHaving mapped the true memory footprint of scale, what does parameter count actually tell us about model capability?
More context & mechanics
Memory demands differ drastically between training and inference:
- Training Memory Breakdown (Mixed Precision FP16/BF16):
- Model Parameters: 2N bytes
- Gradients: 2N bytes (or 4N in FP32)
- Optimizer States (AdamW in FP32):
- FP32 Master Weights: 4N bytes
- Momentum buffer (m_t): 4N bytes
- Variance buffer (v_t): 4N bytes
- Total Optimizer: 12N bytes
- Total Static Memory: ≈ 16N to 20N bytes (for a 7B model ⟹ ≈ 112–140 GB VRAM).
- Dynamic Activations: O(B · L · T · d_(model)).
- Inference Memory Breakdown:
- Weights: 2N bytes (or 0.5N bytes in INT4 quantization).
- KV Cache per token across batch B and context T:
Mem_(KV) = 2 × 2 × B × L × d_(model) × T bytes
Inference servers (such as vLLM) use PagedAttention to fragment the KV cache across physical memory pages, eliminating memory fragmentation and allowing 4× higher concurrent user concurrency on a single GPU.
Sheng et al. (2023) developed FlexGen and Kwon et al. (2023) developed vLLM, establishing that LLM inference is fundamentally memory-bandwidth bound rather than compute bound, shifting hardware design toward high-bandwidth memory (HBM3e).
- Training costs 16–20 bytes per parameter; inference costs 2 bytes plus the KV cache.
- Memory bandwidth, not FLOPS, governs token generation speed.
What is the relationship between model parameter capacity, dataset token volume, and empirical downstream capability under scaling laws?
Think of an Olympic athlete's muscle volume versus their hours of training. A 250-pound bodybuilder has huge muscles (high parameter count), but if they have spent zero hours practicing sprinting on a track (low training tokens), a lean 160-pound decathlete will run circles around them. Similarly, an older 70-billion parameter model (like Chinchilla-era models trained on 1.4T tokens) is beaten soundly by modern 8-billion parameter models (like Llama 3) trained on 15 trillion high-quality tokens. Parameters measure potential capacity; tokens measure realized knowledge.
- Learning goalAnalyze model performance using the Chinchilla scaling tradeoff between parameter capacity N and token volume D.
- Helpful foundationsChinchilla Scaling Laws (Hoffmann et al., 2022) · Overfitting vs underfitting in deep learning
- Common trapAssuming a model with more parameters is always better than a model with fewer parameters. An over-parameterized model trained on too few tokens suffers from underfitting and high inference latency without delivering superior reasoning.
- Next connectionNow let us follow a token through this multi-billion parameter engine to see how language is physically transformed step-by-step.
More context & mechanics
Empirical loss scaling functions:
L(N, D) = E + (A) / (N^α) + (B) / (D^β)
where E is irreducible entropy of natural language, N is parameter count, D is dataset token count, and α ≈ 0.34, β ≈ 0.28.
Key insights:
- Diminishing Returns: Scaling N alone without scaling D results in rapid plateauing of cross-entropy loss reduction.
- Inference Economics (Over-training): While Chinchilla optimal training suggests D ≈ 20N, modern open-source models deliberately train for D > 100N (e.g. Llama 3: 8B parameters trained on 15T tokens ⟹ D / N ≈ 1875). This amortizes training compute across billions of downstream inference queries.
Enterprises prefer deploying heavily over-trained smaller models (like 8B models trained on 15T tokens) because they fit on a single consumer GPU (24GB VRAM in INT4/INT8) while matching the reasoning quality of earlier 70B models.
In 2020, OpenAI's Kaplan et al. proposed scaling laws suggesting parameters should grow faster than data (N ∝ C^(0.73), D ∝ C^(0.27)). In 2022, Hoffmann et al. at DeepMind identified an error in Kaplan's learning rate schedules and published Chinchilla, proving that parameters and data should scale equally (1:1).
- Parameters are the brain's volume; tokens are the books it reads.
- A smaller brain that reads 15 trillion tokens outsmarts a giant that reads only one.
What is the complete end-to-end tensor lifecycle of an input prompt as it travels through a Transformer architecture?
Follow the journey of the word 'Hello': First, it is stamped with a badge number: Token ID 15496. Next, it steps into a dressing room where that single integer expands into a 768-dimensional continuous vector representing its semantic meaning, combined with a positional vector telling it where it stands in line. Then it enters an elevator that takes it up 12 floors (transformer blocks); on each floor, it chats with all previous tokens to update its understanding. At the top floor penthouse, a linear projection expands its 768 numbers into 50,257 voting scores, and softmax converts them into percentage odds for the next word.
- Learning goalTrace the dimensional shape transformations of an input tensor through the complete transformer pipeline: (B, T) → (B, T, d) → (B, T, d) → (B, T, |V|) → (B, |V|).
- Helpful foundationsTensor shapes and multidimensional arrays · Softmax and log-odds calculations
- Common trapThinking that all tokens in the vocabulary are passed into the transformer inputs. Only the prompt tokens are input; the vocabulary size |V| = 50,257 only appears at the very final linear unembedding layer to predict the next token.
- Next connectionThe magical step on each floor of that elevator was 'chatting with all previous tokens'—the mechanism of self-attention.
More context & mechanics
The mathematical pipeline across tensor transformations:
- Input String: `"Hello world"`
- Tokenization: Integer sequence X = [15496, 995] ∈ Z^(1 × T)
- Embedding Layer:
H^((0)) = TokenEmbed(X) + PosEmbed(positions) ∈ R^(B × T × d)
- Stacked Transformer Blocks (l = 1 ṡ L):
H^((l)) = TransformerBlock_l(H^((l-1))) ∈ R^(B × T × d)
- Final Layer Normalization:
H_(norm) = LayerNorm(H^((L))) ∈ R^(B × T × d)
- Unembedding Head:
Z = H_(norm) W_U ∈ R^(B × T × |V|)
- Next-Token Logits Extraction & Softmax:
Taking the last position t = T:
P(w_(T+1)) = softmax(Z[:, -1, :]) ∈ Δ^(|V|-1)
This deterministic forward pass is executed trillions of times daily across modern AI inference clusters. By caching the Key and Value states of past tokens, each new token generation only computes step 7 for t = T rather than re-evaluating the entire sequence.
This architectural pipeline was codified by Radford et al. in GPT-1 (2018), taking the decoder half of Vaswani's translation model and stripping out encoder cross-attention to produce the pure autoregressive decoder architecture that dominates modern LLMs.
- String → IDs → Vectors → Attention Blocks → Logits → Probabilities.
- The input is a sequence of integers; the output is a distribution over the future.
How does the self-attention mechanism enable direct, dynamic pairwise interactions between all tokens in a sequence?
Imagine a room of 100 scholars working on a group research paper. In an old sequential model, scholar 1 whispers to scholar 2, who whispers to scholar 3, until scholar 100 hears a distorted rumor. In Self-Attention, every scholar holds a megaphone (Query), a personal résumé badge (Key), and a briefcase of notes (Value). Scholar 100 asks a question into the megaphone; every other scholar's résumé matches against it with a numerical score. Scholar 100 takes a weighted average of everyone's notes based on those scores. Distance is erased; everyone communicates with everyone in a single step.
- Learning goalDerive scaled dot-product attention Attention(Q, K, V) = softmax(QK^T / √(dₖ))V and explain the scaling factor 1/√(dₖ).
- Helpful foundationsAttention Is All You Need (Vaswani et al., 2017) · Vector dot products as similarity measures · Variance of sum of random variables
- Common trapBelieving that attention scores are static or pre-computed. Attention weights A = softmax(QK^T / √(dₖ)) are dynamically generated on the fly for every unique input sequence, allowing the model to adapt its focus to any context.
- Next connectionWhile attention revolutionized language modeling, not every modern AI architecture is an LLM.
More context & mechanics
Scaled Dot-Product Attention:
Given input matrix X ∈ R^(T × d), compute linear projections:
Q = X W_Q, K = X W_K, V = X W_V (W_Q, W_K, W_V ∈ R^(d × dₖ))
The raw compatibility matrix S = Q K^T ∈ R^(T × T) measures similarity via vector dot products:
S_(i, j) = qᵢ^T kⱼ = ∑_(m=1)^(dₖ) q_(i,m) k_(j,m)
Why divide by √(dₖ)?
If q_(i,m) and k_(j,m) are independent random variables with mean 0 and variance 1, their dot product has mean 0 and variance dₖ. For large dₖ (e.g. 64 or 128), dot products grow large in magnitude, pushing the softmax function into regions with near-zero gradients (vanishing gradients). Dividing by √(dₖ) normalizes the variance back to 1:
Var((qᵢ^T kⱼ) / (√(dₖ))) = 1/dₖ Var(qᵢ^T kⱼ) = 1
Softmax normalizes row-wise into attention weights:
A = softmax((QK^T) / (√(dₖ))), Output = A V
This pairwise dot product is the single most executed mathematical kernel in AI today. Implementations like FlashAttention (Dao et al., 2022) fuse the matrix multiplication, scaling, masking, and softmax into a single GPU SRAM kernel to avoid slow HBM round-trips.
Bahdanau et al. (2014) introduced attention to help RNNs translate long sentences by aligning words between source and target. Vaswani et al. (2017) discarded the RNN entirely, demonstrating that self-attention alone is sufficient for sequence modeling.
- Query seeks; Key matches; Value delivers.
- Divide by √(dₖ) to tame the softmax variance and protect the gradients.
How do Large Language Models differ algorithmically and structurally from other foundational AI architectures like CNNs, GNNs, and Diffusion models?
A Swiss watch, a jet turbine, an electric drill, and a hydraulic crane are all high-precision machines, but you would not use a jet turbine to drill a hole in drywall. In AI, an LLM is specialized for discrete sequential token prediction. If you have tabular banking transactions, gradient boosted decision trees (XGBoost) consistently outperform LLMs. If you have molecular crystal lattices, Graph Neural Networks (GNNs) preserve spatial symmetries. If you generate continuous pixel artwork, Latent Diffusion Models reverse Gaussian noise. Right tool for the right physics.
- Learning goalCategorize neural network architectures (Transformers, CNNs, GNNs, Diffusion models, GBDTs) based on their inductive biases and input data representations.
- Helpful foundationsInductive bias concepts in machine learning · Graph representations and continuous vs discrete spaces
- Common trapAssuming that LLMs are optimal for tabular spreadsheet data or graph relational databases. Tabular data lacks sequential spatial structure; tree ensembles like XGBoost or CatBoost frequently beat LLMs with 1/1000th the compute.
- Next connectionTo contrast inductive biases concretely, let us examine how Convolutional Neural Networks and Transformers build spatial representations differently.
More context & mechanics
Architectural taxonomy across data structures:
- Autoregressive Transformers (LLMs):
- Data: 1D discrete token sequences X = [x₁, ṡ, x_T].
- Inductive Bias: None (all-to-all attention); ordering injected via positional embeddings.
- Objective: Discrete next-token cross-entropy.
- Convolutional Neural Networks (CNNs):
- Data: 2D/3D continuous grid arrays (pixels, voxels).
- Inductive Bias: Locality (nearby pixels interact) and translation equivariance (f(g(x)) = g(f(x))).
- Graph Neural Networks (GNNs):
- Data: Non-Euclidean graphs G = (V, E).
- Inductive Bias: Permutation invariance across node indexing; message passing along edges.
- Diffusion Models:
- Data: Continuous latent manifolds.
- Inductive Bias: Reversible thermodynamic Brownian motion; score matching against Gaussian noise.
Modern autonomous driving (e.g. Tesla FSD, Waymo) combines multiple architectures: CNNs and ViTs process raw camera feeds, GNNs track road topology, and Transformer planners predict vehicle trajectories.
The 2010s saw deep learning specialize into distinct silos: CNNs for vision (AlexNet, 2012), RNNs for speech/language, and GNNs for chemistry. The 2020s witnessed Transformers invading vision (ViT) and biology (AlphaFold), but specialized inductive biases remain vital for tabular and geometric domains.
- Match the architecture's inductive bias to the data's physical geometry.
- Transformers rule sequences; CNNs rule grids; GNNs rule graphs; Trees rule tables.
How do the local inductive biases of CNNs contrast with the global dynamic routing of Transformers?
A Convolutional Neural Network (CNN) is like looking at a painting through a tiny cardboard tube: it scans 3 × 3 patches of pixels, detecting edges on layer 1, textures on layer 2, and noses on layer 3. It has a rigid 'local inductive bias' hardcoded into its structure. A Transformer looks at the entire canvas at once from the very first second. It has zero hardcoded assumptions about locality; it must learn which parts of the image or text relate to each other purely through training data.
- Learning goalContrast the inductive biases, receptive field growth rates, and parameter efficiency of convolutional filters versus self-attention matrices.
- Helpful foundationsConvolution operations and sliding kernels · Translation equivariance and locality
- Common trapThinking that having more inductive bias is always worse. Strong inductive biases (like CNN locality) make models train much faster on small datasets. Transformers only outperform CNNs when trained on massive web-scale datasets where data can overcome the lack of built-in structure.
- Next connectionWhether using a CNN or a Transformer, how does any neural network actually adjust its weights to learn?
More context & mechanics
Mathematical comparison of representations:
- Convolution Operation (Local & Static):
y[i, j] = ∑_(u=-k)^k ∑_(v=-k)^k W[u, v] · x[i+u, j+v]
- Receptive field grows linearly with depth: O(L · k).
- Weight matrix W is fixed regardless of input contents (input-independent weights).
- Invariant to spatial translations.
- Self-Attention Operation (Global & Dynamic):
yᵢ = ∑_(j=1)^T A_(i, j) (xⱼ W_V), A_(i, j) = (exp(qᵢ^T kⱼ / √(d))) / (∑ₘ exp(qᵢ^T kₘ / √(d)))
- Receptive field is global (T) at Layer 1 (O(1) depth to connect any two tokens).
- Weight matrix A_(i, j) is dynamically computed from input contents (input-dependent routing).
- Permutation invariant without positional embeddings.
Hybrid architectures exist: ConvNeXt modernized CNNs using transformer design principles, while Vision Transformers (ViT) replaced convolutions for high-end multimodal models like CLIP and Gemini.
Yann LeCun pioneered CNNs with LeNet-5 (1998) based on Hubel & Wiesel's 1959 biological discovery of local receptive fields in mammalian visual cortex. In 2020, Dosovitskiy et al. (ViT) showed that with enough data, pure attention outperforms biological visual locality.
- CNNs have local blinders; Transformers have global peripheral vision.
- Convolutions use static weights on sliding windows; attention uses dynamic weights across the whole scene.
How do loss functions, automatic differentiation, and gradient descent iteratively update model parameters?
Imagine you are blindfolded on a foggy mountain and your goal is to reach the lowest valley floor (minimum loss). You take a step in all directions to feel which way slopes downward the steepest (calculating the gradient ∇_θ L). Then you take a careful step downhill (gradient descent step). If you take too big a leap (high learning rate), you might plunge off a cliff into numerical instability; if you take tiny baby steps (low learning rate), you will starve on the mountain before reaching the bottom. AdamW keeps a running memory of momentum and friction to glide smoothly to the bottom.
- Learning goalDerive the backpropagation gradient flow ∇_θ L and write the parameter update rule for the AdamW optimizer with weight decay.
- Helpful foundationsMultivariate calculus and chain rule · Gradient descent and optimization dynamics · Decoupled Weight Decay Regularization / AdamW (Loshchilov & Hutter, 2017)
- Common trapConfusing the forward pass with the backward pass. The forward pass computes intermediate activations and scalar loss; the backward pass propagates error gradients in reverse order via the calculus chain rule to calculate (∂ L) / (∂ W) for every weight matrix.
- Next connectionTo drive this optimization loop, the network requires a continuous stream of learning signals: the dataset.
More context & mechanics
The learning loop executes in three phases:
- Forward Pass & Loss:
ŷ = f_θ(x), L(θ) = -∑_(k=1)^(|V|) yₖ log ŷₖ
- Backward Pass (Reverse-Mode Automatic Differentiation):
Applying the calculus chain rule across intermediate activations z_l:
(∂ L) / (∂ W_l) = (∂ L) / (∂ z_l) · ((∂ z_l) / (∂ W_l))^T = δ_l · a_(l-1)^T
- Weight Update via AdamW:
Computes moving averages of gradients m_t (momentum) and squared gradients v_t (uncentered variance):
m_t = β₁ m_(t-1) + (1 - β₁) g_t, v_t = β₂ v_(t-1) + (1 - β₂) g_t²
Bias correction:
m̂_t = (m_t) / (1 - β₁^t), v̂_t = (v_t) / (1 - β₂^t)
Parameter update with decoupled weight decay λ:
θ_t = θ_(t-1) - η ( (m̂_t) / (√(v̂_t) + ε) + λ θ_(t-1) )
PyTorch's `loss.backward()` populates `param.grad` tensors using dynamically constructed directed acyclic computation graphs (DAGs). Calling `optimizer.step()` and `optimizer.zero_grad()` completes the step.
Rumelhart, Hinton, and Williams published their seminal paper on backpropagation in Nature (1986). Kingma & Ba introduced Adam in 2014, and Loshchilov & Hutter corrected weight decay in 2017 with AdamW, which remains the default optimizer for virtually every modern LLM.
- Forward to evaluate; backward to differentiate; step to optimize.
- Gradients tell you the direction of steepest ascent; step negative to descend.
How does self-supervised next-token prediction turn unlabelled natural language corpora into rich supervised learning signals?
In the 2000s, training an AI required paying human annotators millions of dollars to click 'cat' or 'dog' on 1 million images. Self-supervised language modeling performed a brilliant sleight of hand: every sentence written by humanity already contains its own supervision! If an author wrote 'The astronaut boarded the spaceship', the text itself gives you the test: hide 'spaceship', feed the model 'The astronaut boarded the', and measure how surprised the model is when 'spaceship' appears. Trillions of words become trillions of free exam questions.
- Learning goalFormulate self-supervised autoregressive dataset construction as an automated sliding-window token prediction task.
- Helpful foundationsSelf-supervised learning definitions · Cross-entropy loss calculation -∑ y log ŷ
- Common trapCalling LLM pretraining 'unsupervised learning'. LLM pretraining is strictly self-supervised: it uses standard supervised cross-entropy loss, but the target labels are automatically extracted by shifting the raw input text by one token position.
- Next connectionNow that we understand the learning signal and transformer architecture, Lesson 3 examines the critical distinction between pretraining this foundation and fine-tuning it for downstream tasks.
More context & mechanics
Self-supervision via sequence shifting:
Given an arbitrary continuous sequence of token IDs from an internet scrape:
S = [t₁, t₂, t₃, ṡ, t_(k+1)]
We create input features X and target labels Y of length k:
X = [t₁, t₂, ṡ, tₖ]
Y = [t₂, t₃, ṡ, t_(k+1)]
For every position i ∈ 1, ṡ, k, the model predicts yᵢ = t_(i+1) conditioned on prefix x_(≤ i) = [t₁, ṡ, tᵢ].
The cross-entropy loss across all sequence positions is:
L(X, Y) = -1/k∑_(i=1)^k log P_θ(t_(i+1) | t₁, ṡ, tᵢ)
A single document of length 1,024 yields 1,023 distinct supervised prediction signals simultaneously during one parallel forward pass.
This automated data-to-supervision transformation allows AI labs to consume 15 trillion tokens without spending a single dollar on manual human annotation for the base pretraining phase.
Yann LeCun championed the term 'Self-Supervised Learning' to replace the confusing term 'unsupervised learning'. In his famous 'cake analogy', self-supervised learning is the cake itself (infinite bits of feedback from the world), supervised learning is the icing, and reinforcement learning is the cherry on top.
- Shift the sequence by one token: the input creates its own target.
- The text of the world is its own infinite teacher.