Pretraining vs Finetuning
Why is the lifecycle of an LLM fundamentally separated into self-supervised pretraining and supervised fine-tuning?
How this lesson unfolds.
Begin with Why is the lifecycle of an LLM fundamentally separated into self-supervised pretraining and supervised fine-tuning?. Follow the visual diagrams to build an intuitive mental model.
Why is the lifecycle of an LLM fundamentally separated into self-supervised pretraining and supervised fine-tuning?
Think of sculpting a statue from marble. In stage one (pretraining), you quarry a massive 50-ton block of raw stone and use heavy jackhammers and diamond saws for weeks to chisel away 40 tons of rock, shaping the basic anatomical silhouette of a human body—spending 99% of your total energy. In stage two (fine-tuning), you put away the jackhammers, pick up tiny delicate dental chisels and sandpaper, and spend an afternoon etching the facial expression and polishing the fingernails. If you attempted to sculpt the fingernails before quarrying the block, you would have nothing to work on.
- Learning goalFormulate the two-stage transfer learning paradigm of LLMs, distinguishing self-supervised next-token loss from supervised task-adaptation objectives.
- Helpful foundationsTransfer learning and representation learning · Don't Stop Pretraining: Adapt Language Models to Domains and Tasks (Gururangan et al., 2020) · ULMFiT: Universal Language Model Fine-tuning for Text Classification (Howard & Ruder, 2018)
- Common trapThinking that fine-tuning is simply continuing pretraining on a smaller dataset with the same learning rate. Fine-tuning uses learning rates 10x to 100x smaller (10^(-5) vs 10^(-4)), strictly formatted conversational schemas, and often freezes or regularizes parameters to avoid catastrophic forgetting.
- Next connectionTo see why pretraining consumes 99% of the effort, we examine the gargantuan scale and web diet that feeds it.
More context & mechanics
The two-stage training paradigm divides optimization into distinct loss regimes:
- Stage 1: Pretraining (Foundation Representation):
- Data: D_(pretrain) = x₁, x₂, ṡ, x_M, where M ≈ 10^(12)–10^(13) tokens.
- Objective: Minimize empirical cross-entropy over raw next-token prediction:
L_(pretrain)(θ) = -1/M∑_(i=1)^M log P_θ(tᵢ | t_(<i))
- Parameter Trajectory: Weights move far from initial random initialization θ₀, discovering low-loss basins representing syntax, world facts, and reasoning manifolds.
- Stage 2: Fine-Tuning (Behavioral Alignment):
- Data: D_(tune) = (pₖ, rₖ)_(k=1)^K, where K ≈ 10⁴–10⁶ prompt-response pairs.
- Objective: Minimize cross-entropy masked to response tokens only:
L_(SFT)(θ) = -∑_((p, r)) ∑_(j=1)^(|r|) log P_θ(rⱼ | p, r_(<j))
- Parameter Trajectory: Weights make small, localized perturbations (\|θ - θ_(base)\| ≪ \|θ_(base) - θ₀\|) to conform output style to conversational expectations.
This separation underpins the entire AI ecosystem: foundational frontier labs (Meta, Mistral, OpenAI) invest tens of millions of dollars into pretraining base models once; thousands of enterprise software companies fine-tune those base models on internal documents for a few hundred dollars.
Howard & Ruder introduced ULMFiT in 2018, demonstrating that 3-stage transfer learning (general pretraining → domain fine-tuning → task fine-tuning) worked for NLP just as ImageNet pretraining had revolutionized computer vision in 2012.
- Pretraining builds the engine; fine-tuning steers the car.
- Base models learn what the world is; fine-tuned models learn how to speak with humans.
What constitutes the multi-trillion token 'web diet' consumed during base model pretraining?
If a human reads 250 words per minute, 8 hours a day, every day of their life for 70 years, they will consume roughly 300 million words. A modern foundation model like Llama 3 consumes 15 trillion tokens during pretraining—equivalent to the lifelong reading experience of 50,000 human beings combined! This planetary library contains every Wikipedia article in dozens of languages, millions of digitized books, billions of web pages from Common Crawl, the entire open-source code commons of GitHub, and scientific archives from ArXiv.
- Learning goalDesign pretraining data mixture distributions across web crawl, code repositories, academic literature, and encyclopedias, optimizing domain weighting ratios.
- Helpful foundationsTokenization and byte-level representations · The Pile: An 800GB Dataset of Diverse Text for Language Modeling (Gao et al., 2020) · FineWeb: Decanting the Web for the Finest Text Data at Scale (Penedo et al., 2024)
- Common trapAssuming that training data mixtures are just random dumps of the internet. Modern foundation models use carefully tuned mixture distributions: typically 50% filtered web text, 20% code, 15% academic/scientific papers, 10% books, and 5% multilingual sources to balance syntax, reasoning, and factual depth.
- Next connectionRaw web crawl is overwhelmingly noisy and toxic; before any GPU touches it, the data must pass through an extensive cleaning funnel.
More context & mechanics
A representative high-performance pretraining corpus mixture (e.g. FineWeb, The Pile, LLaMA):
- Curated Web Scrapes (50–60%): Common Crawl filtered through URL blocklists, language classifiers, and perplexity scoring.
- Source Code & Documentation (15–20%): GitHub, StackOverflow, permissive open-source repositories. Code teaches models structured logic, execution graphs, and long-range syntactic dependencies.
- STEM & Academic Papers (10–15%): ArXiv, PubMed, OpenAlex. Imparts mathematical rigor, LaTeX equations, and scientific methodology.
- Books & Literary Prose (5–10%): Books3, Gutenberg. Teaches long-context narrative flow and nuanced literary vocabulary.
- Encyclopedias & Reference (3–5%): Wikipedia, Wikihow. Provides densely verified factual knowledge.
Data sampling uses weighted temperature-based upsampling:
p_d = (w_d · S_d) / (∑ₖ wₖ Sₖ)
where high-quality sources (Wikipedia, textbooks) are upsampled multiple epochs (w_d > 1), while noisy web text is downsampled (w_d < 1).
Pretraining data engineers tune mixture ratios dynamically during training (data curriculum): starting with broad web corpora to learn general grammar, then annealing the mixture toward high-quality math, code, and synthetic reasoning datasets in the final 10% of training tokens.
In 2020, EleutherAI created 'The Pile' (825 GB across 22 diverse sub-corpora), proving that diverse data mixtures drastically outperformed monolithic web crawls. In 2024, Hugging Face released FineWeb (15T tokens), demonstrating that meticulous filtering recipes can boost reasoning by 10+ accuracy points on standard benchmarks.
- You are what you eat; an LLM is what it is pretrained on.
- Code teaches logic; literature teaches narrative; science teaches precision.
What algorithmic stages are required to filter, deduplicate, and curate multi-terabyte web corpora into high-quality training tokens?
Imagine panning for gold in a muddy river. For every metric ton of river silt, mud, and plastic trash you shovel into your sluice box, you might recover a few grams of pure gold dust. Raw internet scrape (Common Crawl) is the muddy river: 80% to 90% is SEO spam, machine-translated garbage, duplicate marketing scrapers, casino ads, and toxic rants. A rigorous multi-tier filtration cascade acts as a series of finer and finer mesh screens that discards 85% of the raw tonnage, preserving only the clean, grammatically coherent gold.
- Learning goalImplement the end-to-end data curation pipeline: text extraction, language identification, MinHash deduplication, heuristic filtering, and quality classification.
- Helpful foundationsLocality-Sensitive Hashing (LSH) and MinHash · Perplexity filtering with n-gram or lightweight language models · RefinedWeb Dataset for Falcon LLM (Penedo et al., 2023)
- Common trapBelieving that more data always beats cleaner data. Training on 2 trillion tokens of highly filtered, deduplicated text consistently yields stronger downstream reasoning than training on 10 trillion tokens of uncurated web noise.
- Next connectionOnce a general base model is pretrained on this clean foundation, how do we specialize it for specialized high-stakes domains?
More context & mechanics
The multi-stage data curation funnel:
- Raw HTML Extraction & Text Parsing: Trafilatura or Resiliparse strips CSS, JavaScript tags, and navigation headers from WARC files.
- Language Identification: FastText classifies text; documents below a 0.65 target language confidence score are discarded.
- Heuristic Quality Filtering:
- Word length distribution (average word length between 3 and 10 characters).
- Symbol-to-word ratio (discard pages where >30\% of characters are special punctuation).
- Repetition filters (stripping pages with repetitive unigram/bigram loops caused by scraper glitches).
- Fuzzy Deduplication (MinHash LSH):
- Documents are broken into k-shingles (e.g. 5-grams).
- M hash functions compute the signature:
h_(min)(D) = min_(s ∈ D) hₘ(s)
- Jaccard similarity J(D₁, D₂) = (|D₁ ∩ D₂|) / (|D₁ ∪ D₂|) estimated by hash collisions. Duplicate clusters with J > 0.8 are purged.
- Model-Based Perplexity & Toxicity Filtering:
- A fast n-gram KenLM or small BERT classifier scores readability and toxicity; outlier documents in the high-perplexity tail are purged.
The Falcon RefinedWeb pipeline processed 5 petabytes of raw Common Crawl and discarded 90% of it, resulting in a 600-billion token dataset that propelled Falcon-40B to the top of open-source leaderboards in 2023.
Broder introduced MinHash in 1997 for AltaVista search engine indexing. Two decades later, Lee et al. (2021) in 'Deduplicating Training Data Makes Language Models Better' showed that deduplication directly reduces memorization of PII, prevents regurgitation, and improves training speed.
- Extract, filter, deduplicate, score.
- Throw away 85% of the web to keep the 15% that teaches reasoning.
How does domain-adaptive continued pretraining and fine-tuning specialize a base LLM for law, medicine, or finance?
A brilliantly educated Oxford literature graduate knows how to write exquisite essays and debate logic, but if you drop them into a hospital emergency room and ask them to diagnose an acute myocardial infarction from an ECG printout, they will hesitate. They already understand biology, chemistry, and reading; what they need is a specialized medical residency. Domain fine-tuning takes a generalist base model and steeps it in 100,000 medical case studies, clinical trials, and pharmacological formulas to turn it into a clinical specialist.
- Learning goalExecute domain adaptation using continued pretraining with replay buffers and domain-specific instruction fine-tuning.
- Helpful foundationsDomain adaptation theory · Catastrophic forgetting mechanics
- Common trapAssuming that fine-tuning on domain data alone is sufficient if the domain uses specialized jargon absent from the base tokenizer. If specialized medical or legal terms get split into 6 fragmented byte pieces by a general tokenizer, the model will struggle to learn robust embeddings; domain tokenizers or continued pretraining are required.
- Next connectionWhen adapting a model, engineers face two distinct pathways: conversational instruction tuning versus task-specific classification.
More context & mechanics
Domain specialization involves two progressive techniques:
- Domain-Adaptive Continued Pretraining (DAPT):
- Continue next-token prediction on specialized corpora (e.g. 50B tokens of SEC filings or PubMed abstracts).
- Use a lower learning rate with cosine decay: η_(DAPT) ≈ 0.1 × η_(pretrain).
- Preventing Catastrophic Forgetting: Mix in a replay buffer of general pretraining data:
D_(train) = 0.8 × D_(domain) + 0.2 × D_(general)
- Domain-Specific Instruction Fine-Tuning:
- Supervised fine-tuning on expert-curated instructions (e.g. legal brief drafting, diagnostic case analysis).
- Formatted with explicit domain constraints, citation requirements, and confidence scoring.
BloombergGPT (Wu et al., 2023) trained a 50-billion parameter model on a 363-billion token financial dataset alongside 345 billion tokens of general text, outperforming general LLMs on financial sentiment, NER, and question answering.
Gururangan et al. (2020) at AI2 systematically studied 'Don't Stop Pretraining', showing that continuing pretraining on domain-specific unlabelled text before task fine-tuning achieved consistent state-of-the-art gains across biomedical, computer science, and news domains.
- General knowledge first, domain residency second.
- Always mix in 15–20% general data to protect against catastrophic forgetting.
What are the structural, architectural, and data differences between Instruction Fine-Tuning and Classification Fine-Tuning?
Imagine training a prize dog. In Pathway 1 (Classification Fine-Tuning), you strap a red button and a green button to the dog's collar; whenever it smells a package, it presses the red button for contraband or the green button for safe. The dog never barks or speaks; it just flips a binary switch. In Pathway 2 (Instruction Fine-Tuning), you teach the dog to speak fluent English so that when you ask 'What is inside this package and why is it dangerous?', it can explain the chemical contents and suggest handling precautions.
- Learning goalDifferentiate between replacing the final unembedding layer with a task-specific classification head versus fine-tuning the entire causal decoder via instruction prompt schemas.
- Helpful foundationsCross-entropy classification loss vs next-token causal loss · Chat template formatting schemas (ShareGPT, Alpaca)
- Common trapConfusing parameter-efficient fine-tuning (PEFT/LoRA) with classification fine-tuning. PEFT/LoRA is an optimization method that freezes base weights and trains low-rank adapter matrices; it can be applied to either instruction tuning or classification tuning.
- Next connectionTo appreciate why this two-stage paradigm dominates AI, we must comprehend the staggering financial and computational scale of modern foundation models.
More context & mechanics
Architectural comparison of the two fine-tuning pathways:
- Pathway A: Classification / Task-Specific Fine-Tuning:
- Architecture: Discard the |V|-dimensional unembedding head W_U. Attach a linear classification head W_(cls) ∈ R^d_(model) × C on the final token's hidden state h_T:
P(y = c | X) = softmax(h_T W_(cls))
- Output: Discrete class probabilities over C labels (e.g. spam / not spam).
- Loss: Standard cross-entropy loss over class labels: L = -log P(y_(true) | X).
- Pathway B: Instruction Fine-Tuning (SFT):
- Architecture: Keep the identical autoregressive causal transformer and full vocabulary unembedding head W_U ∈ R^d_(model) × |V|.
- Data Schema: Turn-based prompt-response schemas:
`<|user|> Explain photosynthesis.<|assistant|> Photosynthesis is...<|endoftext|>`
- Loss: Next-token cross-entropy calculated only across the response tokens, masking out prompt token losses.
Classification fine-tuning is preferred for high-throughput, low-latency production pipelines (moderation flags, sentiment routing) where only a label is needed. Instruction fine-tuning powers conversational assistants, agents, and interactive coding copilots.
OpenAI popularized instruction fine-tuning with InstructGPT (Ouyang et al., 2022). Taori et al. (Stanford Alpaca, 2023) demonstrated that a 7B model could be instruction-tuned on 52,000 synthetic examples for under 500, democratizing post-training for the global developer community.
- Classification adds a label head; instruction tuning keeps the language head.
- Labels give you decisions; instructions give you conversation.
What are the capital, energy, and hardware infrastructure requirements of modern foundation model pretraining?
Training a modern frontier LLM is not a software project you run on a laptop in your bedroom; it is a mega-infrastructure engineering feat comparable to constructing a commercial nuclear power plant or launching a satellite constellation. A single training run for a 70B+ model requires 16,000 to 24,000 H100 GPUs networked together via 3.2 Tbps InfiniBand cables, consuming tens of megawatts of electrical power continuously for 3 months, at a cost exceeding 50 million. That is why foundation pretraining is performed only once, while fine-tuning is performed thousands of times by downstream developers.
- Learning goalEstimate the capital costs, GPU cluster requirements, power draw, and training duration of foundation models using hardware FLOPs calculations and MFU metrics.
- Helpful foundationsGPU cluster architecture and interconnect bandwidth (NVLink, InfiniBand) · Model FLOPs Utilization (MFU) formulas
- Common trapAssuming that training compute scales linearly with parameter count alone. Because dataset token counts must scale proportionally to parameter counts to remain compute-optimal (D ∝ N), total training compute scales quadratically (C ≈ 6ND ∝ N²).
- Next connectionHaving surveyed the grand economic and operational scale of foundation models, Lesson 4 dives deep into the mathematical core that makes them work: the basics of Transformers.
More context & mechanics
Quantifying pretraining compute economics:
- Total Compute (FLOPs):
C ≈ 6 × N × D
For Llama 3 70B (N = 70 × 10⁹) trained on 15T tokens (D = 15 × 10^(12)):
C ≈ 6 × (70 × 10⁹) × (15 × 10^(12)) ≈ 6.3 × 10^(24) FLOPs = 6.3 YottaFLOPs
- Hardware Cluster Provisioning:
An NVIDIA H100 SXM5 GPU delivers ≈ 989 × 10^(12) BF16 Tensor Core FLOPs/sec.
Assuming Model FLOPs Utilization (MFU) of 45\% (achievable FLOPs ≈ 445 × 10^(12)):
Total GPU-Seconds = (6.3 × 10^(24)) / (445 × 10^(12)) ≈ 1.415 × 10^(10) GPU-seconds ≈ 3.93 × 10⁶ GPU-hours
On a 16,384 H100 GPU cluster:
Wallclock Training Time = (3.93 × 10⁶) / (16384) ≈ 240 hours ≈ 10 days
- Power Consumption:
16,384 H100 servers (including networking and cooling PUE 1.2) consume ≈ 20 Megawatts of continuous power.
Because base pretraining requires tens of millions of dollars in capital expenditure, the foundation model layer operates as a shared utility: a handful of hyperscalers train base checkpoints, which millions of software developers specialize and deploy via fine-tuning or prompt engineering.
Meta's Llama 3 technical report (2024) disclosed that training the 405B model required over 16,000 H100 GPUs running for 54 days with automated failure recovery handling hundreds of GPU and network link disruptions without halting progress.
- C = 6ND: The yottaflop equation of modern AI.
- Pretrain once at planetary cost; fine-tune everywhere at minimal cost.
Why do recurrent neural networks suffer from an information and gradient bottleneck over long sequences, and how did this failure necessitate the transformer?
Imagine a 100-person game of telephone where each player is only allowed to hold a single 3x5 index card. Player 1 writes down a 20-word message. Player 2 must erase words to fit their own update, and by Player 50, the card contains only fragments of the most recent words. Recurrent Neural Networks (RNNs) force every past token through a fixed-size vector h_t in R^d, acting like a narrow sequential straw that chokes long-range context.
- Learning goalDerive the backpropagation through time (BPTT) Jacobian chain rule and calculate vanishing gradient decay rates over sequential time steps.
- Helpful foundationsAttention Is All You Need (Vaswani et al., 2017) · Learning Phrase Representations using RNN Encoder-Decoder (Cho et al., 2014) · Long Short-Term Memory (Hochreiter & Schmidhuber, 1997)
- Common trapAssuming gating mechanisms like LSTMs or GRUs permanently solved the recurrent bottleneck. While gates alleviate vanishing gradients for moderate lengths (T approx 100), they remain strictly sequential (O(T) forward pass) and cannot parallelize on modern GPU Tensor Cores.
- Next connectionTo eliminate the sequential bottleneck entirely and compute direct connections between every pair of tokens in parallel, Lesson 04 introduces the transformer architecture: 'Attention Is All You Need'.
More context & mechanics
In a vanilla RNN, the hidden state at step t is updated sequentially:
h_t = \tanh(W_(hh) h_(t-1) + W_(xh) x_t + b)
During backpropagation through time (BPTT), the error gradient with respect to the initial state h₁ scales as a product of Jacobian matrices:
(∂ L) / (∂ h₁) = (∂ L) / (∂ h_T) ∏_(k=2)^T (∂ hₖ) / (∂ h_(k-1)) = (∂ L) / (∂ h_T) ∏_(k=2)^T diag(1 - hₖ²) W_(hh)^T
Toy Numerical Trace: Suppose the recurrence weight eigenvalue is λ = 0.5. Over a short sequence of T = 10 tokens, the gradient multiplier scales as:
(0.5)^(10) ≈ 0.0009765
Over T = 50 tokens, (0.5)^(50) ≈ 8.88 × 10^(-16), vanishing to machine precision zero! Conversely, if λ = 1.5, (1.5)^(10) ≈ 57.66 and (1.5)^(50) ≈ 6.38 × 10⁸, triggering catastrophic exploding gradients.
Understanding this recurrent bottleneck explains why frontier models (Llama 3, GPT-4, Claude 3.5) discarded step-by-step recurrence entirely in favor of all-to-all attention mechanisms, enabling massive GPU parallelization across millions of tokens.
Identified mathematically by Sepp Hochreiter in his 1991 German diploma thesis on vanishing gradients, leading to the LSTM in 1997 (Hochreiter & Schmidhuber). Even LSTMs, however, could not escape the sequential O(T) execution barrier on modern GPU hardware.
- Sequential recurrence forces a fixed-width straw; transformers provide an all-to-all optical fiber.
- When λ < 1, gradients vanish to zero; when λ > 1, gradients explode to infinity.