Supervised Finetuning (SFT) Execution Loop & Training Dynamics
How does the full-parameter supervised finetuning loop orchestrate forward passes, masked loss computation, gradient accumulation, and checkpoint saving?
How this lesson unfolds.
Having structured our instruction dataset and custom collator, we now build the complete Supervised Finetuning (SFT) training engine. This loop takes a raw pretrained base model and transforms it into an instruction-following assistant.
How does Supervised Fine-Tuning (SFT) transform an unaligned internet text completer into an instruction-following assistant?
Imagine hiring an eccentric savant who has read all 50 million books in the Library of Alexandria, but has lived alone in a cave their entire life. If you walk up and ask: 'What is the capital of France?', the savant does not answer your question; instead, they start reciting the table of contents of an atlas, or quote three similar quiz questions from a 1980s trivia magazine. They have all human knowledge, but zero manners. Supervised Fine-Tuning is the etiquette finishing school: it teaches the savant the conversational contract of human dialogue: when someone asks a question, provide the direct answer and stop.
- Learning goalExplain how Supervised Fine-Tuning aligns pretrained base models into conversational assistants using formatted instruction-response pairs.
- Helpful foundationsAutoregressive next-token prediction · LIMA: Less Is More for Alignment (Zhou et al., 2023)
- Common trapBelieving that SFT is where the model learns factual knowledge. Pretraining provides 99% of the model's factual knowledge base; SFT merely teaches the model how to surface that knowledge conversationally.
- Next connectionNext, derive the target masking calculus (`ignore_index = -100`) and loss dynamics in Part B.
More context & mechanics
Pretraining and Supervised Fine-Tuning serve two complementary functions:
- The Pretrained Baseline (Document Completer):
A base model trained on raw internet text optimizes next-token prediction across uncurated prose. Given the prompt:
`"Write a poem about the sea."`
A base model might continue with:
`"Write a poem about the mountains. Write a poem about the sky."`
It views the prompt as a list of homework exercises to continue, not as an order to obey.
- The SFT Dialogue Contract:
SFT trains the model on curated demonstration pairs formatted with special delimiter tokens:
`<|im_start|>user
Write a poem about the sea.<|im_end|>
<|im_start|>assistant
The salt spray dances...<|im_end|>`
This teaches the model to recognize conversational boundaries, adopt a helpful persona, and terminate generation using `<|im_end|>`.
- The Format vs. Knowledge Duality:
Research shows that SFT imparts very little new world knowledge; rather, it acts as a superficial alignment filter (the "LIMA hypothesis"), unlocking knowledge already compressed during pretraining.
SFT is the mandatory second stage in every production AI pipeline (ChatGPT, Claude, Llama-Chat). Foundation models spend 99% of their compute on pretraining and less than 1% on SFT, yet SFT delivers 90% of the perceived user experience improvement.
Instruction tuning was pioneered by Jason Wei et al. (FLAN, 2021) and Long Ouyang et al. (InstructGPT, 2022). In 2023, Zhou et al. published "LIMA: Less Is More for Alignment", proving that just 1,000 meticulously curated SFT examples suffice to produce world-class conversational quality.
- Pretraining gives knowledge; SFT gives manners: Pretraining builds the brain; SFT trains the voice.
- Instruct, demonstrate, terminate: Teach the model when to speak, what persona to adopt, and when to shut up.
Why does computing loss on user prompt tokens degrade assistant quality, and how does `-100` target masking restrict gradients to assistant responses?
Imagine training an actor to perform in a theater play. If you punish the actor every time the narrator on stage speaks, the actor becomes terrified of hearing other people talk and forgets their own lines. You only grade the actor when they deliver their lines. In SFT, the user prompt is the narrator, and the assistant answer is the actor. Masking the user prompt with -100 ensures the student is only graded on their own performance.
- Learning goalFormulate the masked SFT loss objective using `ignore_index = -100` and prove that prompt masking focuses gradient updates strictly on response tokens.
- Helpful foundationsCross-entropy loss with `ignore_index` · SFT pipeline principles
- Common trapBelieving that prompt tokens do not need to be processed by the model. Prompt tokens must be processed in the forward pass to provide causal context; they are only masked in the loss computation and backward pass.
- Next connectionNext, learn how causal LLMs can be adapted for text classification via last-token pooling in Chapter 35 Screen 4.
More context & mechanics
#### 1. The Prompt Loss Catastrophe
If standard cross-entropy is applied across the entire concatenated sequence [Prompt, Response]:
L = (1) / (T_p + T_r) ( ∑_(t=1)^(T_p) L_t + ∑_(t=T_p+1)^(T_p + T_r) L_t )
The model wastes significant gradient capacity learning how to predict arbitrary user prompts (e.g. predicting typos, formatting quirks, and questions). This degrades instruction adherence and encourages the model to generate simulated user questions rather than answers.
To focus learning exclusively on the assistant response, target labels for prompt tokens are replaced with the sentinel value `-100`:
y_t^* = -100 & if t ≤ T_(prompt) \ y_t & if t > T_(prompt)
The masked SFT loss objective is:
L_(SFT) = - (1) / (T_(response)) ∑_(t=T_p+1)^(T_p + T_r) ln P(y_t | x_(1:t-1))
Any token with y_t^* = -100 emits exactly zero gradient during backpropagation.
Given a batch of length 6:
- Tokens: `["What", "is", "2+2?", "It", "is", "4"]`
- Prompt tokens: indices 0, 1, 2 (length T_p = 3)
- Response tokens: indices 3, 4, 5 (length T_r = 3)
Target vector with prompt masking:
y = [-100, -100, -100, 15496, 318, 19]
During loss calculation:
- Indices 0, 1, 2: `ignore_index = -100` ⟹ Loss = 0.0, Gradient = 0.0.
- Indices 3, 4, 5: Computed normally.
L_(total) = (L₃ + L₄ + L₅) / (3)
Zero gradient updates affect the user prompt representations.
In frameworks like Hugging Face `TRL` (Transformer Reinforcement Learning) and Axolotl, `DataCollatorForCompletionOnlyLM` automatically identifies response template delimiters (such as `<|assistant|>
`) and masks all preceding tokens with `-100`.
The `-100` convention stems from PyTorch's `nn.CrossEntropyLoss(ignore_index=-100)`. In 2022, Taori et al. (Stanford Alpaca) standardized prompt masking as the required best practice for open-source instruction tuning.
- Mask the prompt with -100: Never penalize the model for what the user wrote.
- Grade only the assistant: Gradients must focus 100\% of their capacity on the response.
How does PyTorch compute average loss across variable unmasked token counts without introducing batch-size or length biases?
Imagine grading a class test where student A wrote 2 answers and student B wrote 10 answers. If you divide both students' total points by 100, student A will look like they failed simply because their test was shorter! Masked loss divides by the exact number of questions answered, ensuring a fair, calibrated score regardless of response length.
- $$\mathcal{L} = \frac{\sum \text{NLL}_i \cdot \mathbb{I}(y_i \neq -100)}{\sum \mathbb{I}(y_i \neq -100)}$$
- Learning goalAudit the reduction denominator of masked cross-entropy loss to prevent artificial gradient dilution.
- Helpful foundationsPyTorch CrossEntropyLoss ignore_index=-100 mechanics · Target masking dilemma in instruction tuning · PyTorch Loss Functions & CrossEntropyLoss Specification
- Common trapDividing the loss sum by the total tensor size (B*T); doing so artificially dilutes loss on heavily padded batches, causing erratic optimization steps.
- Next connectionTrack how this loss evolves across epochs in SFT Training Trajectory (Screen 3).
More context & mechanics
Given logit tensor Z ∈ R^(B × T × V) and target labels Y ∈ Z^(B × T):
L = (∑_(b=1)^B ∑_(t=1)^T -ln p(y_(b, t)) · I(y_(b, t) ≠ -100)) / (∑_(b=1)^B ∑_(t=1)^T I(y_(b, t) ≠ -100))
- Numerator: Sum of negative log likelihoods strictly over valid response positions.
- Denominator: N_(valid) = ∑_(b, t) I(y_(b, t) ≠ -100).
If you mistakenly divide by B × T, batches with heavy padding will report artificially low loss, causing the optimizer to take deceptively small gradient steps.
Consider a batch of B = 2 sequences with T = 4:
- Sample 1: Targets `[-100, -100, 42, 88]` (2 prompt tokens, 2 response tokens). Response losses = 1.2, 0.8.
- Sample 2: Targets `[-100, 15, 29, 31]` (1 prompt token, 3 response tokens). Response losses = 0.5, 0.4, 0.6.
- Numerator Sum: 1.2 + 0.8 + 0.5 + 0.4 + 0.6 = 3.50.
- Valid Token Count (N_(valid)): 2 + 3 = 5 tokens.
- Correct Masked Loss: L = (3.50) / (5) = 0.700 nats.
(Flawed static average: (3.50) / (2 × 4) = 0.4375 — artificially diluted by 37.5% due to padding!).
Configured via `torch.nn.CrossEntropyLoss(ignore_index=-100, reduction='mean')`.
Standardized in deep learning frameworks to handle variable-length sequence modeling and packed attention buffers.
Count only the living tokens: divide loss by valid answers, never by empty air.
What is the typical convergence trajectory of an SFT run, and why does validation loss plateau much earlier than pretraining?
Imagine learning etiquette after already knowing how to speak. It doesn't take 10 years to learn to say 'Please' and 'Thank you'; a weekend workshop is enough. SFT doesn't teach the model new facts; it teaches tone and conversational etiquette. If you drill the etiquette class for 5 years, the student becomes an obsessive robot who repeats canned phrases.
- $$\text{Optimal Epochs}_{SFT} \in [2, 3]$$
- Learning goalAnalyze SFT convergence dynamics and determine the optimal early stopping checkpoint to avoid instructional overfitting.
- Helpful foundationsSFT loop architecture and masked cross-entropy · LIMA: Less Is More for Alignment (Zhou et al., Meta AI, 2023)
- Common trapBelieving SFT should be trained for 10+ epochs until loss reaches near-zero; overtraining SFT destroys creativity and diversity, producing repetitive robotic responses.
- Next connectionAudit live instruction completions during training in Screen 4.
More context & mechanics
The SFT loss trajectory follows three phases:
- Phase 1: Format Alignment (Epoch 0 to 0.5):
Loss drops precipitously from ≈ 3.2 to ≈ 1.5 as the model learns prompt templates, special tokens (`### Instruction`, `### Response`), and `<|endoftext|>` stopping behavior.
- Phase 2: Task Mastery (Epoch 0.5 to 2.0):
Loss gently settles from 1.5 to ≈ 0.9. The model learns concise reasoning and helpful instructional structure.
- Phase 3: Overfitting & Memorization (Epoch > 2.5):
Training loss continues to fall (0.9 → 0.3), but validation loss plateaus and creeps upward. The model begins memorizing specific examples verbatim.
Finetuning GPT-2 on 52,000 Alpaca instruction pairs:
- Step 0: Base Cross-Entropy Loss = 3.45, Perplexity = 31.5.
- Step 500 (Epoch 0.5): Loss = 1.62, Perplexity = 5.05 (Understands prompt headers).
- Step 2,000 (Epoch 2.0): Loss = 0.98, Val Loss = 1.04 (Optimal checkpoint!).
- Step 4,000 (Epoch 4.0): Loss = 0.41, Val Loss = 1.35 (Overfitting: repeats training templates).
Training must be stopped at Epoch 2 to maximize real-world helpfulness.
Standard recipe across industry labs: LLaMA-2 SFT and Mistral-Instruct are trained for strictly 2 to 3 epochs.
Documented by Zhou et al. (Meta AI, 2023) in 'LIMA: Less Is More for Alignment', proving that 1,000 curated instruction pairs are sufficient to align foundation models.
Pretraining takes months to learn the world; SFT takes hours to learn the manners: 2 epochs is the golden rule.
Why are numeric loss values insufficient to gauge conversational quality, and how does in-training generation catch alignment failure modes early?
Imagine judging a chef solely by how much salt and butter they used (numeric metrics). The measurements might be mathematically flawless, but you have no idea if the soup actually tastes good until you take a spoon and taste it! Sampling completions during training is tasting the soup.
- $$\text{Prompt} \xrightarrow{\text{Autoregressive Decode}(T, p)} \text{Sample String}$$
- Learning goalIntegrate periodic autoregressive text generation hooks into the SFT training loop for qualitative alignment auditing.
- Helpful foundationsTemperature and top-p sampling algorithms · Model evaluation mode dynamics
- Common trapRelying solely on loss curves; a model can suffer severe degenerative repetition while reporting deceptively low loss values.
- Next connectionPersist training state safely in SFT Checkpoint Persistence (Screen 5).
More context & mechanics
Every K training steps (e.g. 250 steps), the training loop pauses training and enters evaluation mode:
- `model.eval()`: Freezes dropout and activates deterministic attention.
- Feed fixed benchmark prompts (e.g. 'Explain photosynthesis to a 5-year-old', 'Write a Python function to reverse a list').
- Run autoregressive generation loop with temperature T=0.7, Top-p=0.9.
- Log output strings directly to terminal or Weights & Biases (W&B) dashboard.
- `model.train()`: Restore training state and resume gradient descent.
Tracing prompt: 'Write a poem about the ocean.' across training steps:
- Step 0 (Base Model): 'Write a poem about the ocean. The ocean is blue. The sea is salty. Chapter 4: Geography of Oceans...' (Continues prompt mechanically like a web page).
- Step 250 (Format Emergence): '### Response: The waves crash upon the shore / Deep and blue forevermore / The salty breeze...' (Understands the instruction header!).
- Step 1,000 (Aligned Generation): Generates a complete, lyrical 4-stanza poem, and crucially halts execution by emitting the `<|endoftext|>` token!
Standard diagnostic in all LLM training frameworks to catch catastrophic degradation, hallucinations, and repetitive loop traps.
Established by Karpathy in nanoGPT and standardized across production alignment suites (OpenAI, Anthropic).
Loss tracks the numbers; sampling reveals the soul: always taste the model's words as it learns.
How do atomic file writes, SafeTensors serialization, and complete optimizer state dicts prevent corruption and enable seamless training resumption?
Imagine writing a 500-page novel. If you only save by overwriting the single master file on your flash drive while the power is flickering, a power surge will leave you with a corrupt, unreadable file and you lose everything. Atomic checkpointing writes a backup copy first, verifies its integrity, and only then updates the bookmark.
- $$\text{Size}_{ckpt} = 2 \times N_{params} \text{ (weights)} + 8 \times N_{params} \text{ (optimizer)}$$
- Learning goalImplement atomic SafeTensors checkpoint persistence and fault-tolerant training resumption in Python/PyTorch.
- Helpful foundationsPyTorch state_dict mechanics · Operating system atomic file renaming · SafeTensors: Simple, Safe and Fast Format for Storing Tensors (Hugging Face)
- Common trapSaving only model weights; if you do not save optimizer momentum states (m_t, v_t), resuming training resets optimizer memory, causing severe training spikes and lost progress.
- Next connectionTransitions directly to Chapter 35 to adapt foundation models for classification tasks.
More context & mechanics
A production checkpoint must store complete execution state:
S_(ckpt) = model: W, \ optimizer: m_t, v_t, \ scheduler: α_t, \ step: t, \ loss: L
- SafeTensors Serialization:
Replaces insecure `pickle.dump()` with pure byte-level Safetensors, preventing arbitrary remote code execution and enabling memory-mapped zero-copy loading:
save\_file(model.state\_dict(), 'model.safetensors')
- Atomic Write Protocol:
Write to `checkpoint_temp.pt`, then execute atomic OS rename `os.replace('checkpoint_temp.pt', 'checkpoint_final.pt')`. If the machine crashes mid-write, the existing checkpoint remains 100% pristine.
Checkpoint storage audit for 355M parameter GPT-2:
- Model Weights (FP16): 355 × 10⁶ × 2 bytes = 710 MB (`model.safetensors`).
- Optimizer State (FP32 moments m_t, v_t): 355 × 10⁶ × 8 bytes = 2.84 GB (`optimizer.pt`).
- Metadata JSON: ≈ 2 KB (`config.json`, step counter, loss history).
- Total checkpoint size: ≈ 3.55 GB.
- Saving frequency: Every 1,000 steps, retaining the best 3 validation checkpoints.
Standard protocol implemented in Hugging Face Hub, PyTorch Lightning, and cloud LLM training clusters.
Developed by Hugging Face (2022) with the SafeTensors standard to replace dangerous PyTorch `.bin` pickles following widespread security audits.
Never pickle untrusted bytes; never overwrite a master file directly: atomic writes keep your weights immortal.