LoRA Attention Targets, QLoRA, and Foundation Tradeoffs
Which specific projection layers inside a transformer block yield the highest performance gains when adapted with LoRA?
How this lesson unfolds.
Knowing how LoRA works mathematically is only half the battle. In this chapter, we master where to inject adapters across the transformer block and how to deploy 4-bit QLoRA.
Which specific projection layers inside a transformer block yield the highest performance gains when adapted with LoRA?
Imagine tuning a 12-piece orchestra. If you only tune the first violin and the trumpet (query and value), the music sounds better, but the other sections clash. Giving slight, subtle tuning to every single instrument in the orchestra (all projections with low rank) creates far richer harmony than heavily tuning just two.
- $$\text{Params} = 2 \cdot r \cdot \sum_{l \in \mathcal{L}} d_{\text{in}}^{(l)}$$
- Learning goalCompare adapter placement strategies across transformer projections and explain why adapting all layers with lower rank is optimal.
- Helpful foundationsLoRA matrix factorization · Multi-head attention projections
- Common trapRestricting LoRA exclusively to query and value projections; adapting MLP layers unlocks crucial factual and syntactic alignment capacity.
- Next connectionReduce memory further using 4-bit NormalFloat quantization in Screen 2.
More context & mechanics
A standard transformer block consists of:
- Multi-Head Attention Projections: W_q, Wₖ, W_v ∈ R^(d × d) and output projection W_o ∈ R^(d × d).
- Feedforward / MLP Projections: W_(up) ∈ R^(d × 4d) and W_(down) ∈ R^(4d × d).
Adaptation Strategy Comparison:
- Strategy 1 (Classic LoRA): Apply to W_q and W_v with rank r = 16.
- Strategy 2 (Comprehensive LoRA / QLoRA): Apply to W_q, Wₖ, W_v, W_o, W_(up), W_(down) with rank r = 4.
Empirical findings (Dettmers et al., 2023) show Strategy 2 captures significantly more expressive capacity for the same parameter budget because it permits stylistic adjustments across both attention routing and factual associative memory in the MLP.
Let d = 768 (GPT-2 124M):
- Strategy 1 (Only W_q, W_v with r=16):
- Per layer: 2 × (2 × 768 × 16) = 49,152 parameters.
- Across 12 layers: 12 × 49,152 = 589,824 params.
- Strategy 2 (All 6 projections with r=4):
- Attention (W_q, Wₖ, W_v, W_o): 4 × (2 × 768 × 4) = 24,576 params.
- MLP (W_(1), W_(2)): (2 × 768 × 4) + (2 × 3072 × 4) = 6,144 + 24,576 = 30,720 params.
- Per layer: 24,576 + 30,720 = 55,296 params.
- Across 12 layers: 12 × 55,296 = 663,552 params.
Both strategies use ≈ 0.6 M parameters (< 0.5\% of GPT-2), but Strategy 2 achieves 1.8\% lower validation loss on instruction tasks.
Standard recommendation in QLoRA paper and HuggingFace PEFT config for LLaMA and Mistral tuning.
Pioneered by Dettmers et al. in the QLoRA study (2023), overturning the earlier assumption that adapting W_q, W_v was sufficient.
Spread thin beats piling thick: adapt all projections with low rank rather than two with high rank.
How does QLoRA achieve 4-bit weight precision without catastrophic accuracy degradation during finetuning?
Standard 4-bit integers divide numbers into evenly spaced buckets, which is wasteful because neural network weights are not evenly spread—they bunch up around zero like a bell curve! NF4 crafts bucket borders that perfectly mirror the bell curve, keeping maximum detail where most numbers live.
- $$q_i = \frac{1}{2}\left(Q_X\left(\frac{i}{2^k}\right) + Q_X\left(\frac{i+1}{2^k}\right)\right)$$
- Learning goalExplain the three core mathematical innovations of QLoRA: NF4, Double Quantization, and Paged Optimizers.
- Helpful foundationsLoRA across attention projections · Floating point and integer quantization basics
- Common trapBelieving 4-bit quantization damages gradient calculations; base weights are quantized in 4-bit, but dequantized on the fly to FP16/BF16 during matrix multiplication.
- Next connectionCompare base foundation models across capacity and resource boundaries in Screen 3.
More context & mechanics
QLoRA achieves 16-bit finetuning accuracy over a 4-bit quantized base model via three key pillars:
- NF4 (NormalFloat 4) Quantization: Pretrained neural network weights follow a zero-mean normal distribution N(0, σ²). NF4 builds 16 discrete quantile bins with equal area under the standard normal distribution curve, minimizing information-theoretic quantization error.
- Double Quantization (DQ): Quantization maps blocks of 64 weights to a single 32-bit floating-point scaling constant c₁. DQ quantifies these constants themselves into 8-bit integers (block size 256), reducing memory overhead from 0.5 bits/param to 0.127 bits/param.
- Paged Optimizers: Uses CUDA Unified Memory to automatically page optimizer state tensors between GPU VRAM and CPU RAM during gradient checkpointing spikes, preventing OOM crashes.
Consider storing a 7-billion parameter base model:
- FP16 Storage (16-bit):
7 × 10⁹ × 2 bytes = 14.0 GB
- Standard 4-bit (with 0.5 bit/param FP32 scaling constants):
7 × 10⁹ × (4 + 0.5) bits = (7 × 10⁹ × 4.5) / (8) ≈ 3.94 GB
- QLoRA NF4 with Double Quantization (4.127 bits/param):
Memory = (7 × 10⁹ × 4.127) / (8 × 10⁹) ≈ 3.61 GB
A 7B parameter foundation model fits into less than 4 GB of VRAM, allowing instruction tuning on a standard consumer laptop GPU!
Standard backbone of libraries like bitsandbytes, HuggingFace TRL, and Axolotl.
Tim Dettmers et al. (University of Washington, 2023) published QLoRA, enabling democratized LLM fine-tuning worldwide.
NF4 shapes buckets to the bell; double quant compresses the scale; paged memory cures the spikes.
How do GPT-2 Small (124M) and GPT-2 Medium (355M) compare across memory footprints, training throughput, and downstream instruction capability?
Choosing a base model is like choosing between a nimble sports coupe and a heavy delivery truck. The sports coupe (GPT-2 124M) is ultra-fast, fits in any garage, and tests ideas in minutes. The truck (355M) carries triple the cargo and handles rough terrain, but requires more fuel and wider roads.
- $$\text{FLOPs}_{\text{step}} \approx 6 \cdot \Phi \cdot B \cdot T$$
- Learning goalSelect appropriate base model capacity by evaluating memory, compute throughput, and reasoning requirements.
- Helpful foundationsGPT-2 architecture specifications · FLOP calculation basics
- Common trapAssuming larger models always require more training data; larger models often learn downstream instruction styles in fewer training epochs than small models.
- Next connectionSynthesize full finetuning, LoRA, and QLoRA in the master tradeoff matrix in Screen 4.
More context & mechanics
Architectural comparison between foundation tiers:
| Metric | GPT-2 Small (124M) | GPT-2 Medium (355M) | Factor |
| :--- | :--- | :--- | :--- |
| Layers (L) | 12 | 24 | 2.0× |
| Hidden Dim (d) | 768 | 1024 | 1.33× |
| Attention Heads (h) | 12 | 16 | 1.33× |
| Parameters | 124.4× 10⁶ | 354.8× 10⁶ | 2.85× |
| Base Weight VRAM (FP32) | ≈ 498 MB | ≈ 1,420 MB | 2.85× |
| Base Weight VRAM (FP16) | ≈ 249 MB | ≈ 710 MB | 2.85× |
Instruction Performance Difference:
The 124M model handles direct classification and short template formatting reliably, but frequently struggles with multi-step reasoning. The 355M model reliably maintains coherent chain-of-thought structures across 200+ generated tokens.
Running instruction tuning on an 8GB GPU with batch size B=4 and sequence length T=512:
- GPT-2 Small (124M):
- Forward + Backward FLOPs per step ≈ 6 × Φ × B × T = 6 × (124× 10⁶) × 4 × 512 ≈ 1.52 TFLOPs.
- Throughput: 42 batches/sec. Training epoch takes 1.8 minutes.
- GPT-2 Medium (355M):
- FLOPs per step ≈ 6 × (355× 10⁶) × 4 × 512 ≈ 4.36 TFLOPs.
- Throughput: 15 batches/sec. Training epoch takes 5.1 minutes.
The 355M model requires 2.8× more compute time, but achieves a 34\% lower perplexity on complex instructions.
Prototyping on 124M before scaling final production fine-tunes to 355M, 7B, or 8B parameter variants.
OpenAI staged the release of GPT-2 sizes throughout 2019 (124M, 355M, 774M, 1.5B) to study scaling dynamics.
Prototype small to verify the code; scale to medium to polish the mind: compute trades time for depth.
How do engineering teams systematically choose between full-parameter fine-tuning, LoRA, and QLoRA for production systems?
Think of traveling: full fine-tuning is moving your entire house in moving vans; LoRA is packing a lightweight custom suitcase; QLoRA is vacuum-sealing the suitcase into a carry-on backpack. If you have infinite budget, move the house. If you want speed and versatility, pack the suitcase.
- $$\text{Storage}_{\text{MultiTenant}} = S_{\text{base}} + N \cdot S_{\text{adapter}} \ll N \cdot S_{\text{base}}$$
- Learning goalEvaluate engineering tradeoffs between full fine-tuning, LoRA, and QLoRA to select the optimal strategy for a given system constraint.
- Helpful foundationsLoRA architecture · QLoRA 4-bit NormalFloat mechanics
- Common trapBelieving QLoRA is always strictly better than LoRA; QLoRA involves an approximate 25% training throughput speed penalty due to on-the-fly dequantization kernels.
- Next connectionImplement the complete instruction finetuning training loop in Chapter 41.
More context & mechanics
The decision matrix unifies the three tuning methodologies:
| Attribute | Full Fine-Tuning | LoRA | QLoRA |
| :--- | :--- | :--- | :--- |
| Trainable Params | 100\% | 0.1\% - 1.0\% | 0.1\% - 1.0\% |
| Base Model Precision | 16-bit / 32-bit | 16-bit / 32-bit (Frozen) | 4-bit NF4 (Frozen) |
| Training VRAM (7B) | > 80 GB (A100) | ≈ 16 - 24 GB | ≈ 6 - 8 GB |
| Relative Training Speed | 1.0× (Fastest) | 0.9× - 0.95× | 0.65× - 0.75× (Quant dequant cost) |
| Checkpoint Storage | Full model (14 GB) | Adapter only (20 MB) | Adapter only (20 MB) |
| Serving Overhead | Zero | Zero (Merged) or Minor (Dynamic) | Minor (4-bit dequantization) |
| Multi-Tenant Serving | Impossible on 1 GPU | Elite (Swap adapters on the fly)| Elite (Swap adapters on the fly)|
A SaaS platform serving 50 specialized domain fine-tunes:
- Full Fine-Tuning Approach:
- Storage: 50 × 14 GB = 700 GB of checkpoints.
- Serving: Needs 50 separate GPU instances arrow \40,000/month cloud cost.
- LoRA Approach (Shared Base Model):
- Storage: 14 GB (base) + 50 × 20 MB = 15 GB total storage!
- Serving: 1 single A100 GPU hosting the base model, dynamically routing requests through 50 cached lightweight adapters arrow \1,500/month.
The LoRA architectural choice yields a 26× cost reduction with zero quality loss.
The foundational architectural blueprint adopted by enterprise LLM infrastructure platforms (AnyScale, Together AI, Databricks).
Consolidated across 2021–2024 as the standard taxonomy of parameter-efficient adaptation.
Full for maximum raw throughput; LoRA for flexible adapters; QLoRA for extreme budget VRAM: match the tool to the hardware.