Model Compression: Knowledge Distillation, Pruning & 4-Bit Quantization

Knowledge distillation minimizes a combined loss consisting of hard cross-entropy with ground truth and soft Kullback-Leibler (KL) divergence with softened teacher probabilities:

L_(KD) = (1 - α) L_(CE)(y, p_(student)) + α T² · D_(KL)(p_(teacher)^(τ) ∥ p_(student)^(τ))

where softened probabilities are computed at distillation temperature τ > 1:

pᵢ^(τ) = (exp(zᵢ / τ)) / (∑ⱼ exp(zⱼ / τ))

The τ² scaling factor ensures gradient magnitudes remain balanced as temperature scales.

Used to create world-class Small Language Models: Microsoft Phi-3, DistilBERT, Apple MobileLLM, and Gemma-2 (distilled from Gemma-27B).

Introduced by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean (2015) in 'Distilling the Knowledge in a Neural Network', formalizing Rich Caruana's earlier model compression concepts (2006).

Dark knowledge lives in the non-zero probabilities of incorrect classes: soften targets to transfer deep intuition.

The combined compression ratio is multiplicative:

Compression Ratio = (N_(teacher)) / (N_(student)) × (Bits_(FP16)) / (Bits_(INT4)) = (14B/2B) × (16/4) = 7 × 4 = 28×

A 28 GB teacher model collapses into a 1 GB mobile artifact that fits inside iPhone unified memory.

The standard deployment path for edge-first intelligence (e.g. Apple Intelligence, Google Nano, Microsoft Phi-Silica).

Synthesized between 2022 and 2024 as researchers realized that smaller distilled models quantize with significantly less degradation than raw unpruned models.

Distill to shrink the architecture; quantize to shrink the precision: double compression makes edge AI possible.

1. Unstructured Pruning: Sets individual weights with magnitude |W_(ij)| < θ to 0. Yields sparse matrices requiring specialized sparse BLAS libraries with high index overhead.
2. Structured Pruning: Removes entire attention heads, channels, or MLP layers:

W_(pruned) ∈ R^d_(in) × d_(reduced)

This produces smaller, dense matrices that execute with 100% hardware efficiency on standard mobile NPUs.

Standard technique in mobile NPU deployment where hardware neural accelerators lack sparse matrix multiply units.

Pioneered by Song Han et al. (2015) in 'Deep Compression' and refined by Michel et al. (2019) showing that many attention heads in transformers can be pruned with zero loss in performance.

Unstructured saves parameters on paper; structured saves latency on hardware.

Uniform quantization transforms continuous neural network weights and activations into low-precision discrete integers:

1. The Floating-Point Footprint Challenge:

Standard single-precision floating-point numbers (FP32) consume 4 bytes (32 bits: 1 sign bit, 8 exponent bits, 23 mantissa bits). Storing a 7-billion parameter model in FP32 requires:

7 × 10⁹ × 4 bytes = 28 GB of memory

Edge and mobile hardware rarely possess sufficient VRAM to load 28 GB weights.

1. The Discrete Integer Mapping:

INT8 quantization maps the continuous dynamic range [x_(min), x_(max)] onto 256 discrete integer values:

Unsigned INT8: [q_(min), q_(max)] = [0, 255]

Signed INT8: [q_(min), q_(max)] = [-128, 127]

1. The Uniform Linear Grid:

The continuous range is partitioned into 2⁸ - 1 = 255 equal-width segments (steps). Any real value falling within a segment is snapped to the nearest discrete integer bucket.

1. Compression Payoffs:

• 4× Storage Reduction: Shrinks a 7B model from 28 GB down to 7 GB.
• Faster Math: INT8 integer matrix multiplication (DP4A / Tensor Core GEMM) delivers up to 2× to 4× higher throughput than FP32 floating-point units.

Fundamental compression protocol for edge deployment frameworks including ONNX Runtime, TensorRT, llama.cpp, and Android NNAPI.

Uniform quantization originated in digital signal processing (Pulse Code Modulation by Alec Reeves in 1937). Its application to deep learning was pioneered in the mid-2010s (e.g., Jacob et al., 2018 at Google) to make neural networks run on battery-powered mobile phones.

• From continuous spectrum to 256 notches: 32 bits of float become 8 bits of integer.
• Divide by scale, round to bucket: The two-step compression pipeline.

#### 1. The Affine Quantization Equations

Affine (asymmetric) uniform quantization maps real x ∈ [x_(min), x_(max)] to integer q ∈ [q_(min), q_(max)]:

1. The Scale Factor (S):

Determines the step size of each discrete bucket:

S = (x_(max) - x_(min)) / (q_(max) - q_(min))

Where S is stored as a high-precision floating-point scalar (FP32).

1. The Zero-Point (Z):

Ensures that real x = 0.0 maps exactly to an integer without rounding error:

Z = round(-(x_(min)) / (S)) + q_(min)

Where Z ∈ [q_(min), q_(max)] is an integer of the same type as q.

1. The Quantization Operator:

q = clamp( round( x/S ) + Z, q_(min), q_(max) )

Where clamp(v, a, b) = min(max(v, a), b).

1. The Dequantization Operator:

Reconstructs the real-valued approximation x̂ ≈ x:

x̂ = S · (q - Z)

Consider an activation tensor with dynamic range [x_(min), x_(max)] = [-2.0, +6.0] quantized to unsigned INT8 [0, 255]:

• Scale:

S = (6.0 - (-2.0)) / (255 - 0) = (8.0) / (255) ≈ 0.0313725

• Zero-Point:

Z = round(-(-2.0) / (0.0313725)) + 0 = round(63.75) = 64

• Quantizing x = 0.0:

q = round(0 / S) + 64 = 64 (Exact integer alignment!)

• Quantizing x = 3.5:

q = round(3.5 / 0.0313725) + 64 = round(111.56) + 64 = 112 + 64 = 176

• Dequantizing q = 176:

x̂ = 0.0313725 × (176 - 64) = 0.0313725 × 112 = 3.51372

Quantization error: |x - x̂| = |3.5 - 3.51372| = 0.01372, strictly bounded by S / 2 = 0.01568.

This exact scale and zero-point formulation is the core standard for PyTorch quantization (`torch.ao.quantization`), TensorFlow Lite, and ONNX Runtime QDQ (Quantize-Dequantize) nodes.

Benoit Jacob and Google colleagues formalized this integer-arithmetic-only quantization scheme in their 2018 CVPR paper, Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, unlocking real-time inference on mobile CPUs.

• S is range over levels; Z anchors real zero: S = Δ x / 255, Z = -x_(min) / S.
• Quantize: round and shift; Dequantize: subtract and scale: q = x/S + Z, x̂ = S(q - Z).

Quantizing modern Large Language Models presents a unique structural breakdown:

1. The Emergent Outlier Crisis (LLMs > 6.7B):

When transformer language models scale beyond 6.7B parameters, their internal representations undergo an emergent phase transition:

• Approximately 0.1\% of hidden dimensions develop extreme activation spikes with magnitudes up to 100× larger than normal channels (e.g., normal activations ∈ [-3, +3], but outlier channels reach ± 250).
• Crucially, these outliers are concentrated in a few specific channels across all tokens.

1. Why Standard Quantization Fails:

When quantizing activations along the token dimension:

S_X = (max(|X|)) / (127) = 250/127 ≈ 1.97

Normal channels with values around 0.2 are completely zeroed out by rounding:

round(0.2 / 1.97) = 0

This causes catastrophic accuracy collapse, causing the language model to output repetitive gibberish.

1. The SmoothQuant Insight (Weight-Activation Equivalence):

Weights are much easier to quantize than activations because weight distributions are static, Gaussian-like, and lack extreme spikes.

SmoothQuant applies a per-channel smoothing scale factor s ∈ R^C:

• Divide activation channels by s: X̂ = X · diag(s)^(-1)
• Multiply corresponding weight rows by s: Ŵ = diag(s) · W

This flattens the activation peaks, squeezing them into an easily quantifiable range while static weights absorb the variance.

SmoothQuant is integrated into NVIDIA TensorRT-LLM, vLLM, and Intel Neural Compressor to enable W8A8 (8-bit weight, 8-bit activation) integer GEMM execution on production serving clusters.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao-Shu Fang, Song Han (MIT) introduced SmoothQuant in their 2022 landmark paper, SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, unlocking INT8 matrix multiplication without accuracy loss.

• Smooth the activations, absorb into the weights: Shrink the dynamic spikes before rounding.
• Divide activations, multiply weights: An invariant mathematical trade that keeps the output identical.

#### 1. The Invariant Linear Map Proof

Consider a standard linear layer in a transformer (such as a projection in attention or MLP):

Y = X W

Where X ∈ R^(T × C) is the activation matrix and W ∈ R^(C × D) is the weight matrix.

Let s ∈ R^C be a positive per-channel scaling vector (sⱼ > 0 for all j ∈ [1, C]), and define the diagonal scaling matrix:

diag(s) = s₁ & 0 & ṡ & 0 \ 0 & s₂ & ṡ & 0 \ … & … & ⋱ & … \ 0 & 0 & ṡ & s_C

Since all sⱼ > 0, diag(s) is invertible, and:

diag(s)^(-1) diag(s) = I

Inserting this identity matrix between X and W:

Y = X · I · W = X · (diag(s)^(-1) diag(s)) · W

By associativity of matrix multiplication:

Y = ( X diag(s)^(-1) ) ( diag(s) W ) = X̂ Ŵ

Where:

• X̂ = X diag(s)^(-1) is the smoothed activation tensor.
• Ŵ = diag(s) W is the pre-scaled weight matrix.

The continuous mathematical output Y is strictly identical.

To balance quantization difficulty equally between activations and weights, the per-channel scaling factor sⱼ is computed as:

sⱼ = (max_(k) |X_(k, j)|^α) / (max_(l) |W_(j, l)|^(1 - α))

Where:

• max_(k) |X_(k, j)|: Maximum absolute activation value in channel j.
• max_(l) |W_(j, l)|: Maximum absolute weight value in channel j.
• α ∈ [0, 1]: The migration strength hyperparameter.

• α = 1.0: Migrates all difficulty to weights (activations become completely uniform; weights become harder to quantize).
• α = 0.0: No difficulty migration (weights remain untouched; activations retain severe outliers).
• α = 0.5 (Optimal Compromise): Splits the dynamic range burden equally between activations and weights, allowing both X̂ and Ŵ to be quantized to INT8 with minimal precision loss.

Because weights W and the scaling vector s are static after pre-training, the transformation Ŵ = diag(s) W is computed offline once. During runtime, the model merely divides incoming activations by s before executing high-speed INT8 integer GEMM.

Deployed universally in enterprise inference engines (TensorRT-LLM, FasterTransformer) to serve 70B+ parameter models on half the number of GPUs while maintaining full FP16 perplexity.

Introduced by MIT's HAN Lab in late 2022, SmoothQuant solved the long-standing open question of how to execute INT8 matrix multiplication on both weights and activations simultaneously without fine-tuning or retraining.

• Y = (X s^(-1))(s W): The identity sandwich that saves INT8 inference.
• α = 0.5 balances the scales: Equalize the dynamic range of activations and weights.

1. GPTQ (Second-Order Error Minimization): Uses Hessian matrix H = 2 X^T X to update unquantized weights as each column is quantized, minimizing reconstruction error:

w_(rem) arrow w_(rem) - (w_q - ŵ_q) / ([H^(-1)]_(qq)) H^(-1)_(:, q)

1. AWQ (Activation-Aware Weight Quantization): Protects salient weights by observing activation magnitudes, scaling salient channels by s > 1 before round-to-nearest (RTN) quantization:

W' = round(W · s) · 1/s

The two dominant 4-bit weight formats in production: GPTQ for ultra-fast GPU matrix kernels, and AWQ for superior perplexity preservation.

GPTQ was developed by Frantar et al. (2022) based on Optimal Brain Surgeon; AWQ was created by Ji Lin et al. (MIT HAN Lab, 2023).

Protect the salient 1%: activation awareness allows 4-bit weights to retain 99% of 16-bit intelligence.

The GGUF (GPT-Generated Unified Format) binary format serializes:

1. Magic number (`GGUF` in ASCII)
2. Comprehensive metadata key-value pairs (architecture, context length, tokenizer vocab, rope frequencies)
3. Tensor information array (names, dimensions, quantization type: `Q4_K_M`, `Q5_K_S`)
4. Raw 32-byte aligned tensor buffers ready for instant `mmap` zero-copy memory mapping.

The universal standard for local AI inference across millions of developers using `llama.cpp`, Ollama, LM Studio, and Apple MLX.

Created by Georgi Gerganov and the open-source community in August 2023, succeeding GGML and GGJT to provide extensible metadata and backward compatibility.

Single file, zero dependencies, universal binary: GGUF brings artificial intelligence to consumer hardware.

Under the IEEE 754 standard, a real value V is represented as:

V = (-1)^s × ( 1 + ∑_(i=1)^M b_(-i) 2^(-i) ) × 2^(e - bias)

Comparing common precision formats in AI:

• FP32 (Standard Single Precision):
• Total: 32 bits (4 bytes)
• 1 sign bit, 8 exponent bits (bias 127), 23 mantissa bits
• Dynamic range: ≈ 10^(-38) to 10^(38), ≈ 7.2 decimal digits of precision
• FP16 (Half Precision):
• Total: 16 bits (2 bytes)
• 1 sign bit, 5 exponent bits (bias 15), 10 mantissa bits
• Dynamic range: 6 × 10^(-5) to 65504 (prone to overflow!)
• BF16 (Brain Floating Point - Google Brain):
• Total: 16 bits (2 bytes)
• 1 sign bit, 8 exponent bits (bias 127), 7 mantissa bits
• Same dynamic range as FP32, preventing training instability!

Modern LLM training runs almost exclusively in BF16 (bfloat16) on NVIDIA H100 and Google TPU v5 hardware, saving 50\% GPU VRAM and doubling tensor compute throughput over FP32.

William Kahan led the drafting of IEEE 754 in 1985, for which he won the Turing Award. In 2018, Google Brain designed BF16 specifically for deep learning accelerators.

• Sign decides direction; exponent decides scale; mantissa decides precision.
• BF16 preserves FP32's dynamic range with half the bits.

Affine quantization maps real continuous values x ∈ [α, β] to discrete integers q ∈ [q_(min), q_(max)] (for INT8, [-128, 127]):

q = clip( ⌊ x/S ⌉ + Z, q_(min), q_(max) )

Where:

• Scale factor (S ∈ R^+):

S = (β - α) / (q_(max) - q_(min))

• Zero-point (Z ∈ Z):

Z = round( (-α) / (S) ) + q_(min)

To dequantize back to floating-point:

x̂ = S · (q - Z)

Symmetric quantization forces α = -β, ensuring Z = 0 and eliminating integer subtraction overhead during matrix multiplication.

Quantization libraries like `bitsandbytes`, `llama.cpp`, and AWQ use this affine mapping to compress 70-billion parameter models from 140 GB of VRAM down to 35 GB (INT4/INT8), allowing frontier models to run on MacBook laptops and local edge chips.

Benoît Jacob et al. at Google published 'Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference' in 2018, establishing modern INT8 deployment standards.

• Scale stretches the ruler; zero-point slides the origin.
• Float = Scale * (Int - ZeroPoint).

Given teacher logits z^T and student logits z^S:

1. Compute softened probability distributions at temperature τ > 1:

pᵢ^T = (e^(zᵢ^T / τ)) / (∑ₖ e^(zₖ^T / τ)), pᵢ^S = (e^(zᵢ^S / τ)) / (∑ₖ e^(zₖ^S / τ))

1. The distillation loss is the Kullback-Leibler (KL) divergence between softened distributions:

L_(KD) = τ² · D_(KL)(p^T ∥ p^S) = τ² ∑_(i=1)^K pᵢ^T log ( (pᵢ^T) / (pᵢ^S) )

The τ² scaling factor is mathematically required because dividing logits by τ scales the gradient magnitudes by 1 / τ²; multiplying by τ² ensures the distillation gradient remains balanced with the hard-label cross-entropy loss L_(CE):

L_(total) = α L_(CE) + (1 - α) L_(KD)

Knowledge distillation is used to create efficient edge models: distilling Whisper-Large into Whisper-Tiny, or distilling 70B frontier LLMs into 8B and 1B SLMs (e.g. TinyLlama, Gemma-2-2B, Phi-3).

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean formalized 'Distilling the Knowledge in a Neural Network' at Google in 2015, introducing the term 'dark knowledge' to describe non-target class probabilities.

• Soften the teacher's logits to reveal the dark knowledge.
• Multiply the KL loss by tau² to keep the gradient alive.

For two discrete probability distributions p, q ∈ Δ^(K-1):

D_(KL)(p ∥ q) = ∑_(i=1)^K pᵢ ln( pᵢ/qᵢ ) = ∑_(i=1)^K pᵢ ln pᵢ - ∑_(i=1)^K pᵢ ln qᵢ

Fundamental mathematical properties:

1. Non-negativity (Gibbs' Inequality):

D_(KL)(p ∥ q) ≥ 0, ∀ p, q

With D_(KL)(p ∥ q) = 0 ⟺ p = q.

Proof via Jensen's inequality using strictly convex function f(t) = -ln(t):

D_(KL)(p ∥ q) = ∑_(i=1)^K pᵢ (-\lnqᵢ/pᵢ) ≥ -ln(∑_(i=1)^K pᵢ qᵢ/pᵢ) = -ln(∑_(i=1)^K qᵢ) = -ln(1) = 0

1. Asymmetry: In general, D_(KL)(p ∥ q) ≠ D_(KL)(q ∥ p). Therefore, KL divergence is a pre-metric (divergence), not a true Euclidean distance metric.

KL divergence is used in Reinforcement Learning from Human Feedback (RLHF / PPO) to penalize the aligned policy from drifting too far from the original base model: r_(total) = r_(reward) - β D_(KL)(π_θ ∥ π_(ref)).

Solomon Kullback and Richard Leibler published this relative directed divergence in 1951, providing the foundational measure for hypothesis testing and minimum discrimination information.

• Expected log ratio of P over Q.
• Always non-negative; zero only when P equals Q.

Let P be a multi-modal target distribution and Q_θ a parameterized unimodal model.

1. Forward KL (Zero-Avoiding / Mode-Covering):

D_(KL)(P ∥ Q_θ) = ∫ P(x) log( (P(x)) / (Q_θ(x)) ) dx

• If P(x) > 0 and Q_θ(x) → 0, the ratio (P(x)) / (Q_θ(x)) → ∞, causing infinite loss penalty!
• Therefore, Q_θ(x) MUST be non-zero everywhere P(x) has mass.
• Behavior: Q_θ spreads wide to cover all modes, allocating high probability to the empty low-density valley between peaks.

1. Reverse KL (Zero-Forcing / Mode-Seeking):

D_(KL)(Q_θ ∥ P) = ∫ Q_θ(x) log( (Q_θ(x)) / (P(x)) ) dx

• If P(x) = 0, Q_θ(x) MUST equal 0; otherwise, the ratio (Q_θ(x)) / (P(x)) = ∞.
• But if P(x) > 0 and Q_θ(x) = 0, the integrand evaluates to 0 log(0) = 0 (zero penalty!).
• Behavior: Q_θ happily drops entire modes of P, locking tightly onto the single mode with the highest density.

• Forward KL (D_(KL)(P_(teacher) ∥ Q_(student))): Maximum Likelihood Pretraining and standard Hinton Knowledge Distillation: the model is zero-avoiding, ensuring the student covers all modes of the teacher.
• Reverse KL (D_(KL)(Q_(student) ∥ P_(teacher))): Variational Inference (VAEs) and Generative Distillation (e.g. MiniLLM): the student is zero-forcing and focuses on capturing sharp, high-confidence modes without hallucinating.

Tom Minka formalized the divergence dichotomy in his 2005 paper 'Divergence measures and message passing', clarifying how Expectation Propagation (Forward KL) differs from Variational Bayes (Reverse KL).

• Forward KL covers all modes (fears missing out).
• Reverse KL picks one mode (fears being wrong).

The Lottery Ticket Hypothesis (Frankle & Carbin, 2018):

Let f(x; w₀) be a dense neural network initialized with w₀ ~ D.

When trained for T steps, it reaches test accuracy A.

There exists a binary pruning mask m ∈ 0, 1^P with sparsity \|m\|₀ ≪ P such that the sparse subnetwork f(x; m \odot w₀), when trained from initial weights w₀, achieves test accuracy A' ≥ A in fewer or equal iterations.

Iterative Magnitude Pruning (IMP) with Rewinding:

1. Initialize dense weights w₀. Train to step k (often k = 0 or early step k = 1000).
2. Train dense network to convergence step T, obtaining weights w_T.
3. Prune lowest magnitude weights: mᵢ = 0 if |w_(T, i)| < τ.
4. Rewind surviving weights back to their step-k initialization:

w_(rewound) = m \odot wₖ

1. Retrain the sparse subnetwork directly.

This principle guides model compression, weight pruning, and hardware sparse accelerators (e.g. NVIDIA Ampere 2:4 structured sparsity).

Jonathan Frankle and Michael Carbin won the ICLR 2019 Best Paper Award for 'The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks'.

• Dense networks contain winning tickets.
• Prune by magnitude; rewind to initialization; train in isolation.

In large language models (>6.7B parameters), systematic activation outliers emerge in a few specific channels (x_(c) > 100), making activation quantization (W8A8) fail.

Weights are easy to quantize; activations are hard.

SmoothQuant introduces a diagonal per-channel migration matrix S = diag(s₁, ṡ, s_d):

Y = X W = (X S^(-1)) (S W) = X̂ Ŵ

Where:

• Scaled activations: X̂ = X S^(-1) (channel j divided by sⱼ).
• Scaled weights: Ŵ = S W (row j multiplied by sⱼ).

The optimal per-channel migration scale sⱼ balances activation and weight dynamic ranges:

sⱼ = (max(|X_(·, j)|)^α) / (max(|W_(j, ·)|)^(1 - α))

Where migration strength α ∈ [0, 1] (typically α = 0.5 balances the difficulty equally).

Both X̂ and Ŵ are now smooth and can be quantized directly using standard INT8 GEMM kernels.

SmoothQuant powers modern W8A8 inference runtimes in NVIDIA TensorRT-LLM and vLLM, doubling serving throughput on Hopper and Ada Lovelace GPUs.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao-Shu Fang, Song Han, and Song Han published SmoothQuant at ICML 2023 ('SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models').

• Divide activations by S; multiply weights by S.
• S cancels out mathematically while smoothing the outliers.

A signal processing quantization diagram on light ivory canvas (#fbfaf3) with modern rounded borders.

Top section: Title in modern sans-serif: "Asymmetric (Affine) Quantization: The Zero-Point Map". Subtitle: "Mapping arbitrary real intervals [x_min, x_max] onto unsigned integer range [0, 255]."

Middle section: Dual axis calibration diagram.

Top Axis (Real Float32): Number line spanning [-2.0, +6.0], with true real zero x = 0.0 highlighted.

Bottom Axis (Quantized UINT8): Integer line spanning [0, 255], with zero-point integer Z = 64 highlighted.

Show dashed vertical projection lines mapping -2.0 to 0, 0.0 to 64, and +6.0 to 255.

Formulas: Scale S = (x_(max) - x_(min)) / (q_(max) - q_(min)) = (8.0) / (255) ≈ 0.03137; Zero-point Z = round(-x_(min)/S) = 64.

Bottom section: Formula card: "q = clamp(round(x/S) + Z, 0, 255); \ x̂ = S · (q - Z) ⟹ Preserves exact real zero to prevent padding bias."

Clean axis ticks, generous spacing, high legibility.

This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.

"Asymmetric (Affine) Quantization: The Zero-Point Map: Invariance and structure determine stability under spatial and numerical transformations."

A hardware acceleration mathematical diagram on light ivory canvas (#fbfaf3) with 12px rounded borders.

Top section: Title in modern sans-serif: "Symmetric INT8 Quantization: Eliminating GEMM Overhead". Subtitle: "Forcing zero-point Z = 0 eliminates costly cross-terms on NPU Tensor Cores."

Middle section: An algebraic cross-term expansion diagram:

General Quantized Matrix Multiply:

X̂ Ŵ = [S_X (q_X - Z_X)] [S_W (q_W - Z_W)] = S_X S_W [q_X q_W - Z_W q_X - Z_X q_W + Z_X Z_W (Zero-point runtime cross-terms)].

Symmetric Weight Quantization (Z_W = 0):

Scale S_W = (||W||_∞) / (127). Since Z_W = 0, the cross-terms collapse to a single vector subtraction:

X̂ Ŵ = S_X S_W [q_X q_W - Z_X ∑ q_W].

Bottom section: Card: "Symmetric weight quantization allows hardware integer matrix multiply units (INT8 Tensor Cores) to execute at full speed without runtime dequantization overhead."

Clear algebraic brackets, teal hardware callouts, spacious layout.

This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.

"Symmetric INT8 Quantization: Eliminating GEMM Overhead: Invariance and structure determine stability under spatial and numerical transformations."

A modern quantization architecture diagram on light ivory canvas (#fbfaf3) with rounded card borders.

Top section: Title in modern sans-serif: "INT4 Grouping & NormalFloat4 (NF4) Quantiles". Subtitle: "Information-theoretically optimal 4-bit representation for normally distributed weights."

Middle section: Two visual concept containers side-by-side.

Left Container: "Group-Wise Scaling" - A weight matrix sliced into small blocks of size B = 64 or 128. Each block has its own independent float16 scale factor S_g = (max |W_g|) / (7), containing outlier corruption to a single block.

Right Container: "NormalFloat4 (NF4) Quantile Map" - A standard normal bell curve N(0, 1) partitioned into 16 equal-probability bins. The 16 discrete integer codes represent the exact theoretical quantiles of the Gaussian distribution.

Bottom section: Memory packing diagram: Two 4-bit nibbles packed into a single uint8 byte: byte = (q₁ \ \& \ 0x0F) | ((q₂ \ \& \ 0x0F) ≪ 4) ⟹ 4x memory compression.

Crisp vector curves, generous whitespace, pristine lines.

This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.

"INT4 Grouping & NormalFloat4 (NF4) Quantiles: Invariance and structure determine stability under spatial and numerical transformations."

A formal mathematical derivation card on light ivory canvas (#fbfaf3) with 12px rounded borders.

Top section: Title in modern sans-serif: "Knowledge Distillation: Taylor Proof of T² Scaling". Subtitle: "Why multiplying the KL divergence loss by T² is mathematically mandatory."

Middle section: A step-by-step boxed proof:

1. Softmax with Temperature T: pᵢ = (e^(zᵢ / T)) / (∑ⱼ e^(zⱼ / T)).
2. Loss Gradient: (∂ L_(KL)) / (∂ zᵢ^S) = 1/T (pᵢ^S - pᵢ^T).
3. Taylor Series Expansion for large T: e^(z/T) ≈ 1 + z/T ⟹ pᵢ ≈ (1 + zᵢ / T) / (N + ∑ zⱼ / T) ≈ 1/N + (1) / (N T)(zᵢ - z̄).
4. Subtract student and teacher probabilities:

(pᵢ^S - pᵢ^T) ≈ (1) / (N T)(zᵢ^S - zᵢ^T).

1. Substitute into gradient: (∂ L_(KL)) / (∂ zᵢ^S) ≈ (1) / (N T²)(zᵢ^S - zᵢ^T).

Bottom section: Card: "Gradients shrink by 1/T² as temperature softens probabilities; multiplying loss by T² cancels this shrinkage, preserving gradient norm balance with hard-label loss."

Clear algebraic layout, ample breathing space, authoritative typography.

This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.

"Knowledge Distillation: Taylor Proof of T² Scaling: Invariance and structure determine stability under spatial and numerical transformations."

A precision quantization number-line diagram in square 1:1 format on dark slate (#0f172a) with clean borders.

Top section: Title in white sans-serif: "Asymmetric (Affine) Quantization: The Zero-Point Map". Subtitle in muted teal: "Mapping arbitrary real values [x_min, x_max] to unsigned 8-bit integers [0, 255]."

Middle section: Two parallel aligned horizontal scales:

Top Scale (Real Numbers R): Continuous range from x_(min) = -1.2 to x_(max) = 3.8. Highlight the real value 0.0 with a vertical marker.

Bottom Scale (Quantized Integers uint8): Discrete integer grid from q = 0 to q = 255.

Show the mapping equations:

"S = (x_(max) - x_(min)) / (q_(max) - q_(min)) = (3.8 - (-1.2)) / (255 - 0) = 0.0196 (Scale)"

"Z = round(-(x_(min)) / (S)) = round(-(-1.2) / (0.0196)) = 61 (Zero-Point)".

Show that real 0.0 maps exactly to integer 61, ensuring that zero-padding in activations does not incur quantization error.

Bottom section: Dequantization formula: "x ≈ S × (q - Z). Affine quantization minimizes clipping error on asymmetric distributions."

Crisp number line rulers, exact numeric callouts, balanced square layout.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"Affine (Asymmetric) Quantization: The Zero-Point Shift: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A computational efficiency diagram in square 1:1 format on dark slate background (#0f172a) with modular cards.

Top section: Title in white sans-serif: "Symmetric INT8 Quantization: Eliminating GEMM Overhead". Subtitle in muted cyan: "Forcing zero-point Z = 0 transforms matrix multiplication into pure integer dot products."

Middle section: Comparative computation flow:

Left (Asymmetric Pain): Matrix product requires tracking cross-terms: (q_w - Z_w)(q_x - Z_x) = q_w q_x - Z_x q_w - Z_w q_x + Z_w Z_x, requiring costly accumulator corrections.

Right (Symmetric Glory): Forcing symmetric range [-x_(max), x_(max)] sets Z = 0.

The product simplifies to:

"W · x ≈ (S_w S_x) × (q_w · q_x)".

Show that integer matrix multiplication (q_w · q_x) executes directly on high-speed hardware tensor cores (NVIDIA INT8 Tensor Cores / Apple NPU) with zero zero-point offset overhead.

Bottom section: Scale formula: "S = (max(|x_(min)|, |x_(max)|)) / (127). Preferred for model weights where distributions are centered at zero."

Clean algebraic comparison, hardware acceleration badges, high legibility.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"Symmetric INT8 Quantization: Eliminating GEMM Overhead: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A statistical quantization distribution diagram in square 1:1 format on dark slate (#0f172a) with clean borders.

Top section: Title in white sans-serif: "INT4 Grouping & NormalFloat4 (NF4) Quantiles". Subtitle in muted cyan: "Information-theoretically optimal 4-bit representation for normally distributed neural weights."

Middle section: Visual comparison of two quantization grids:

Top (Uniform INT4): 16 equally spaced bins on a Gaussian bell curve. Shows severe over-allocation of bins in empty tails and high rounding error near zero.

Bottom (NormalFloat4 / QLoRA): 16 non-linear quantiles chosen such that each bin holds an exact equal probability area (1/16th) under standard normal distribution N(0, 1).

Annotate the 16 exact NF4 quantile levels: [-1.0, -0.696, -0.525, ..., 0.723, 1.0].

Show Groupwise scaling: Weights divided into small blocks of 64 or 128 parameters, each with its own independent scale factor to isolate outliers.

Bottom section: Efficiency card: "QLoRA NF4 reduces model memory footprint by 75% with zero degradation in perplexity."

Beautiful Gaussian curve, exact quantile bin dividers, modern AI systems design.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"INT4 Groupwise Quantization & NormalFloat4 (NF4): Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A mathematical derivation proof card in square 1:1 format on dark slate (#0f172a) with clean modular borders.

Top section: Title in white sans-serif: "Knowledge Distillation: Taylor Proof of T² Scaling". Subtitle in muted gold: "Why the distillation loss must be multiplied by T² to match standard gradient magnitude."

Middle section: Step-by-step Taylor expansion derivation:

1. Distillation loss: L_(KD) = T² · D_(KL)(p_(τ) || q_(τ)), where p_(τ) = Softmax(z_t / T) and q_(τ) = Softmax(z_s / T).
2. At high temperature T, logits are small relative to T. Expand softmax via first-order Taylor series:

e^(zᵢ / T) ≈ 1 + zᵢ/T.

1. Softmax probabilities become: pᵢ ≈ 1/K (1 + (zᵢ - z̄) / (T)).
2. Differentiate loss with respect to student logit z_s:

(∂ L_(KD)) / (∂ z_s) = 1/T (q_(τ) - p_(τ)) ≈ (1) / (T² K) (z_s - z_t).

1. The 1/T² factor completely crushes student gradients at high temperatures!

Bottom section: Golden rule: "Multiplying L_(KD) by T² cancels the 1/T² shrinkage, ensuring student gradients remain at constant scale across all temperatures."

Clear LaTeX derivation lines, prominent conclusion banner, elegant dark styling.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"Knowledge Distillation: The T² Gradient Scaling Proof: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A binary bit-level architecture diagram in square 1:1 format on dark slate (#0f172a) with modular bit strips.

Top section: Title in white sans-serif: "Binary Floating-Point Anatomy: FP32, FP16, and BF16". Subtitle in muted cyan: "Deconstructing Sign, Exponent, and Mantissa bits across deep learning numerical formats."

Middle section: Three horizontal bit-level strips:

1. FP32 (Full Precision - 32 bits):

[1 Sign bit (Grey)] [8 Exponent bits (Magenta)] [23 Mantissa/Fraction bits (Gold)]

Dynamic Range: 10^(± 38). Extremely precise, high memory footprint.

1. FP16 (Half Precision - 16 bits):

[1 Sign bit] [5 Exponent bits] [10 Mantissa bits]

Dynamic Range: 10^(± 5) (Max value = 65,504). Narrow dynamic range prone to underflow/overflow; requires loss scaling.

1. BF16 (Brain Floating-Point - 16 bits):

[1 Sign bit] [8 Exponent bits] [7 Mantissa bits]

Dynamic Range: 10^(± 38) (Identical to FP32!). Trades precision for dynamic range; completely eliminates underflow.

Bottom section: AI training standard: "BF16 has become the universal default for LLM pretraining because its 8-bit exponent prevents gradient underflow without requiring loss scaling."

Sharp bit-cell graphics, color-coded components, pristine monospace numbers.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"Binary Floating-Point Anatomy: FP32, FP16, and BF16: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A matrix sparsity pattern comparison diagram in square 1:1 format on dark slate (#0f172a) with clean matrix grids.

Top section: Title in white sans-serif: "Structured vs. Unstructured Weight Pruning". Subtitle in muted emerald: "Why removing individual weights fails to accelerate hardware, and how structured pruning solves it."

Middle section: Two contrasting 8x8 matrix weight grids:

Left (Unstructured Pruning - Random Sparse Grid):

Individual weights below threshold are set to 0 (scattered green dots on dark background).

Problem: Standard GPU Tensor Cores cannot execute irregular sparse patterns efficiently; requires specialized sparse formats (CSR) and yields zero practical speedup on standard hardware.

Right (Structured Pruning - Block/Channel Removal):

Entire rows, columns, or attention heads are removed completely!

Matrix physically shrinks from 8x8 down to 6x6.

Benefit: Remains a dense matrix of smaller size! Runs immediately on standard GEMM libraries with guaranteed linear hardware speedups and direct VRAM reduction.

Bottom section: Engineering tradeoff: "Unstructured pruning preserves higher perplexity but runs slow; structured pruning permanently deletes entire hardware channels for real inference acceleration."

Clear matrix grid cells, high contrast visual sparsity, balanced square layout.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"Structured vs. Unstructured Weight Pruning: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A mathematical scale migration diagram in square 1:1 format on dark slate (#0f172a) with modular cards.

Top section: Title in white sans-serif: "SmoothQuant: Migrating Outlier Scale to Weights". Subtitle in muted amber: "Xiao et al. (2022): Solving activation outlier clipping by scaling activations and weights inversely."

Middle section: The mathematical migration insight:

Matrix multiplication: Y = X · W.

Introduce diagonal per-channel scaling matrix s ∈ R^D:

"Y = (X · diag(s)^(-1)) · (diag(s) · W) = X̂ · Ŵ".

Visual explanation:

1. Activations X have massive outliers (spikes up to 100x average magnitude in certain feature channels), making INT8 activation quantization fail.
2. Weights W are smooth and easy to quantize.
3. SmoothQuant divides activation channels by scale s (squashing the outlier spikes down!).
4. SmoothQuant multiplies weight channels by s (slightly expanding weights, which easily absorb the scale).
5. Both X̂ and Ŵ are now smooth and can be quantized to INT8 with zero degradation!

Bottom section: Per-channel scale formula: "sⱼ = (max(|X_(:, j)|)^α) / (max(|W_(j, :)|)^(1-α)). Setting α = 0.5 splits quantization difficulty equally."

Clear spike smoothing visualization, elegant formula presentation, modern dark aesthetic.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"Activation Outliers & SmoothQuant: Migrating Quantization Difficulty: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

An edge deployment framework comparison card in square 1:1 format on dark slate (#0f172a) with modular containers.

Top section: Title in white sans-serif: "Edge Deployment Runtimes: GGUF, ONNX, & MLX". Subtitle in muted emerald: "Modern file containers and runtime engines for on-device local execution."

Middle section: Three container framework cards:

1. GGUF (llama.cpp Ecosystem):

Single self-contained binary file containing quantized weights (k-quants: Q4_K_M, Q5_K_M) AND full tokenizer metadata. Direct memory mapping, optimized for CPU and Apple Metal unified memory.

1. ONNX (Open Neural Network Exchange):

Cross-platform compute graph standard. Executes across heterogeneous hardware (NVIDIA TensorRT, Intel OpenVINO, Qualcomm QNN NPUs).

1. MLX (Apple Silicon Native):

Apple's unified memory framework. Arrays live in shared RAM accessible simultaneously by CPU and Metal GPU with zero memory copy overhead.

Bottom section: Production takeaway: "GGUF is standard for edge consumer chatbots; ONNX is standard for industrial cross-hardware pipelines; MLX provides maximum memory bandwidth on Apple Silicon."

Sharp framework badges, distinct hardware targets, clean square formatting.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"Single-File Deployment Architectures: GGUF, ONNX, and MLX: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A neural subnetwork discovery diagram in square 1:1 format on dark slate (#0f172a) with clean algorithmic stages.

Top section: Title in white sans-serif: "The Lottery Ticket Hypothesis: Winning Subnetworks". Subtitle in muted gold: "Frankle & Carbin (2018): Dense networks contain sparse subnetworks that match full accuracy when trained from scratch."

Middle section: The iterative magnitude pruning and rewinding cycle:

1. Initialization: Randomly initialize dense network weights W₀.
2. Train: Train network for T epochs to reach converged weights W_T.
3. Prune: Prune lowest magnitude weights (e.g., lowest 80%), creating binary mask m.
4. The Rewinding Secret: Reset the remaining 20% weights back to their exact original initialization W₀ (NOT random noise!).
5. Retrain: Train the sparse masked network m \odot W₀ from scratch.

Result: The sparse subnetwork matches or exceeds the accuracy of the original dense network in the same number of epochs!

Bottom section: Core insight: "Dense networks are easy to train because they provide many random initialization 'lottery tickets'. Pruning isolates the winning ticket that had the optimal initial geometry."

Clear cyclical flow diagram, highlighted weight rewinding step, pristine square composition.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"The Lottery Ticket Hypothesis: Finding Sparse Subnetworks: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

An information-theoretic distribution comparison diagram in square 1:1 format on warm cream parchment (#fcf0db) with a sepia frame.

Top section: Title in navy serif: "KL Divergence: The Information Distance". Subtitle: "D_(KL)(P || Q) measures the average extra bits needed when coding distribution P using codebook Q."

Middle section: Visual distribution comparison and formula breakdown:

Two continuous probability densities plotted on the same axis:

• Solid Blue Curve: True target distribution P(x).
• Dashed Amber Curve: Model approximation distribution Q(x).

Shade the area between them representing wasted code length.

Continuous Formula:

"D_(KL)(P || Q) = ∫ P(x) log((P(x)) / (Q(x))) dx = E_(x ~ P)[log P(x) - log Q(x)]".

Discrete Formula:

"D_(KL)(P || Q) = ∑ᵢ P(i) log((P(i)) / (Q(i)))".

Gibbs' Inequality:

"D_(KL)(P || Q) ≥ 0 with equality if and only if P(x) = Q(x) almost everywhere."

Bottom section: Asymmetry warning: "D_(KL)(P || Q) ≠ D_(KL)(Q || P). KL divergence is a directed divergence, NOT a true distance metric."

Clean probability density curves, distinct shaded divergence region, balanced square layout.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"Kullback-Leibler (KL) Divergence: Measuring Relative Entropy: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."

A bimodal distribution approximation comparison diagram in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.

Top section: Title in navy serif: "Forward KL vs. Reverse KL: Mode Fitting". Subtitle: "Why supervised training covers all modes, while reinforcement learning locks onto single modes."

Middle section: A bimodal true distribution P(x) (Two distinct Gaussian peaks at x = -2 and x = +2):

We fit a unimodal Gaussian model Q_θ(x) to this bimodal target:

1. Forward KL (Left - argmin_θ D_(KL)(P || Q_θ)):

Formula: E_(x ~ P)[log(P/Q)].

Behavior: If P(x) > 0, then Q(x) MUST be > 0 (avoiding log(P/0) → ∞).

Result: Mode-Covering (Zero-Avoiding)! Model stretches a wide, blurry single peak over BOTH modes. (Used in MLE, cross-entropy pretraining).

1. Reverse KL (Right - argmin_θ D_(KL)(Q_θ || P)):

Formula: E_(x ~ Q)[log(Q/P)].

Behavior: If P(x) ≈ 0, then Q(x) MUST be 0 (avoiding Q log(Q/0)).

Result: Mode-Dropping / Mode-Seeking! Model completely ignores one mode and locks with laser precision onto the other mode. (Used in RLHF, GANs).

Bottom section: Alignment connection: "RLHF with high policy temperature suffers from mode-dropping, generating repetitive high-reward answers."

Sharp bimodal curves, clear visual contrast between wide blur vs single locked peak, balanced square framing.

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Xiao et al. (2023), SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs, ICML ↗

"Forward KL vs. Reverse KL: Mode-Covering vs. Mode-Dropping: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."