Capstone 1: Fashion-MNIST Classification Engine From Scratch
How do data normalization, multi-layer forward propagation, cross-entropy backpropagation, and Adam optimization unite to classify complex real-world clothing images?
How this lesson unfolds.
Having mastered neurons, forward pipelines, loss calculus, backpropagation engines, and adaptive optimizers, we now assemble all components into our first complete capstone: building a high-accuracy Fashion-MNIST classification system from scratch.
How are raw 28x28 pixel byte arrays transformed into zero-mean unit-variance input tensors for deep neural network ingestion?
Imagine weighing ingredients for a recipe where flour is measured in tons and salt is measured in milligrams. The machine will ignore the salt completely because the numbers for flour are a million times larger! Raw pixels range from 0 to 255, creating massive input signals that saturate neurons. Dividing by 255 puts every pixel on a level playing field between 0.0 and 1.0.
- $$\mathbf{x}_{std} = \frac{\mathbf{x} / 255 - \mu}{\sigma}$$
- Learning goalConstruct an end-to-end image ingestion and normalization pipeline for Fashion-MNIST in Python/NumPy.
- Helpful foundationsVector flattening and matrix representations · Z-score standardization formula · Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning (Xiao et al., 2017)
- Common trapPassing raw integers [0, 255] into neural networks; unnormalized inputs cause initial logits to blow up into hundreds, causing softmax saturation and zero learning.
- Next connectionFeed these normalized 784-dimensional vectors into the Deep Multilayer Classification Engine (Screen 2).
More context & mechanics
Each image is an uncompressed 28 × 28 matrix of unsigned 8-bit integers X_(raw) ∈ 0, ṡ, 255^(28 × 28).
The data pipeline executes three transformations:
- Spatial Flattening:
x = vec(X) ∈ R^(784) (28 × 28 = 784 feature inputs)
- Min-Max Scaling:
x_(norm) = (x) / (255.0) ∈ [0.0, 1.0]^(784)
- Global Standardization (Z-Score):
x_(std) = (x_(norm) - μ_(global)) / (σ_(global) + ε) (μ ≈ 0.286, σ ≈ 0.353)
Suppose pixel at coordinate (14, 14) has raw byte intensity x_(raw) = 178:
- Step 1 (Min-Max): x_(norm) = 178/255 = 0.6980.
- Step 2 (Global Standardize):
x_(std) = (0.6980 - 0.2860) / (0.3530) = (0.4120) / (0.3530) = +1.1671
A dark background pixel with x_(raw) = 0:
x_(std) = (0.0 - 0.2860) / (0.3530) = -0.8102
The entire input volume is centered around zero, ensuring that initial weight updates Δ W = x^T δ do not introduce systematic directional bias.
Standard preprocessing for PyTorch `torchvision.transforms.Compose([ToTensor(), Normalize((0.2860,), (0.3530,))])`.
Created by Han Xiao, Kashif Rasul, and Roland Vollgraf (Zalando Research, 2017) to replace the trivially solvable MNIST digits dataset with a modern benchmark.
Flatten the grid, divide by 255, center around zero: well-conditioned inputs make well-behaved gradients.
How do affine layer projections, ReLU activations, dropout masks, and softmax output heads synchronize to achieve >88% test accuracy?
Think of an art appraisal board. The raw pixels (784 features) enter the junior inspectors (Layer 1, 256 neurons), who detect simple textures like wool, cotton, or leather straps. The senior inspectors (Layer 2, 128 neurons) combine textures into shapes like shoe heels, trouser legs, or coat collars. The final judge (Layer 3, 10 neurons) tallies the evidence and stamps the verdict: 'Sneaker (98% probability)'.
- $$\mathbf{Z}_3 = ((\text{ReLU}(\mathbf{X}\mathbf{W}_1 + \mathbf{b}_1)\mathbf{W}_2 + \mathbf{b}_2))\mathbf{W}_3 + \mathbf{b}_3$$
- $$A 3-layer architecture (784 -> 256 -> 128 -> 10) with He normal initialization, ReLU activations, inverted dropout (p=0.2), and AdamW optimization achieves 88.5% test accuracy on Fashion-MNIST with zero framework dependencies.$$
- Learning goalConstruct, train, and evaluate a 3-layer neural network on Fashion-MNIST in pure Python/NumPy achieving >88% test accuracy.
- Helpful foundationsFashion-MNIST data normalization pipeline · Combined softmax cross-entropy gradient collapse · AdamW optimizer dynamics · Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet (He et al., 2015)
- Common trapThinking deep networks need convolutional layers to classify Fashion-MNIST; a well-regularized 3-layer MLP with He initialization readily achieves 88-89% accuracy.
- Next connectionCompare pure NumPy implementations with modern GPU acceleration in Chapter 33.
More context & mechanics
The network architecture is defined by parameter matrices:
W₁ ∈ R^(784 × 256), W₂ ∈ R^(256 × 128), W₃ ∈ R^(128 × 10)
Forward propagation computes:
Z₁ = X W₁ + b₁, A₁ = Dropout(ReLU(Z₁), p=0.2)
Z₂ = A₁ W₂ + b₂, A₂ = Dropout(ReLU(Z₂), p=0.2)
Z₃ = A₂ W₃ + b₃, P = Softmax(Z₃)
Backward propagation injects δ₃ = 1/B(P - Y), propagating through transposed weights and ReLU masks.
Parameter footprint calculation:
- Layer 1: 784 × 256 + 256 = 200,704 + 256 = 200,960 parameters.
- Layer 2: 256 × 128 + 128 = 32,768 + 128 = 32,896 parameters.
- Layer 3: 128 × 10 + 10 = 1,280 + 10 = 1,290 parameters.
- Total Parameters: 200,960 + 32,896 + 1,290 = 235,146 parameters (0.94 MB in FP32).
When trained for 20 epochs with batch size B=64, AdamW (α=10^(-3), λ=10^(-2)), validation accuracy surpasses 88.5\%.
Serves as the canonical baseline for multi-class computer vision before moving to Convolutional Neural Networks.
Represents the modern synthesis of Rumelhart's backpropagation (1986), Nair & Hinton's ReLU (2010), and Kingma & Ba's Adam (2014).
784 inputs detect textures in 256, build shapes in 128, and decide classes in 10: depth structures hierarchy.