Mental Model Behind Mean & Variance
What is the mathematical foundation of mental model behind mean & variance?
How this lesson unfolds.
Begin with How do seven conceptual steps transform raw scattered observations into standardized units of real-world spread? The sequence then connects The First Attempt Failed: Why Raw Deviations Always Cancel Out and builds toward The First Attempt Failed: Why Raw Deviations Always Cancel Out.
How do seven conceptual steps transform raw scattered observations into standardized units of real-world spread?
Data science is a distillation process: raw observations are counted into frequencies, normalized into probabilities, balanced at the mean, measured in squared variance, and restored to physical reality via standard deviation.
- $$\mathbb{E}[X] = \sum x_i P(X = x_i), \quad \sigma^2 = \sum (x_i - \mu)^2 P(X = x_i), \quad \sigma = \sqrt{\sigma^2}$$
- Learning goalTrace the 7-stage pipeline distilling raw data into standard deviation.
- Helpful foundationsFrequency distribution · Relative frequency · Expected value · Standard deviation · NIST/SEMATECH e-Handbook of Statistical Methods · SciPy Stats
- Common trapMean, variance, and standard deviation are connected pipeline stages, not isolated formulas.
- Next connectionExamine why averaging raw deviations always fails catastrophically.
More context & mechanics
The transformation pipeline follows an unbroken logical progression:
- Raw Data (xᵢ): The unprocessed collection of individual measurements.
- Frequency (fᵢ): Tallying the occurrence count of each unique value.
- Relative Frequency (fᵢ / N): Converting discrete counts into normalized proportions of the total sample.
- Probability P(X = x): Interpreting proportions as the true underlying likelihood of future realizations.
- Mean / Expectation E[X] = sum x P(X = x): Finding the physical center of gravity of the distribution.
- Variance Var(X) = E[(X - mu)²]: Quantifying the average squared distance of all outcomes from the mean.
- Standard Deviation sigma = sqrt(Var(X)): Taking the square root to return dispersion back to original physical units (meters, dollars, seconds).
In deep learning vision models (e.g., ResNet, ViT), input images undergo this exact pipeline: pixel intensities are aggregated, their mean and variance across batches computed (Batch Normalization / Layer Normalization), and standardized by subtracting batch mean and dividing by standard deviation to prevent exploding/vanishing gradients.
Adolphe Quetelet in the 1830s pioneered the concept of the 'average man' (l'homme moyen) by tabulating human physical measurements into relative frequency distributions, inspiring Galton and Pearson to invent standard statistical dispersion metrics.
Mnemonic ladder: 'Values -> Counts -> Proportions -> Probabilities -> Center -> Squared Spread -> True Scale.'
V-C-P-P-C-S-T.
Why does attempting to compute average spread by summing raw deviations (x - mu) always result in zero for any dataset?
The arithmetic mean acts like the fulcrum of a seesaw. By physical equilibrium, the total torque of points to the right exactly balances the torque of points to the left, guaranteeing their algebraic sum is zero.
- $$\sum_{i=1}^N (x_i - \mu) = 0 \implies \frac{1}{N}\sum_{i=1}^N (x_i - \mu) = 0$$
- $$By definition, the mean is the exact balance point where positive and negative deviations are equal in magnitude, causing their sum to strictly equal zero.$$
- Learning goalProve algebraically why raw deviations sum to zero, necessitating squared differences.
- Helpful foundationsArithmetic mean · Summation properties · Center of mass · NIST/SEMATECH e-Handbook of Statistical Methods · SciPy Statistical Functions Documentation
- Common trapZero sum of raw deviations indicates signed cancellation around the mean, not zero spread.
- Next connectionExplore how squared deviations quantify variance across fundamental probability models.
More context & mechanics
Consider Dataset A: 1, 2, 3, 4, 5.
The mean is mu = (1 + 2 + 3 + 4 + 5) / 5 = 15 / 5 = 3.
Now let us try the obvious first idea: measure spread by averaging the deviations from the mean (x - mu):
- (1 - 3) = -2
- (2 - 3) = -1
- (3 - 3) = 0
- (4 - 3) = +1
- (5 - 3) = +2
Sum of deviations: (-2) + (-1) + 0 + 1 + 2 = 0!
Average deviation: 0 / 5 = 0.
This is not a coincidence of this dataset; it is an algebraic identity for ANY dataset:
sum_(i=1)^N (xᵢ - mu) = sum xᵢ - sum mu = N mu - N mu = 0.
Because positive and negative distances perfectly destroy each other, raw deviations communicate zero information about spread.
The solution: Eliminate signs before averaging. While absolute values |x - mu| are possible (Mean Absolute Deviation), squaring (x - mu)² yields differentiable, additive geometry:
(-2)² + (-1)² + 0² + 1² + 2² = 4 + 1 + 0 + 1 + 4 = 10, giving Var = 10 / 5 = 2.
In numerical optimization, gradient descent cannot optimize zero-sum residuals directly. Loss functions must penalize deviation magnitudes. Mean Squared Error (MSE = 1/N sum (y - ŷ)²) is the cornerstone loss function of linear regression, neural network regression, and Kalman filters because it is smooth, convex, and non-canceling.
Carl Friedrich Gauss addressed this exact dilemma in his 1809 work on celestial orbit calculation. Gauss demonstrated that squaring residuals corresponds to maximum likelihood estimation under a Gaussian error distribution, establishing the method of least squares over Boscovich's absolute deviation approach.
Mnemonic: 'The mean is the fulcrum; raw balance is always zero.'
To measure distance without cancellation, we must square before we share.