Law Of Large Numbers
How does repeating an unpredictable random experiment an enormous number of times forge an immutably stable empirical average?
How this lesson unfolds.
Begin with Why does individual unpredictability transform into statistical certainty when an experiment is repeated thousands of times? Use the picture, intuition and explanation together before moving on.
Why does individual unpredictability transform into statistical certainty when an experiment is repeated thousands of times?
A single coin flip is pure chaos: 100% heads or 0% heads. But flip it 10,000 times, and random fluctuations drown each other out, locking the average rock-solid at 0.500.
- $$\bar{X}_n \xrightarrow{P} \mu, \quad \operatorname{Var}(\bar{X}_n) = \frac{\sigma^2}{n} \to 0, \quad P\left(\lim_{n \to \infty} \bar{X}_n = \mu\right) = 1$$
- Learning goalProve the Weak Law of Large Numbers using Chebyshev's inequality and variance shrinkage.
- Helpful foundationsSample mean · Linearity of expectation · Variance of sums of independent variables · Chebyshev's Inequality · Foundations of the Theory of Probability (Andrey Kolmogorov, 1933) · An Essay towards solving a Problem in the Doctrine of Chances (Thomas Bayes, 1763)
- Common trapAverages do not compensate past streaks; convergence occurs through dilution, not memory.
- Next connectionExplore how the Law of Total Probability partitions composite outcomes.
More context & mechanics
Let X₁, X₂, ..., Xₙ be i.i.d. random variables with finite mean μ and variance σ². Define the sample mean X̄ₙ = 1/n∑_(i=1)^n Xᵢ. The expectation is E[X̄ₙ] = μ, and the variance is Var(X̄ₙ) = (σ²) / (n). Notice the denominator n in the variance! As n grows, the variance of the sample average shrinks toward zero: lim_(n → ∞) Var(X̄ₙ) = lim_(n → ∞) (σ²) / (n) = 0. By Chebyshev's Inequality, for any margin ε > 0: P(|X̄ₙ - μ| ≥ ε) ≤ (σ²) / (n ε²) →(n → ∞) 0. This is the Weak Law of Large Numbers (WLLN): the probability that the sample average deviates from the true mean by even the tiniest sliver ε vanishes completely.
Casinos (e.g. in Las Vegas or Macau) build billion-dollar business empires on the Law of Large Numbers: on any single roulette spin, a player might win (chaos), but across 10 million spins a year with a 5.26% house edge, the casino's net gross revenue converges to exactly 5.26% with zero financial risk.
Jacob Bernoulli proved the first version for binary trials in Ars Conjectandi (1713) after 20 years of research. Siméon Denis Poisson generalized it in 1837 and coined the name la loi des grands nombres ('The Law of Large Numbers'). Later, Andrey Kolmogorov in 1930 proved the Strong Law of Large Numbers (SLLN), establishing almost sure convergence P(lim_(n → ∞) X̄ₙ = μ) = 1.
The Dilution Principle: 'Past noise is diluted by future volume'. Variance of the average is σ² / n; as n → ∞, uncertainty vanishes.