Variance Question On Average

For an indicator random variable I with P(I = 1) = p and P(I = 0) = 1 - p, the expected value is E[I] = p.

Let us examine the exact contribution of each outcome to variance:

1. Outcome I = 1 (Success):

• Squared distance from mean: (1 - p)².
• Frequency (Probability): p.
• Contribution: p * (1 - p)².

1. Outcome I = 0 (Failure):

• Squared distance from mean: (0 - p)² = p².
• Frequency (Probability): (1 - p).
• Contribution: (1 - p) * p².

Total Variance:

Var(I) = p(1 - p)² + (1 - p)p²

= p(1 - p)[(1 - p) + p]

= p(1 - p)[1] = p(1 - p).

Every piece of the formula reflects physical mechanics: how far times how often.

In credit scoring models (e.g., FICO or XGBoost loan default classifiers), default is rare (e.g., p = 0.02). When a borrower defaults, their squared deviation from the expected rate is huge (1 - 0.02)² = 0.9604, but its frequency is only 0.02. Good risk calibration balances rare massive deviations against common negligible deviations.

Pafnuty Chebyshev in 1867 popularized the method of moments in probability, demonstrating that the expected squared deviation from the center is the minimal quadratic measure capable of bounding arbitrary tail risks.

Mnemonic: 'Distance-squared times Probability.'

(1-p)² p + p² (1-p) = p(1-p). The cross-terms harmonize into a single product.

Suppose an indicator has P(I = 1) = p = 0.7. The mean is E[I] = 0.7.

The two squared distances from the mean are:

• For outcome 1: (1 - 0.7)² = 0.3² = 0.09.
• For outcome 0: (0 - 0.7)² = (-0.7)² = 0.49.

A naive student might ask: 'Since there are only two squared distances (0.09 and 0.49), shouldn't we simply average them?'

Naive calculation:

(0.09 + 0.49) / 2 = 0.58 / 2 = 0.29.

But the true variance is:

Var(I) = p(1 - p) = 0.7 * 0.3 = 0.21!

Why did the naive average produce 0.29 instead of 0.21?

Because the naive average implicitly assigns a 50% weight to outcome 0 and a 50% weight to outcome 1:

0.5 0.09 + 0.5 0.49 = 0.29.

In reality, outcome 1 happens 70% of the time, and outcome 0 happens only 30% of the time. The smaller squared distance (0.09) occurs over twice as frequently as the large squared distance (0.49).

In data analytics and survey analysis, unweighted averages across stratified samples produce selection bias. If an online platform surveys 10 power users and 10 casual users, but power users represent 1% of the actual user base, unweighted variance calculations wildly distort user behavior metrics.

Adolphe Quetelet in nineteenth-century social physics cautioned against 'the fallacy of the equal weight', proving that demographic averages are meaningless unless weighted by census frequencies.

Mnemonic: 'Don't split 50/50 if reality doesn't split 50/50.'

Equal weighting assumes a fair coin; weighted averaging respects the true p.

Suppose we record 10 actual observations of an indicator with 7 ones and 3 zeros. Mean x̄ = 0.7.

The traditional sample formula sums the squared deviations and divides by 10:

Var = [ (1 - 0.7)² + (1 - 0.7)² + ... (7 times) + (0 - 0.7)² + ... (3 times) ] / 10

Step 1: Group identical terms:

Var = [ 7 (1 - 0.7)² + 3 (0 - 0.7)² ] / 10

Step 2: Distribute the denominator 10 across the addition:

Var = (7 / 10) (1 - 0.7)² + (3 / 10) (0 - 0.7)²

Step 3: Recognize the relative frequencies:

7 / 10 = 0.7 = p, and 3 / 10 = 0.3 = (1 - p).

Var = p (1 - p)² + (1 - p) p².

The division by 10 did not disappear—it became the weights 0.7 and 0.3!

In streaming algorithms and distributed data systems (e.g., Apache Spark or Flink), computing variance over petabytes of data cannot wait to divide by total count N at the end. Instead, online algorithms (like Welford's algorithm) update weighted running probabilities at each batch.

B. P. Welford in 1962 published an elegant one-pass algorithm for calculating sample mean and variance on digital computers, exploiting this exact algebraic distribution of the sample size denominator to prevent catastrophic numerical cancellation.

Mnemonic: 'Denominator distributes into counts.'

Counts / N becomes Probability. [7(gap²) + 3(gap²)] / 10 = 0.7(gap²) + 0.3(gap²).

Look at the parallel mathematical structures:

1. Expected Value (Mean):

• What is being averaged: The raw values x.
• For an indicator: 0 and 1.
• Weighted formula: E[X] = sum x P(X = x) = 0 (1 - p) + 1 * p = p.

1. Variance:

• What is being averaged: The squared distances (x - mu)².
• For an indicator: (0 - p)² and (1 - p)².
• Weighted formula: Var(X) = E[(X - mu)²] = (0 - p)² (1 - p) + (1 - p)² p = p(1 - p).

The mathematical operator E[...] is identical in both equations.

Variance is not an exotic alien concept; it is merely an average wearing a different hat.

In quantitative risk management and reinforcement learning (e.g., Value-at-Risk and Actor-Critic architectures), generalized expectations E[g(X)] are computed for diverse objective functions g(x): linear g(x)=x gives expected return, quadratic g(x)=(x-μ)² gives volatility risk, and threshold indicator g(x)=1_(x < c) gives probability of ruin.

Henri Lebesgue in 1904 revolutionized mathematical analysis by introducing the Lebesgue integral, unifying discrete weighted sums and continuous integrals under a single abstract measure-theoretic expectation operator ∫ Ω g(ω) dP(ω).

Mnemonic: 'Mean averages locations; Variance averages distances-squared.'

E[Payload]: change the payload, keep the averaging rule.

Suppose we collect 100 measurements: ninety-nine values of 1, and a single value of 99.

Step 1: Raw Uncompressed Average:

Average = (1 + 1 + ... [99 times] + 99) / 100

Step 2: Compress Repeated Values:

Instead of writing 1 ninety-nine times, group identical outcomes:

Average = [ 99 (1) + 1 (99) ] / 100

Step 3: Move Denominator to Form Weights:

Average = (99 / 100) (1) + (1 / 100) (99) = 0.99 (1) + 0.01 (99) = 0.99 + 0.99 = 1.98.

Weights are not arbitrary numbers chosen by an analyst; they are the natural consequence of data compression. Each weight wᵢ = nᵢ / N reflects the exact empirical fraction of the dataset represented by that value.

In data warehouse query engines (e.g., Snowflake, BigQuery, ClickHouse), Run-Length Encoding (RLE) and dictionary compression compress billions of repeated rows into unique values and count vectors, allowing lightning-fast weighted aggregations without scanning raw rows.

John Graunt in 1662 published the Natural and Political Observations Made upon the Bills of Mortality, the first demographic study in history. Graunt compressed London death rolls into tabular summary frequencies, inventing empirical weighted estimation.

Mnemonic: 'Count, Group, Divide.'

Raw tally -> Count times Value -> Divide by total -> Weight times Value.

When p = 0.7:

Outcomes occur in an asymmetric 70:30 ratio:

• Outcome 1 occurs 70% of the time, producing squared distance (1 - 0.7)² = 0.09.
• Outcome 0 occurs 30% of the time, producing squared distance (0 - 0.7)² = 0.49.

Compare two calculations:

1. False Assumption (Equal 50/50 Weight):

Spread = 0.5 (0.09) + 0.5 (0.49) = 0.045 + 0.245 = 0.290.

1. Actual Reality (True 70/30 Weight):

Spread = 0.7 (0.09) + 0.3 (0.49) = 0.063 + 0.147 = 0.210.

The equal-weight assumption over-weights the rare 0.49 distance by a factor of 5/3, artificially inflating perceived variance from 0.21 to 0.29.

Equal weights are only valid when p = 0.5. For all other probabilities, equal weights distort the measurement.

In medical clinical trials for rare disease treatments, healthy control outcomes vastly outnumber adverse reactions. If a predictive diagnostic algorithm evaluates classification error using unweighted accuracy (equal weight on both classes), it achieves 99% accuracy simply by predicting no one is sick—a catastrophic failure known as the class imbalance problem.

Jerzy Neyman in 1934 published his landmark paper on stratified sampling, proving that optimal sample allocation (Neyman allocation) requires weighting strata by both their population proportion and their internal variance, formally discrediting naive unweighted sampling.

Mnemonic: 'Equal weights = coin toss fantasy.'

Unless p is 0.5, equal weights are a mathematical lie.

In colloquial English, 'weight' often connotes value, significance, or moral priority ('give more weight to safety').

In mathematics and statistical engineering, WEIGHT MEANS REPRESENTATION.

If an observation has weight wᵢ = 0.7, it does not mean that observation is 'better' or 'more important' than one with weight 0.3. It means that out of every 10 draws from the physical universe, that state represents 7 of them.

This conceptual clarity is the foundation of:

1. Horvitz-Thompson Estimation (1952): Unbiased population estimation from unequal probability sampling by weighting each sample by 1 / πᵢ (the inverse probability of selection).
2. Importance Sampling: Estimating properties of an expensive target distribution P by sampling from an easy proposal distribution Q and re-weighting by the likelihood ratio w(x) = P(x) / Q(x).
3. Large Language Model Alignment: In Reinforcement Learning from Human Feedback (RLHF) and Proximal Policy Optimization (PPO), token updates are weighted by importance sampling ratios π_θ(a|s) / π_(old)(a|s) to ensure safe, stable policy improvement.

In autonomous vehicle perception systems, edge cases (e.g., pedestrians crossing at night in heavy snow) occur rarely in raw driving logs (<0.01%). In model training loss functions, these rare samples receive higher loss weights (focal loss or inverse frequency weighting) to force the neural network to represent safety-critical rare events accurately.

Daniel Horvitz and Donovan Thompson formulated their general Horvitz-Thompson estimator in 1952, proving that inverse probability weighting is the unique linear unbiased estimator for survey sampling from finite populations.

Mnemonic: 'Weight = Census, not Status.'

Weight measures how many votes an outcome casts in reality, not how much we like it.

The expected generalization error of any predictive model decomposes into three distinct components (The Bias-Variance Decomposition):

MSE = Bias² + Variance + σ²_(ε)

Where:

• Bias: Error from erroneous assumptions in the learning algorithm (underfitting; the model is too rigid to capture the truth).
• Variance: Error from sensitivity to small fluctuations in the training dataset (overfitting; the model changes wildly if trained on a different sample from the same population).
• σ²_(ε): Irreducible error (intrinsic noise in the environment).

When model complexity escalates (e.g., high-degree polynomials or deep neural networks without regularization), Bias drops to near zero, but Variance explodes.

The overfitted model captures every random scatter point in the training data. Because the scatter points contain random noise, the model's predictions fluctuate wildly on unseen test data.

In financial algorithmic trading, a quantitative strategy with hundreds of tunable parameters can achieve a 100% win rate on historical backtests. But because of high estimator variance, it collapses immediately in live trading when facing unseen market conditions. Quants use L2 regularization (Ridge / Weight Decay), dropout, and cross-validation to penalize parameter variance.

Stuart Geman, Elie Bienenstock, and René Doursat published their foundational paper 'Neural Networks and the Bias/Variance Dilemma' in 1992, formalizing how statistical learning theory governs generalization across non-parametric architectures.

Mnemonic: 'High Bias = Too stubborn (rigid line); High Variance = Too hyperactive (jagged line).'

Overfitting is model hyperactivity driven by unchecked variance.

Consider 10 indicator outcomes containing 7 ones and 3 zeros:

1. Ordinary Sample Average:

x̄ = [ 7(1) + 3(0) ] / 10 = [ 7 + 0 ] / 10 = 7 / 10 = 0.7.

1. Rewrite Using Relative Frequency:

x̄ = (7 / 10) 1 + (3 / 10) 0 = 0.7 1 + 0.3 0 = 0.7.

1. General Theoretical Expectation:

For any indicator random variable I in 0, 1 with P(I = 1) = p and P(I = 0) = 1 - p:

E[I] = 1 P(I = 1) + 0 P(I = 0)

= 1 p + 0 (1 - p)

= p.

Because multiplying by 0 eliminates the second term completely, the expected value of any indicator variable is ALWAYS identically equal to the probability of the indicated event:

E[1_A] = P(A).

This identity E[1_A] = P(A) is the theoretical bedrock of Monte Carlo simulation. To compute the probability of a complex high-dimensional event A (e.g., an exotic derivative finishing in the money or a nuclear particle penetrating a shield), computers simulate N runs, record indicator 1_A, and compute the sample average. By the Law of Large Numbers, this average converges to the exact probability P(A).

Stanislaw Ulam and John von Neumann invented the Monte Carlo method at Los Alamos in 1946 while working on neutron diffusion, relying directly on the identity that probability can be computed as the average of indicator realizations.

Mnemonic: 'Zeros vanish; only ones speak.'

E[I] = 1 p + 0 (1-p) = p. The mean of an indicator IS its probability.