Dependence Does Not Change the Mean: The Power of Linearity
Why does statistical dependence radically alter system variance while leaving the expected value of a sum completely unchanged?
How this lesson unfolds.
Begin with Why does changing the circuit topology from series to parallel alter joint outcomes without altering individual initial failure probabilities? The sequence then connects Layering the Experiment: The Temporal Onset of Dependence and builds toward Covariance: Why Scaling Does Not Change the Mean.
Why does changing the circuit topology from series to parallel alter joint outcomes without altering individual initial failure probabilities?
Whether two lightbulbs are wired in series or parallel, gravity, material fatigue, and thermal stress strike bulb A first 50% of the time and bulb B first 50% of the time. The wiring cannot rewrite physics before the first break.
- $$P(\text{first failure} = A) = P(\text{first failure} = B) = \frac{1}{2}$$
- Learning goalDistinguish unconditional initial component failure probabilities from dependent joint system outcomes.
- Helpful foundationsBasic Circuit Topology · Conditional Probability · System Reliability · NIST/SEMATECH e-Handbook of Statistical Methods · SciPy Statistical Functions Documentation
- Common trapDependent system wiring alters joint outcomes, not the intrinsic unconditional failure rates of individual components.
- Next connectionDeconstructing the experiment into two distinct temporal layers.
More context & mechanics
Consider two identical lightbulbs, A and B:
- In a Series Circuit, both bulbs must function for current to flow. Whichever bulb fails first, the circuit breaks, and both end up OFF.
- In a Parallel Circuit, each bulb has its own independent branch. If bulb A fails first, bulb B remains lit; if bulb B fails first, bulb A remains lit.
Look at the probabilities:
- In both circuits, the chance that bulb A fails first is 1/2, and bulb B fails first is 1/2.
- What changes? The downstream joint behavior! In series, failure propagates instantaneously; in parallel, failure is quarantined.
Dependence does not alter initial event probabilities; it alters what happens after an event occurs.
In Reliability Engineering and Fault-Tolerant System Design (e.g., dual jet engines, redundant cloud servers in AWS), engineers configure systems in parallel specifically to ensure that the conditional failure probability P(System Fails | Node 1 Fails) remains isolated.
Thomas Edison and early electrical engineers in the 1880s revolutionized residential lighting by patenting parallel wiring circuits, solving the fatal flaw of series Christmas lights where one dead bulb plunged the entire house into darkness.
Rule: "First Draw is Fair; Wiring Dictates What Follows."
Initial chances are equal; the architecture controls propagation.
For rigorous mathematical formulations of linear expectation operators, system reliability modeling, and joint failure distributions, consult the NIST/SEMATECH e-Handbook of Statistical Methods . For multivariate distributions, indicator simulations, and system reliability analysis in Python, explore SciPy Stats (`scipy.stats`).
At what exact operational stage does statistical dependence activate in an interconnected system?
Think of a game of dominoes: picking which domino gets tapped first is an isolated action. The falling chain reaction of dependence happens only after the initial push.
- $$P(B \text{ off} \mid A \text{ fails}) = 1 \quad \text{vs.} \quad P(B \text{ fails first}) = 0.5$$
- Learning goalDeconstruct dependent systems into sequential temporal layers to isolate where dependence operates.
- Helpful foundationsConditional Probability · Markov Property · Stochastic Processes · NIST/SEMATECH e-Handbook of Statistical Methods · SciPy Statistical Functions Documentation
- Common trapDependence reflects conditional responses after events occur, not pre-existing baseline correlation prior to triggers.
- Next connectionDiscovering why indicator variables postpone dependence when computing expected totals.
More context & mechanics
To resolve the apparent paradox of dependence, decompose the physical process into two chronological layers:
- Layer 1 (The Trigger): Nature selects which bulb fails first. Bulb A and Bulb B have equal probabilities: P(A first) = 1/2, P(B first) = 1/2. At this instant, neither bulb has been affected by the other.
- Layer 2 (The Reaction): The circuit responds to the first failure. In a series circuit, current stops. The second bulb loses power not because its filament burned out, but because the system cut its power line.
Dependence starts AFTER the first event occurs.
Statistical dependence describes how information or physical causation propagates through Layer 2; it cannot reach backward in time to distort the unconditional probabilities of Layer 1.
In Financial Contagion Modeling (e.g., the 2008 Lehman Brothers collapse), Layer 1 is the idiosyncratic default of a single bank; Layer 2 is the counterparty liquidity freeze that cascades through credit default swaps to solvent institutions.
Andrey Markov in 1906 formalized stochastic processes with memory (Markov chains), demonstrating that future states depend only on the present state, formalizing the step-by-step propagation of conditional transitions over time.
Mnemonic: "Layer 1 Picks; Layer 2 Propagates."
First chances are blind; dependencies unfold in the aftermath.
How do binary indicator variables bypass complex web-like dependencies when computing total system expectations?
If you want to know how many people on average attend an office party, you don't need to model all the workplace romances, rivalries, and gossip networks. You just ask each person: 'What is your individual chance of coming?' and add them up.
- $$\mathbb{E}\left[\sum_{k=1}^m I_k\right] = \sum_{k=1}^m \mathbb{E}[I_k] = \sum_{k=1}^m P(I_k = 1)$$
- Learning goalUse indicator variables and expectation linearity to calculate expected totals across dependent networks.
- Helpful foundationsLinearity of Expectation · Indicator Random Variables · Sum of Random Variables · NIST/SEMATECH e-Handbook of Statistical Methods · SciPy Statistical Functions Documentation
- Common trapCalculating expected totals requires only marginal probabilities, not complex joint distributions across components.
- Next connectionContrasting invariant expectation against variance sensitivity to covariance terms.
More context & mechanics
Real-world systems are fiercely complex: components are interconnected, servers share network switches, and employees influence each other.
However, if we define an indicator variable for each component:
Iₖ = 1 & if component k is functioning \ 0 & if component k fails
The total number of functioning components is simply W = ∑_(k=1)^m Iₖ.
By the mathematical superpower of Linearity of Expectation:
E[W] = E[∑_(k=1)^m Iₖ] = ∑_(k=1)^m E[Iₖ] = ∑_(k=1)^m P(component k works)
Notice what just happened: the expected total is completely indifferent to whether the components are wired in series, wired in parallel, or entangled in chaotic feedback loops!
Indicator variables do not erase dependence; they postpone dealing with it until you calculate variance.
In Randomized Algorithms and Theoretical Computer Science, the 'Method of Indicator Random Variables' is used to prove expected runtime bounds (e.g., QuickSort's O(n log n) average comparisons) despite intricate comparisons between dependent pivot partitions.
Paul Erdős pioneered the Probabilistic Method in discrete mathematics in the 1940s, using indicator sums and linearity of expectation to prove the existence of complex combinatorial graph structures without ever constructing them.
Mnemonic: "Expectation Asks the Parts, Not the Connections."
To find the expected sum, just sum the probabilities. Ignore the wiring.
Why does introducing dependence leave the mean completely untouched while radically reshaping system variance?
Two singers can sing individual solos at an average volume of 70 decibels. If they sing together in harmony, the average loudness of each remains 70 dB, but the combined acoustic peaks and dead zones (variance) swing dramatically depending on acoustic interference.
- $$\mathbb{E}[A + B] = \mathbb{E}[A] + \mathbb{E}[B], \quad \text{Var}(A + B) = \text{Var}(A) + \text{Var}(B) + 2\text{Cov}(A, B)$$
- Learning goalContrast expectation invariance under dependence against the strong sensitivity of variance to covariance.
- Helpful foundationsLinearity of Expectation · Variance of Sum · Covariance Definition
- Common trapDependence reshapes variance through covariance terms without altering the expected sum of variables.
- Next connectionDeepening the mathematical proof of why expectation survives dependence.
More context & mechanics
Compare the formulas for two random variables I_A and I_B:
- The Expected Total (Mean):
E[I_A + I_B] = E[I_A] + E[I_B]
This identity requires NO assumptions whatsoever regarding independence. It holds for series circuits, parallel circuits, complete independence, or perfect correlation. The mean looks strictly at parts.
- The System Variance:
Var(I_A + I_B) = Var(I_A) + Var(I_B) + 2Cov(I_A, I_B)
Variance looks at relationships!
- Independent Case: Cov(I_A, I_B) = 0 ⟹ Var = Var(I_A) + Var(I_B).
- Positive Dependence (Series / Shared Failure): Cov(I_A, I_B) > 0. When one fails, both fail; when one works, both work. The outcomes polarize to 0 or 2, making variance explode!
- Negative Dependence (Hedging / Alternation): Cov(I_A, I_B) < 0. Outcomes stay close to 1, making variance shrink.
Dependence changes variance far more than mean because covariance enters the variance equation, but is completely absent from the mean.
In Modern Portfolio Theory, combining two stocks with identical expected returns (10\%) yields a portfolio with the exact same expected return (10\%). However, if the stocks have negative covariance, portfolio variance collapses, delivering the same return with minimal risk.
Harry Markowitz (1952) based his entire revolution of quantitative finance on this mathematical distinction: that diversification alters variance without penalizing expected return.
Maxims:
- "Mean is Blind to Bonds."
- "Variance is the Price of Partnership."
Expectation adds parts; Variance adds parts plus twice the covariance.
What fundamental property of mathematical integration ensures that expectation survives arbitrary dependence?
If you slice an apple and an orange, weighing the apple pieces and orange pieces separately gives the exact same total as weighing them together on the scale. The physical connection between them cannot add or destroy mass.
- $$\iint (x+y) f(x,y) \,dx\,dy = \mathbb{E}[X] + \mathbb{E}[Y]$$
- Learning goalProve analytically that linearity of expectation holds for arbitrary dependent random variables.
- Helpful foundationsDouble Integrals · Marginal Distributions · Lebesgue Integration
- Common trapLinearity of expectation is an unconditional property requiring no independence between summands.
- Next connectionResolving the final philosophical puzzle of dependent systems.
More context & mechanics
To see why E[X + Y] = E[X] + E[Y] survives any dependence, examine the continuous joint expectation:
E[X + Y] = ∫_(-∞)^(∞) ∫_(-∞)^(∞) (x + y) f_(X,Y)(x, y) dx dy
Distribute the sum inside the integral:
= ∫_(-∞)^(∞) ∫_(-∞)^(∞) x f_(X,Y)(x, y) dx dy + ∫_(-∞)^(∞) ∫_(-∞)^(∞) y f_(X,Y)(x, y) dx dy
In the first term, integrate out y: ∫_(-∞)^(∞) f_(X,Y)(x, y) dy = f_X(x) (the marginal density of X!).
= ∫_(-∞)^(∞) x f_X(x) dx + ∫_(-∞)^(∞) y f_Y(y) dy = E[X] + E[Y]
The joint density f_(X,Y)(x, y) collapses into marginals f_X(x) and f_Y(y) automatically.
Expectation never needs to know how X and Y are wired together; integration over the full space absorbs all mutual influences.
In Computational Physics and Monte Carlo Integration, this property justifies estimating aggregate thermodynamic quantities across millions of strongly interacting quantum particles by summing single-particle marginal expectations.
Henri Lebesgue in 1902 introduced the Lebesgue integral, establishing that integration is fundamentally a linear measure-preserving operation that preserves additivity over arbitrary measurable spaces.
Visual Rule: "Integrate Out the Partner."
∫ f(x, y) dy = f(x). The joint distribution collapses into the marginal, leaving each expectation untouched.
How can components intensely depend on one another while their aggregate expectation remains strictly additive?
Two weights on a seesaw can be tied together with a rigid rope or free to roll. How they sway and collide changes drastically, but the seesaw's balancing pivot (the center of mass) remains fixed in place.
- $$\mathbb{E}\left[\sum_{i=1}^n X_i\right] = \sum_{i=1}^n \mathbb{E}[X_i]$$
- Learning goalSynthesize how dependence reshapes probability variance while strictly preserving aggregate expectation.
- Helpful foundationsLinearity of Expectation · Probability Mass Function · Extreme Value Theory
- Common trapInterdependent component failures alter variance, yet expected survival remains the sum of marginal probabilities.
- Next connectionTransitioning from theoretical probability models to empirical survey frames.
More context & mechanics
Return to the two lightbulbs in series. If bulb A fails, bulb B goes dark. They clearly exert massive mutual physical influence.
How can their expected working total still be E[I_A] + E[I_B]?
- Look at the probability space:
If independent: Outcomes (1,1), (1,0), (0,1), (0,0) each have probability 1/4.
Expected total: 2(1/4) + 1(1/4) + 1(1/4) + 0(1/4) = 1.0.
- In series: The circuit cannot have (1,0) or (0,1). The outcomes collapse to (1,1) with probability 1/2 and (0,0) with probability 1/2.
Expected total: 2(1/2) + 0(1/2) = 1.0.
The expected total is IDENTICAL!
Dependence shifted probability from the middle outcomes (1 bulb lit) to the extreme outcomes (2 lit or 0 lit).
This radical shift in probability mass from center to tails causes variance to quadruple, but the mean remains rock solid at 1.0.
In Insurance and Catastrophe Modeling (hurricanes, earthquakes), policies are heavily correlated: when a hurricane hits Florida, thousands of homes file claims simultaneously. The insurer's long-run expected average payout is the sum of individual house risks, but the variance is astronomical, requiring massive reinsurance reserves to survive tail risk.
Emil Julius Gumbel (1958) pioneered the statistics of extremes, analyzing how dependence in environmental disasters alters extreme value distributions without distorting historical mean values.
The Golden Rule of Statistics:
"Dependence Redistributes Probability Across Outcomes; It Never Moves the Center of Mass."
Mean adds parts; Variance measures the spread created by their entanglement.
How does Covariance: Why Scaling Does Not Change the Mean operate mathematically at scale?
Visual lever balance comparison:
- $$Subtitle: "\mathbb{E}[a X + b] = a \mathbb{E}[X] + b.$$
- Learning goalMaster the algebraic and geometric derivation of Covariance: Why Scaling Does Not Change the Mean.
- Helpful foundationsFoundations of Statistics · Coordinate Invariance & Geometric Projections · Wasserman (2004), All of Statistics: A Concise Course in Statistical Inference, Springer
- Common trapAssuming Covariance: Why Scaling Does Not Change the Mean is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
An educational statistics balance diagram in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.
Top section: Title in navy serif: "Linearity of Expectation: Scaling and Shifting". Subtitle: "E[a X + b] = a E[X] + b. The mean tracks spatial shifts and stretches linearly."
Middle section: Visual lever balance comparison:
Show a physical fulcrum (the center of mass / expected value μ) supporting data point weights along a beam:
- Base Distribution X: Mean is at fulcrum point μ = 5.
- Shift by Constant b (X + 3): Every data point shifts 3 units to the right; the fulcrum slides exactly 3 units to μ + 3 = 8 (Shape is unchanged).
- Scale by Factor a (2X): Every point's distance from origin doubles; the fulcrum moves to 2 × μ = 10 (Spread expands by 2x, variance expands by 4x!).
Bottom section: Independence theorem: "E[X + Y] = E[X] + E[Y] holds ALWAYS, even when X and Y are completely dependent! Linearity of expectation requires zero independence assumptions."
Pristine balance lever graphic, clear coordinate shift callouts, balanced square layout.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
Wasserman (2004), All of Statistics: A Concise Course in Statistical Inference, Springer
"Covariance: Why Scaling Does Not Change the Mean: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."