A/B Testing: From an Observed Lift to a Trustworthy Decision
When a treatment beats a control, how can we tell whether the difference is real, practically useful, and attributable to the treatment?
How this lesson unfolds.
Begin with Why do major technology companies test changes instead of trusting intuition? The sequence then connects Turn the product question into a metric and builds toward The same reasoning powers everyday experiments.
Why do major technology companies test changes instead of trusting intuition?
A/B testing randomly separates comparable users so product changes can be evaluated with evidence rather than anecdotes.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionTurn the product question into a metric
What would a search-ranking experiment actually measure?
Compare click-through rates for the current and new rankings; a useful causal question starts with an observable outcome.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionCreate comparable groups
How can one experiment compare two rankings fairly?
Random assignment divides eligible users into control and treatment groups that run concurrently, attempting to balance device, location, intent and browsing habits.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionDefine success before seeing results
Which outcome counts as improvement?
Predeclare the primary metric, supporting metrics and guardrails so the goal does not move after the data arrive.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionObserve the two rates
What does the experiment report before inference?
The control records 500 clicks among 5,000 users and the treatment 600 among 5,000, giving observed rates of 10% and 12%.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionEstimate the treatment lift
What changed in the observed sample?
The estimated absolute lift is 12% minus 10%, or 2 percentage points; relative lift would be a different quantity.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionFind the variance of one click indicator
Where does uncertainty in a sample proportion begin?
A Bernoulli click indicator has variance p(1-p); averaging n indicators divides that variance by n.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionUnderstand treatment-control covariance
Why are one user's control and treatment indicators dependent?
For one user, control and treatment are mutually exclusive. Under a 50:50 assignment, C_iT_i is always zero, so Cov(C_i,T_i)=0-(0.5)(0.5)=-0.25.
Under a 50:50 assignment, C_iT_i is always zero, so Cov(C_i,T_i)=0-(0.5)(0.5)=-0.25.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionRandom assignment can create dependence
Does random assignment always mean independent treatment indicators?
No. Independent user outcomes and dependent allocation indicators are different ideas; fixing the number of treatment slots couples the assignments.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionA difference is an estimate
Would another random assignment produce exactly two points of lift?
Repeated comparable experiments yield different observed lifts; their spread is the sampling uncertainty of the estimator.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionGive uncertainty the right unit
Is an error of two large or small?
Uncertainty must be expressed in the metric's unit; for CTR the standard error and lift are percentage-point quantities.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionCompare signal with random noise
How many standard errors away from zero is the observed lift?
The z-score divides the estimated difference by its standard error; here the two-point lift is about 3.19 standard errors from zero.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionFour tools answer four questions
What do variance, standard error, z-score and confidence interval each tell us?
Variance describes fluctuation, standard error describes estimator precision, the z-score measures distance from a reference, and an interval displays plausible effect sizes under its assumptions.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionWhy 1.96 appears
Where does the familiar 95% multiplier come from?
For a standard Normal variable, about 95% of the distribution lies between -1.96 and 1.96; 1.96 is a chosen quantile, not a universal safety constant.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionTurn standard error into a margin
How wide should a 95% uncertainty band be?
Multiplying the 0.626 percentage-point standard error by 1.96 gives an approximate margin of error of 1.23 percentage points.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionBuild the interval
Which effect sizes remain plausible under the model?
An estimate of 2.00 points plus or minus 1.23 gives an approximate interval from 0.77 to 3.23 percentage points.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionTranslate the interval into business effects
What range of additional clicks is compatible with the estimate?
At 100,000 search-result views, a lift of 0.77 to 3.23 percentage points corresponds to roughly 770 to 3,230 additional clicks, assuming the effect generalizes.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionProtect the comparison from calendar shocks
What if a festival, outage or campaign changes user behaviour during the test?
Concurrent control and treatment groups share time-varying external conditions, whereas a before-after comparison can confuse a calendar effect with a treatment effect.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionRepeat randomization when contexts differ
How can a festival effect be separated when events are cancelled or contexts change?
Run randomized comparisons within relevant contexts and compare their estimated lifts rather than treating unmatched groups as one experiment.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionThe same reasoning powers everyday experiments
What makes an A/B test successful across products?
Reliable experiments combine random assignment, a prespecified metric, a clearly estimated effect, quantified uncertainty, and cautious interpretation.
- Helpful foundationsfractions and percentages · probability · sample proportions
- Next connectionPower, sample size, and sequential testing