Inclusion Probability
What is the mathematical foundation of inclusion probability?
How this lesson unfolds.
Begin with In survey sampling, what is truly random: the measured values of the population units, or the mechanism that selects which units enter the sample? Use the picture, intuition and explanation together before moving on.
In survey sampling, what is truly random: the measured values of the population units, or the mechanism that selects which units enter the sample?
The fish in a lake have fixed, unchanging weights. Randomness enters only through the net you cast: each fish has a specific probability piᵢ of being caught.
- $$\pi_i = P(i \in \mathcal{S}) = E[I_i] \implies \hat{Y}_{\mathrm{HT}} = \sum_{i \in \mathcal{S}} \frac{y_i}{\pi_i}, \quad E[\hat{Y}_{\mathrm{HT}}] = Y$$
- Learning goalDefine first-order inclusion probabilities and prove design-unbiasedness of Horvitz-Thompson estimators.
- Helpful foundationsIndicator expectation · Horvitz-Thompson estimator · Design-based vs model-based inference · Foundations of the Theory of Probability (Andrey Kolmogorov, 1933) · An Essay towards solving a Problem in the Doctrine of Chances (Thomas Bayes, 1763)
- Common trapAssuming survey units require equal selection probabilities; weighting by 1/piᵢ guarantees design-unbiased estimates.
- Next connectionBuild probability foundations from empirical sample averages.
More context & mechanics
A profound philosophical paradigm shift occurs in survey sampling. In standard classical statistics, data values yᵢ are modeled as random variables drawn from a distribution F. In design-based survey sampling (Neyman-Rubin framework), the population values y₁, y₂, ... y_N are fixed, deterministic constants! The only random quantity is the sample indicator vector: Iᵢ = 1 if unit i is selected into sample S, and Iᵢ = 0 otherwise. The inclusion probability of unit i is piᵢ = P(unit i in S) = E[Iᵢ]. The Horvitz-Thompson Estimator uses these inclusion probabilities to estimate the population total Y = sum_(i=1)^N yᵢ:
hatY_(HT) = sum_(i in S) (yᵢ / piᵢ) = sum_(i=1)^N Iᵢ * (yᵢ / piᵢ).
Evaluating its expectation: E[hatY_(HT)] = sum_(i=1)^N E[Iᵢ] (yᵢ / piᵢ) = sum_(i=1)^N piᵢ (yᵢ / piᵢ) = sum_(i=1)^N yᵢ = Y!
Because piᵢ cancels cleanly in expectation, the Horvitz-Thompson estimator is unconditionally design-unbiased for ANY sampling scheme where piᵢ > 0 for all units.
National census bureaus (e.g., US Census Bureau, Indian National Sample Survey Office - NSSO) and ecological wildlife monitoring (capture-recapture estimation) rely on inclusion probabilities to compute unbiased demographic and population totals.
Daniel Horvitz and Donovan Thompson revolutionized survey statistics in 1952 by establishing this estimator, proving that unequal probability sampling can yield perfectly unbiased population estimates when weighted by inverse inclusion probabilities.
'Divide by Inclusion to Remove Sampling Illusion' — weight each observed unit by 1/piᵢ to recover the population truth.