Estimators from Scratch: Weights, Auxiliary Data, and Error Repair

Suppose an environmental agency inspects 4 out of 10 forest reserves and counts [18, 14, 20, 19] tigers, summing to 71. If an analyst reports 'There are 71 tigers in the region', they commit the fundamental sample sum fallacy. The remaining 6 unsampled forests are not barren; they simply were not visited. A sample sum is merely evidence about a subset, not an estimator of the whole. To extrapolate from 71 observed tigers to the true total Y, we must account for the sampling mechanism that selected these specific 4 forests.

Cybersecurity vulnerability scanners probe a 5% sample of company microservices and find 71 unpatched zero-days. Reporting 'there are 71 vulnerabilities in our company' dangerously ignores the other 95% of uninspected infrastructure.

Wildlife census pioneer C.G.J. Petersen in 1896 developed capture-recapture methods for European plaice fish, proving that observed counts must be mathematically scaled by capture probability to reflect true ecological populations.

Ecologist Axiom: "Unseen is Not Nonexistent." If you look in 4 out of 10 rooms and find 71 items, the house total is far higher than 71.

In nature, forests are not identical. A massive national park (Reserve A) is easy to access and houses large biodiversity, while a remote mountainous tract (Reserve D) requires helicopter transport. Rather than enforcing equal sampling rates that waste budget, researchers assign varying inclusion probabilities: π_A = 0.80, π_B = 0.60, π_C = 0.50, and π_D = 0.20. Because these probabilities are explicitly designed before entering the field, unequal inclusion does not introduce bias—provided our estimator correctly accounts for them.

Distributed telemetry pipelines at Netflix sample video streams with unequal probabilities: common high-bandwidth 4K sessions might be sampled at π = 0.01, while rare video failure sessions are sampled at π = 1.0 (deterministic logging).

P.C. Mahalanobis in India during the 1930s pioneered large-scale sample surveys with varying probabilities proportional to acreage for Bengal crop yield forecasting, establishing the Indian Statistical Institute.

Design Rule: "Unequal Chances, Equal Rigor." As long as every πᵢ > 0 and is known, unequal sampling is an intentional, controllable feature, not a bug.

If Reserve A has π_A = 0.80, we expect to observe it in 8 out of 10 surveys. It barely represents anyone other than itself (w_A = 1/0.8 = 1.25). But Reserve D was drawn with π_D = 0.20—only a 1-in-5 chance. When Reserve D actually appears in our sample, it must shoulder the burden of representing the 5 remote forests that had similar chances but were left unvisited. Inverting probability converts rarity into responsibility: w_D = 1/0.20 = 5.00. Each tiger seen in Reserve D represents 5 tigers in the regional total.

National consumer price index (CPI) calculations weight price quotes from rural grocery stores far higher than urban supermarkets, because thousands of rural markets are represented by just a few sampled locations.

Daniel G. Horvitz and Donovan J. Thompson introduced inverse probability weighting in 1952, proving it is the unique linear weighting that guarantees design-unbiasedness for any arbitrary probability sampling plan.

Mnemonic: "Rare Speaks for Many." wᵢ = 1/πᵢ. The smaller your chance of entering the room, the louder your voice must be to represent those left outside.

Let us compute the weighted impact of each sampled reserve: Reserve A's 18 tigers expand to 18 × 1.25 = 22.50; Reserve B's 14 tigers expand to 14 × 1.667 = 23.33; Reserve C's 20 tigers expand to 20 × 2.00 = 40.00; and Reserve D's 19 tigers expand to 19 × 5.00 = 95.00. Notice how Reserve D, despite having nearly the same raw count as Reserve A (19 vs 18), contributes over four times as much to the estimated total because it represents a large, rarely sampled stratum.

Ad-tech bidding engines scale observed ad conversions by inverse display probabilities: an ad conversion from a rarely targeted demographic is weighted up to estimate total market potential.

In 1952, Horvitz and Thompson presented their paper at the American Statistical Association, showing that without these inverse weights, government agricultural estimates in Iowa were biased by up to 30%.

Formula: "Count times Weight equals Representation." yᵢ^* = wᵢ yᵢ. Raw count is what you saw; weighted count is what exists in the world.

The Horvitz-Thompson estimator Ŷ_(HT) = ∑_(i ∈ S) (yᵢ) / (πᵢ) is the bedrock of modern survey statistics. By replacing the unobserved population sum with a weighted sample sum, it guarantees mathematical unbiasedness. In our 4-forest sample, summing the expanded contributions yields: Ŷ_(HT) = 22.50 + 23.33 + 40.00 + 95.00 = 180.83 tigers. Instead of the naive 71, our estimator accounts for the entire landscape.

US Census Bureau surveys (like the American Community Survey) use Horvitz-Thompson style weighting on millions of housing units to estimate national household income and poverty rates without universal censuses.

Horvitz and Thompson published their foundational paper in the Journal of the American Statistical Association in 1952, establishing the canonical estimator cited in tens of thousands of survey studies since.

The HT Formula Hook: "Sum of (Value over Pi)." Ŷ_(HT) = ∑ (yᵢ) / (πᵢ). Every sampled item is divided by its own ticket chance.

The proof of HT unbiasedness is one of the most elegant in statistics:

E[Ŷ_(HT)] = E[∑_(i=1)^N Iᵢ (yᵢ) / (πᵢ)] = ∑_(i=1)^N E[Iᵢ] (yᵢ) / (πᵢ) = ∑_(i=1)^N πᵢ ((yᵢ) / (πᵢ)) = ∑_(i=1)^N yᵢ = Y_(total)

The πᵢ in the numerator and denominator cancel perfectly! However, in our single sample, Reserve D appeared, carrying weight 5.0. If Reserve D happened to be omitted in another sample, the estimate would be lower. Unbiasedness guarantees that the target center is dead-on; it does not eliminate variance.

Financial risk models require unbiased loss estimators so that over thousands of banking trades, gains and losses do not suffer from systemic under-capitalization drift.

C.R. Rao in 1952 examined the admissibility of the Horvitz-Thompson estimator, proving that while it is always unbiased, its variance can be high if inclusion probabilities πᵢ are poorly matched to values yᵢ.

Mnemonic: "Unbiased is an Honest Archer." An honest archer hits the bullseye on average across 100 arrows, but individual arrows still land in a ring around the center.

While counting tigers requires dangerous, expensive ground expeditions, modern satellites map forest areas (xᵢ) instantaneously. We know the exact land area of every single one of the 10 forests, summing to X_(total) = 870 km². Because larger forests generally harbor more tigers, x and y are strongly positively correlated. If our sample contains an unusually small total forest area, we know our tiger count is likely an underestimate—and vice versa. Auxiliary data provides the anchor to correct sample flukes.

E-commerce advertising models lack direct purchase intent for every user, but have complete auxiliary browsing history and device metadata for 100% of the user base, using it to calibrate conversion predictions.

Karl Pearson and George Udny Yule in the late 1890s developed the theory of auxiliary correlation and multiple regression to exploit side-channel measurements in biological populations.

Analogy: "The Satellite and the Tiger." You cannot count tigers from space, but you can measure their trees. Use the easy trees to calibrate the elusive tigers.

Let us test our design weights on the forest areas: Sampled forests have areas [70, 55, 80, 62]. Applying the inverse weights: X̂_(HT) = (70) / (0.8) + (55) / (0.6) + (80) / (0.5) + (62) / (0.2) = 87.5 + 91.67 + 160.0 + 310.0 = 649.17 km². But the official satellite registry confirms the true total area of all 10 forests is X = 870 km²! Our sample's represented area (649.17) fell significantly short of reality (870). This reveals immediate intelligence: our sample drew forests that were smaller than average, meaning our tiger estimate Ŷ_(HT) is almost certainly an underestimate!

National election exit polls apply weights to respondents' reported age and gender, checking if the weighted sample matches census demographics. If young voters are under-represented, the poll recalibrates before releasing vote totals.

Swedish survey statisticians Carl-Erik Särndal and Bengt Swensson formalized calibration estimation in the 1980s, showing that estimators should be constrained to reproduce known auxiliary totals exactly.

Diagnostic Hook: "The Scale on Known Weights." If you weigh a known 10 kg dumbbell on your scale and it reads 8 kg, you know your scale is reading low today.

Since our sample represented only 649.17 km² out of the true 870 km², our sample captured only 649.17 / 870 ≈ 74.6\% of the physical forest habitat. The ratio correction factor is simply the inverse: K_R = (X) / (X̂_(HT)) = (870) / (649.17) ≈ 1.3402. This tells us that our sample under-covered the terrain by roughly 34\%. If tigers scale proportionally with territory, we should multiply our raw tiger estimate by 1.3402 to bring it up to proper physical scale.

Financial portfolio risk systems scale sampled stock volatility by market-wide index volatility ratios during extreme market turbulence, repairing sample lag.

John Graunt in 1662 pioneered ratio estimation in London by comparing registered parish christenings against known death records to estimate the population of London without a census.

Ratio Hook: "True over Estimated." K_R = X / X̂. If your sample only saw 3/4 of the map, multiply your answer by 4/3.

Multiplying our baseline estimate by the scale correction gives Ŷ_R = 180.83 × 1.3402 = 242.35 tigers. The ratio estimator implicitly assumes that y is proportional to x through the origin: yᵢ ≈ R xᵢ, where R = Y/X. When this proportional relationship is strong (high correlation ρ > 0.5 and the line passes near (0,0)), the ratio estimator dramatically reduces variance compared to Ŷ_(HT). The random fluctuations of sample selection cancel out in the ratio Ŷ_(HT) / X̂_(HT).

E-commerce warehouse inventory audits count physical items in 2% of storage bins, computing total stock via ratio estimation scaled by known total digital shelf volume.

William G. Cochran in 1940 published the definitive sampling theory for ratio and regression estimators, proving that when ρ > 1/2(CV(X)) / (CV(Y)), ratio estimation is guaranteed to have lower variance than simple expansion.

Formula: "HT times Truth over Guess." Ŷ_R = Ŷ × (X / X̂). Scale the tigers by how much the trees were undercounted.

Plotting our sampled forests (Area on horizontal axis, Tigers on vertical axis) reveals a clear upward trend: larger forests have more tigers. However, look closely at small reserves: even a small reserve of 20 km² has a baseline territorial pride of 5 tigers. There is an intercept β₀ > 0! Ratio estimation forces the regression line through the origin (0,0). When β₀ ≠ 0, forcing the line through (0,0) rotates the slope, distorting predictions for both very small and very large units. We need estimators that accommodate offsets.

Cloud compute billing models feature a fixed minimum base cost per cluster plus variable costs per vCPU-hour. Modeling cost as pure ratio through (0,0) severely misprices small test clusters.

Francis Galton in 1885 noted that child heights regress toward the mediocrity of the human mean rather than scaling as a pure multiple of parental height, establishing the necessity of regression intercepts.

Visual Check: "Check the Origin." If the cloud points to (0,0), use Ratio. If the cloud points to a non-zero intercept on the wall, use Regression or Difference.

Instead of asking 'By what percentage did we undercount area?', we can ask 'How many square kilometers of forest were missed?' The area gap is an exact additive quantity: Δ_X = X - X̂_(HT) = 870 - 649.17 = 220.83 km². If we know from historical wildlife biology that forest land in this region supports roughly B = 0.25 tigers per square kilometer, each uncounted square kilometer represents 0.25 missed tigers. We can simply add 220.83 × 0.25 = 55.21 tigers to our baseline estimate.

Supply chain inventory management adjusts warehouse stock additively: observed uncounted shelf space is multiplied by average item density per square meter to adjust estimated total stock.

Jerzy Neyman and later P.V. Sukhatme developed difference estimation in the 1930s-1940s, proving that if the constant B is pre-chosen independently of the sample, the difference estimator is strictly unbiased.

Additive Hook: "Gap times Rate equals Fix." Δ_Y = B × Δ_X. Find how much terrain was missed and multiply by the tiger density.

The proof of unbiasedness for the difference estimator is immediate from the linearity of expectation:

E[Ŷ_D] = E[Ŷ_(HT)] + B(X - E[X̂_(HT)]) = Y + B(X - X) = Y + 0 = Y

Because E[X̂_(HT)] = X, the correction term has an expectation of zero! No matter what constant B you choose, you can never introduce bias. If B is close to the true rate of change β, the variance of Ŷ_D shrinks dramatically: Var(Ŷ_D) = Var(Ŷ_(HT)) + B² Var(X̂_(HT)) - 2B Cov(Ŷ_(HT), X̂_(HT)). The covariance term subtracts away the majority of the sampling noise.

Telecom billing dispute audits estimate overcharges using difference estimators: baseline billed amounts are known (X), and auditors add sampled discrepancy adjustments.

Indian statistician P.V. Sukhatme published the variance properties of difference estimators in 1954, demonstrating its superiority in agricultural surveys where historical yield rates B were well established.

Formula Anchor: "Baseline + Rate × Error." Ŷ_D = Ŷ + B(X - X̂). It is Horvitz-Thompson plus an additive correction for the auxiliary territory you missed.

Rather than guessing a slope B, we let the sample tell us both the base intercept and the marginal rate. Using survey-weighted regression, we fit yᵢ = β₀ + β₁ xᵢ + εᵢ. In our tiger sample, the fitted line is ŷ = 1.82 + 0.242 x. The intercept β₀ = 1.82 represents the baseline tiger count in tiny pockets of woodland, while β₁ = 0.242 means each additional square kilometer yields approximately 0.242 additional tigers. We have captured both level and rate.

Real estate valuation platforms (Zillow Zestimate) use weighted regression on sampled home sales to predict property values across millions of unsampled homes using square footage and bedroom counts.

Carl Friedrich Gauss and Adrien-Marie Legendre developed least squares, but it was Ronald Fisher in 1925 who synthesized regression with sample survey variance estimation.

Regression Rule: "Level from Intercept, Rate from Slope." ŷ = β̂₀ + β̂₁ x. Never guess what the data can teach you directly.

This is the crowning achievement of modern survey statistics: the model-assisted paradigm. We take our fitted line ŷᵢ = β̂₀ + β̂₁ xᵢ and generate a prediction for all N=10 forests in the population (since area xᵢ is known for all 10). Summing these predictions gives the model baseline ∑_(i=1)^N ŷᵢ. But what if our model was slightly flawed? We calculate residuals eᵢ = yᵢ - ŷᵢ for our sampled units and expand them using design weights: ∑_(i ∈ S) wᵢ eᵢ. If the model overpredicted, the residuals are negative and adjust the estimate downward; if it underpredicted, they push it up.

Search engines estimate global ad revenue impact by running heavy predictive models across the entire advertiser graph, then applying weighted residual corrections from randomized A/B test holdouts.

Carl-Erik Särndal, Bengt Swensson, and Jan Wretman unified this paradigm in their 1992 masterpiece Model Assisted Survey Sampling, proving that models provide precision while design weights protect truth.

The Twin Engine Rule: "Model for Precision, Weights for Protection." The model predicts the whole world; the weighted residuals fix where the model was wrong.

Evaluating GREG for our wildlife census:

1. Population Predictions: Summing predictions ŷᵢ = 1.82 + 0.242 xᵢ over all 10 forests yields ∑_(i=1)^(10) ŷᵢ = 228.72 tigers.
2. Sample Residuals: On our 4 sampled forests, observed counts minus predictions yield e = [yᵢ - ŷᵢ]. Expanding by design weights: ∑_(i ∈ S) wᵢ eᵢ = +6.41 tigers.
3. GREG Total: Ŷ_(GREG) = 228.72 + 6.41 = 235.13 tigers.

GREG combines the best of all worlds: if the model is brilliant, residuals are near zero and variance collapses; if the model is poor, the design weights guarantee convergence to the true population total.

National statistical agencies (e.g., Statistics Canada, Eurostat, US Bureau of Labor Statistics) use GREG as the industry standard for publishing official GDP, employment, and agricultural statistics.

Cassell, Särndal, and Wretman introduced GREG in 1976, resolving decades of ideological warfare between purely design-based frequentists and purely model-based Bayesians.

GREG Formula Anchor: "Predict All, Repair Sample." Ŷ_(GREG) = ∑_(U) ŷᵢ + ∑_(S) wᵢ eᵢ. Add all predictions, then add the weighted residual errors.

We can organize all 5 estimators by what they repair:

1. Sample Sum (71): Naive count. Assumes sampled units are the whole universe.
2. Horvitz-Thompson (180.8): Weights by 1/πᵢ. Eliminates selection bias, but suffers high single-sample variance.
3. Ratio (242.4): Multiplies by X / X̂. Eliminates scale errors assuming proportionality through origin.
4. Difference (236.0): Adds B(X - X̂). Eliminates auxiliary errors additively with zero bias risk.
5. GREG (235.1): Fits optimal regression and repairs sample residuals. Minimizes variance across multiple auxiliary features.

Modern data science platforms (like Databricks and Snowflake) provide automated estimation libraries where engineers can toggle between HT, Ratio, and GREG with a single configuration flag.

Leslie Kish, author of Survey Sampling (1965), championed systematic taxonomies of survey estimators, emphasizing that estimator choice is an engineering decision balancing variance, bias, and operational cost.

The 5-Step Ladder: "Sum → Weight → Ratio → Difference → GREG." Each rung adds a smarter tool to eliminate noise without adding bias.

Never jump straight to a complex neural network or multivariate GREG model in production. High-stakes statistical systems are built as modular pipelines. First, compute the naive sample sum (sanity check). Second, apply inverse design weights Ŷ_(HT) (guarantees unbiasedness). Third, compute auxiliary check X̂_(HT) against known registry X (diagnoses representation). Fourth, if auxiliary correlation is proven, apply Ratio or GREG. If an advanced stage throws an exception, the pipeline gracefully falls back to Horvitz-Thompson.

Production machine learning systems at Stripe and Adyen use shadow pipelines: new experimental estimators run silently in parallel with proven Horvitz-Thompson baselines for months before taking over production billing.

Software engineering pioneer Fred Brooks in The Mythical Man-Month (1975) emphasized building modular systems with verifiable baseline interfaces—a lesson that transformed modern computational statistics.

Engineering Rule: "Crawl, Walk, Run, Fly." Raw sum (crawl) → HT (walk) → Ratio (run) → GREG (fly). Never fly without knowing how to walk.

Once you are in the field, true tiger total Y is unknown, so you can never know how close your estimate was. Therefore, you test your estimator before fielding the survey! Using historical census data or synthetic digital populations where Y_(true) is 100% known, a computer program draws M = 10,000 independent simulated samples using the planned sampling design. For each sample m, it computes Ŷ_(HT)^((m)), Ŷ_R^((m)), and Ŷ_(GREG)^((m)). By plotting the 10,000 results, we empirically verify that Bias is zero and confirm that GREG delivers the smallest Mean Squared Error (MSE).

Autonomous vehicle developers run billions of simulated miles in CARLA or NVIDIA DRIVE Sim to test trajectory estimators before releasing firmware to physical cars.

Enrico Fermi and Stanislaw Ulam in the 1940s established Monte Carlo simulation on early digital mainframes, establishing simulation as the third pillar of science alongside analytical theory and physical experimentation.

Flight Simulator Analogy: "Crash in the Simulator, Not with Passengers." Test your estimator across 10,000 computer simulations before trusting it with a real billion-dollar survey.

When government statistics determine Congressional seat allocations or billions in healthcare funding, 'black box' algorithms cannot be permitted. The GREG estimator is the gold standard of public policy because every dollar of adjustment is explainable: 'We started with 71 observed units. Design weights expanded this to 180.8 because remote regions were sampled at lower rates. Auxiliary satellite data revealed our sample was 220 square kilometers light on forest cover. Applying the empirical tiger-to-land rate of 0.242 added 54.3 tigers, yielding our final estimate of 235.1.' Every step is defensible in court, in parliament, and in peer review.

Financial regulators (like the SEC and Federal Reserve) legally require banks to use explainable risk models (SR 11-7 model risk management guidance), rejecting uninterpretable deep nets for capital reserve calculations.

William Edwards Deming championed operational definitions in the 1950s, stating that no measurement has meaning unless the precise method of measurement and mathematical adjustment is transparently documented.

The Audit Mantra: "Show Your Receipts." A great estimator doesn't just give an answer; it gives an itemized invoice of every correction it made along the way.