Back to the lesson·Lesson 1 of 8·The sample is not the population
The mean of your sample is not the mean of anything
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- Two numbers from one file
- Where the gap comes from
- Why anyone would design it that way
- The stratum estimates need no weights
- The direction is not always the same
- What to do when there is no frame
- What comes next
Speaker notes
32.8% against 29.1% on the same 996 interviews, and 62.8% against 69.8% on the same households. The gap is the design, and it is not noise.Two numbers from one file — In Python
import pandas as pd survey = pd.read_csv("household-survey-2025.v1.csv") frame = pd.read_csv("household-survey-frame-2025.v1.csv") unweighted = (survey["food_insecure"] == "true").mean() print(f"unweighted: {unweighted:.1%}")Speaker notes
Take the household survey, count the households classified food insecure, divide by the number of households. That is 32.8%. Now weight each household by the number of households it represents in the population, and the same file gives 29.1%.Two numbers from one file — In R
library(dplyr) library(readr) survey <- read_csv("household-survey-2025.v1.csv") frame <- read_csv("household-survey-frame-2025.v1.csv") mean(survey$food_insecure == "true")Speaker notes
Both numbers are computed correctly. One of them is an estimate of the population and the other is a description of who happened to be interviewed, and only one of those is what the report claims to be reporting.Where the gap comes from
Stratum Households in the frame Areas sampled Households interviewed Food insecure Urban 21,270 25 316 14.6% Rural accessible 28,958 25 337 36.2% Rural remote 6,200 25 343 46.4% Speaker notes
Nothing about the arithmetic. Everything about how the sample was drawn.Where the gap comes from — In Python
by_stratum = ( survey.assign(insecure=survey["food_insecure"] == "true") .groupby("stratum") .agg(interviews=("insecure", "size"), rate=("insecure", "mean")) ) frame_totals = frame.groupby("stratum")["households"].sum() comparison = by_stratum.join(frame_totals.rename("frame_households")) comparison["sample_share"] = comparison["interviews"] / comparison["interviews"].sum() comparison["population_share"] = ( comparison["frame_households"] / comparison["frame_households"].sum() ) print(comparison.round(3))Speaker notes
Read the first and last columns together. Rural remote is 11% of the population and 34% of the sample, and it has the worst food insecurity by a wide margin. An unweighted average of the three columns therefore over-represents the worst-off stratum by a factor of three. The 32.8% is not the district's food insecurity; it is the food insecurity of a population in which a third of households are remote rural, and no such population exists.Where the gap comes from — In R
survey |> summarise(interviews = n(), rate = mean(food_insecure == "true"), .by = stratum) |> left_join(summarise(frame, frame_households = sum(households), .by = stratum), by = "stratum") |> mutate(sample_share = interviews / sum(interviews), population_share = frame_households / sum(frame_households))Where the gap comes from
Stratum Sample share Population share Urban 31.7% 37.7% Rural accessible 33.8% 51.3% Rural remote 34.4% 11.0% Where the gap comes from
- When those two columns differ, an unweighted estimate is biased — and the size and direction of the bias are entirely…
Speaker notes
When those two columns differ, an unweighted estimate is biased, and the size and direction of the bias are entirely predictable from the table.Why anyone would design it that way
- Proportional allocation gives the best national estimate and weak sub-national ones.
- Equal allocation gives comparable sub-national estimates and requires weighting for anything national.
Speaker notes
The obvious reaction is that the sample was drawn badly. It was not, and knowing why is most of what this lesson is for. Equal allocation across unequal strata is a deliberate, standard choice, and it buys something specific: a usable estimate for each stratum separately. Rural remote holds 11% of households, so a sample proportional to population would have put about 110 interviews there — enough for a national figure and nowhere near enough to say anything about the stratum on its own. The trade is explicit: DHS, MICS and most humanitarian assessments choose the second, because the whole point of the survey is usually to compare places. The weights are not a correction for a mistake. They are the price of the design.The stratum estimates need no weights — In Python
print(by_stratum["rate"].round(3))Speaker notes
A useful thing falls out of the design and it is worth seeing early.The stratum estimates need no weights — In R
survey |> summarise(rate = mean(food_insecure == "true"), .by = stratum)Speaker notes
Within a stratum, this design is self-weighting — every household had the same probability of selection — so the unweighted stratum rate is already the estimate. Weights matter only when you combine strata. That is why a report can legitimately show unweighted stratum figures beside a weighted total, and why doing so without saying which is which confuses everyone.The direction is not always the same — In Python
for outcome in ["food_insecure", "improved_water_source"]: print(f"{outcome:24} unweighted {(survey[outcome] == 'true').mean():.1%}")Speaker notes
Food insecurity falls when you weight. Improved water access rises, and by more.The direction is not always the same — In R
survey |> summarise(across(c(food_insecure, improved_water_source), ~ mean(.x == "true")))The direction is not always the same
Outcome Unweighted Weighted Food insecure 32.8% 29.1% Improved water source 62.8% 69.8% The direction is not always the same
- So you cannot correct an unweighted figure by rule of thumb — and a report that presents one with a note saying…
Speaker notes
Seven points on the second one. There is no general rule that unweighted estimates are too high or too low: the bias goes in whichever direction the oversampled stratum differs, and it differs by a different amount for every outcome. So you cannot correct an unweighted figure by rule of thumb, and a report that presents one with a note saying "unweighted, so the true figure is somewhat lower" is guessing.What to do when there is no frame
- Report by stratum only, if the strata are recorded. Sub-national figures are usually what the audience wanted…
- Reconstruct approximate weights from an external population source, and document that they are approximate.
- Say the survey cannot support a population estimate. This is a real answer and it is better than a number that will…
Speaker notes
Sometimes you inherit a dataset with no frame and no weights. Three honest positions, and none of them is to report the unweighted mean as an estimate.What comes next
- The weights are computable, from the frame this dataset ships.
Speaker notes
The weights are computable, from the frame this dataset ships. The next lesson builds them from first principles — selection probability at each stage, base weight, non-response adjustment — and then proves them, by checking that they sum to the number of households the frame says exist.