Back to the lesson·Lesson 4 of 8·What the design buys
The design effect, and the one that came out below one
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- What it is
- Why clustering costs you
- Why stratification buys some back
- The net effect, measured
- The one below one
- Where design effects come from when you have none
- Report it
- What comes next
Speaker notes
Clustering costs precision, stratification buys it back, and the design effect is the net. 1.60 for food insecurity and 0.91 for water access, on the same 996 interviews.What it is — Example
deff = variance under this design / variance under simple random samplingSpeaker notes
The design effect is a ratio:What it is — Example
effective n = n / deffSpeaker notes
A design effect of 2 means your estimate is as precise as a simple random sample half the size would have been. It converts a sample of 996 into an effective sample size — the number of independent observations your survey is actually worth. That second quantity is the one to put in a report. "996 households, effective sample 623" says more than either number alone, and it is the honest answer to "how big was your survey".Why clustering costs you — Example
deff_clustering = 1 + (m - 1) * ICCSpeaker notes
Households in the same enumeration area resemble each other. They share a water point, a market, a road, a harvest. So the second household you interview in an area tells you less than the first did, and the fourteenth tells you very little. The measure of that resemblance is the intra-cluster correlation, and it maps to the clustering penalty directly:Why clustering costs you — In Python
import pandas as pd by_area = ( survey.assign(insecure=survey["food_insecure"] == "true") .groupby("ea_id")["insecure"] .agg(["mean", "size"]) ) p = (survey["food_insecure"] == "true").mean() m = by_area["size"].mean() between = by_area["mean"].var(ddof=0) icc = (between - p * (1 - p) / m) / (p * (1 - p)) print(f"mean cluster size {m:.1f}, ICC {icc:.3f}, " f"clustering deff {1 + (m - 1) * icc:.2f}")Speaker notes
withmthe number of interviews per cluster.Why clustering costs you — In R
by_area <- survey |> summarise(rate = mean(food_insecure == "true"), n = n(), .by = ea_id) p <- mean(survey$food_insecure == "true") m <- mean(by_area$n) icc <- (var(by_area$rate) * (nrow(by_area) - 1) / nrow(by_area) - p * (1 - p) / m) / (p * (1 - p)) c(m = m, icc = icc, deff = 1 + (m - 1) * icc)Why clustering costs you
- It is a property of the outcome, not of the survey. Water source is highly clustered — a village has a borehole or…
- Published values are the best guess you have. DHS and SMART reports publish design effects, and using last round's…
Speaker notes
On this survey: 13.3 interviews per area, ICC 0.11, clustering design effect about 2.4. Two things about ICC worth carrying:Why stratification buys some back
- Stratification does the opposite.
Speaker notes
Stratification does the opposite. By forcing the sample to cover each stratum, it removes the possibility of a sample that landed mostly in one — and that possibility is part of the variance of a simple random sample. The gain is large exactly when strata differ a lot on the outcome. Improved water access runs 88% urban against 41% rural remote, and stratification guarantees those are represented in fixed proportions rather than by luck.The net effect, measured — In Python (cont.)
import numpy as np def design_effect(df, outcome, weight="weight"): p = np.average(df[outcome] == "true", weights=df[weight]) total_w = df[weight].sum() variance = 0.0 for _, stratum in df.groupby("stratum"): areas = stratum.groupby("ea_id").apply( lambda g: (g[weight] * ((g[outcome] == "true") - p)).sum() ) n = len(areas) if n < 2: continue variance += n / (n - 1) * ((areas - areas.mean()) ** 2).sum()Speaker notes
The two pull in opposite directions, so compute the design effect from the data rather than reasoning about it. Estimate the variance under the real design and divide by what simple random sampling would have given.The net effect, measured — In Python (cont.)
variance /= total_w**2 srs = p * (1 - p) / len(df) return p, np.sqrt(variance), variance / srs for outcome in ["food_insecure", "improved_water_source"]: p, se, deff = design_effect(survey, outcome) print(f"{outcome:24} p={p:.1%} se={se:.4f} deff={deff:.2f} " f"n_eff={len(survey) / deff:.0f}")The net effect, measured — In R
library(survey) design <- svydesign(ids = ~ea_id, strata = ~stratum, weights = ~weight, data = survey, nest = TRUE) svymean(~I(food_insecure == "true"), design, deff = TRUE) svymean(~I(improved_water_source == "true"), design, deff = TRUE)The net effect, measured
Outcome Estimate Design effect Effective n Food insecure 29.1% 1.60 623 Improved water source 69.8% 0.91 1,098 The one below one
- So "design effect" is not a synonym for "penalty" — It is the net of two opposing forces, and which one wins depends on…
Speaker notes
Read the second row again. A design effect of 0.91 means the survey is more precise than a simple random sample of the same size. That is not an error, and it is the most useful result in this lesson. Water access differs enormously between strata — 88%, 62%, 41% — and stratification guarantees all three are represented in the estimate in fixed proportions. The variance that a simple random sample carries from not knowing how many urban households it will happen to catch is removed entirely, and that gain outweighs the clustering loss. Food insecurity also differs between strata, but its within-area correlation is higher, so clustering wins and the net is 1.60. So "design effect" is not a synonym for "penalty". It is the net of two opposing forces, and which one wins depends on the outcome. A survey report quoting one design effect for the whole survey has averaged something that should not be averaged. Note also that the clustering-only figure computed earlier was 2.4, and the full design effect for the same outcome is 1.60. The difference is what stratification bought, and it is worth reporting when you are defending a design.Where design effects come from when you have none
- The previous round of the same survey, same indicator, same area. Almost always available and almost always ignored.
- A published survey report for the same indicator in a comparable setting. DHS reports tabulate design effects;…
- A convention. SMART surveys commonly assume 1.5 for anthropometry; humanitarian assessments often use 2.0 for…
- 1.0 is never a defensible assumption for a cluster survey, and a protocol that uses it has under-sized the survey…
Speaker notes
You need one before the survey exists, and there is no data yet. In order of preference:Report it — In Python
summary = pd.DataFrame({ "outcome": ["Food insecure", "Improved water source"], "estimate": ["29.1%", "69.8%"], "ci_95": ["25.5-32.7%", "67.0-72.5%"], "deff": [1.60, 0.91], "n": [996, 996], "n_effective": [623, 1098], })Report it — In R
tibble::tribble( ~outcome, ~estimate, ~deff, ~n_effective, "Food insecure", "29.1%", 1.60, 623, "Improved water source", "69.8%", 0.91, 1098 )Speaker notes
Five columns, one row per indicator, and it belongs in the annex of every survey report. It lets a reader judge the estimate, compare it to another survey, and size the next round — none of which is possible from the percentage alone.What comes next
- You have a design effect, which means you have a standard error, which means you can stop computing these things by hand.
Speaker notes
You have a design effect, which means you have a standard error, which means you can stop computing these things by hand. The next lesson declares the design once to the software and lets every estimate inherit it —surveyin R, and the equivalent arithmetic in Python.