The design effect, and the one that came out below one — Survey Analysis, Sampling and Weighting — Data Analysis

What it is

The design effect is a ratio:

deff = variance under this design / variance under simple random sampling

A design effect of 2 means your estimate is as precise as a simple random sample half the size would have been. It converts a sample of 996 into an effective sample size — the number of independent observations your survey is actually worth.

effective n = n / deff

That second quantity is the one to put in a report. “996 households, effective sample 623” says more than either number alone, and it is the honest answer to “how big was your survey”.

Why clustering costs you

Households in the same enumeration area resemble each other. They share a water point, a market, a road, a harvest. So the second household you interview in an area tells you less than the first did, and the fourteenth tells you very little.

The measure of that resemblance is the intra-cluster correlation, and it maps to the clustering penalty directly:

deff_clustering = 1 + (m - 1) * ICC

with m the number of interviews per cluster.

import pandas as pd

by_area = (
    survey.assign(insecure=survey["food_insecure"] == "true")
    .groupby("ea_id")["insecure"]
    .agg(["mean", "size"])
)

p = (survey["food_insecure"] == "true").mean()
m = by_area["size"].mean()
between = by_area["mean"].var(ddof=0)
icc = (between - p * (1 - p) / m) / (p * (1 - p))

print(f"mean cluster size {m:.1f}, ICC {icc:.3f}, "
      f"clustering deff {1 + (m - 1) * icc:.2f}")

by_area <- survey |>
  summarise(rate = mean(food_insecure == "true"), n = n(), .by = ea_id)

p <- mean(survey$food_insecure == "true")
m <- mean(by_area$n)
icc <- (var(by_area$rate) * (nrow(by_area) - 1) / nrow(by_area) -
        p * (1 - p) / m) / (p * (1 - p))

c(m = m, icc = icc, deff = 1 + (m - 1) * icc)

On this survey: 13.3 interviews per area, ICC 0.11, clustering design effect about 2.4.

Two things about ICC worth carrying:

It is a property of the outcome, not of the survey. Water source is highly clustered — a village has a borehole or it does not. Household size is barely clustered at all. One design effect per outcome, always.
Published values are the best guess you have. DHS and SMART reports publish design effects, and using last round’s for the same indicator in the same place is far better than assuming 1.

Why stratification buys some back

Stratification does the opposite. By forcing the sample to cover each stratum, it removes the possibility of a sample that landed mostly in one — and that possibility is part of the variance of a simple random sample.

The gain is large exactly when strata differ a lot on the outcome. Improved water access runs 88% urban against 41% rural remote, and stratification guarantees those are represented in fixed proportions rather than by luck.

The net effect, measured

The two pull in opposite directions, so compute the design effect from the data rather than reasoning about it. Estimate the variance under the real design and divide by what simple random sampling would have given.

import numpy as np


def design_effect(df, outcome, weight="weight"):
    p = np.average(df[outcome] == "true", weights=df[weight])
    total_w = df[weight].sum()

    variance = 0.0
    for _, stratum in df.groupby("stratum"):
        areas = stratum.groupby("ea_id").apply(
            lambda g: (g[weight] * ((g[outcome] == "true") - p)).sum()
        )
        n = len(areas)
        if n < 2:
            continue
        variance += n / (n - 1) * ((areas - areas.mean()) ** 2).sum()
    variance /= total_w**2

    srs = p * (1 - p) / len(df)
    return p, np.sqrt(variance), variance / srs


for outcome in ["food_insecure", "improved_water_source"]:
    p, se, deff = design_effect(survey, outcome)
    print(f"{outcome:24} p={p:.1%} se={se:.4f} deff={deff:.2f} "
          f"n_eff={len(survey) / deff:.0f}")

library(survey)

design <- svydesign(ids = ~ea_id, strata = ~stratum, weights = ~weight,
                    data = survey, nest = TRUE)

svymean(~I(food_insecure == "true"), design, deff = TRUE)
svymean(~I(improved_water_source == "true"), design, deff = TRUE)

Outcome	Estimate	Design effect	Effective n
Food insecure	29.1%	1.60	623
Improved water source	69.8%	0.91	1,098

The one below one

Read the second row again. A design effect of 0.91 means the survey is more precise than a simple random sample of the same size.

That is not an error, and it is the most useful result in this lesson. Water access differs enormously between strata — 88%, 62%, 41% — and stratification guarantees all three are represented in the estimate in fixed proportions. The variance that a simple random sample carries from not knowing how many urban households it will happen to catch is removed entirely, and that gain outweighs the clustering loss.

Food insecurity also differs between strata, but its within-area correlation is higher, so clustering wins and the net is 1.60.

So “design effect” is not a synonym for “penalty”. It is the net of two opposing forces, and which one wins depends on the outcome. A survey report quoting one design effect for the whole survey has averaged something that should not be averaged.

Note also that the clustering-only figure computed earlier was 2.4, and the full design effect for the same outcome is 1.60. The difference is what stratification bought, and it is worth reporting when you are defending a design.

Where design effects come from when you have none

You need one before the survey exists, and there is no data yet. In order of preference:

The previous round of the same survey, same indicator, same area. Almost always available and almost always ignored.
A published survey report for the same indicator in a comparable setting. DHS reports tabulate design effects; SMART plausibility reports state them.
A convention. SMART surveys commonly assume 1.5 for anthropometry; humanitarian assessments often use 2.0 for household-level outcomes.
1.0 is never a defensible assumption for a cluster survey, and a protocol that uses it has under-sized the survey by half.

Report it

summary = pd.DataFrame({
    "outcome": ["Food insecure", "Improved water source"],
    "estimate": ["29.1%", "69.8%"],
    "ci_95": ["25.5-32.7%", "67.0-72.5%"],
    "deff": [1.60, 0.91],
    "n": [996, 996],
    "n_effective": [623, 1098],
})

tibble::tribble(
  ~outcome,                ~estimate, ~deff, ~n_effective,
  "Food insecure",           "29.1%",  1.60,          623,
  "Improved water source",   "69.8%",  0.91,         1098
)

Five columns, one row per indicator, and it belongs in the annex of every survey report. It lets a reader judge the estimate, compare it to another survey, and size the next round — none of which is possible from the percentage alone.

What comes next

You have a design effect, which means you have a standard error, which means you can stop computing these things by hand. The next lesson declares the design once to the software and lets every estimate inherit it — survey in R, and the equivalent arithmetic in Python.