What precision did the budget buy?

The question is always asked too late

Sample size is decided in a proposal, months before anyone analyses anything, and usually by whoever is writing the budget. The analyst inherits it.

So this lesson runs in both directions. Forwards, to size a survey you are designing. Backwards, to work out what precision the survey you already have can support — which is the version you will need more often, and the one that decides what you are allowed to publish.

The formula, and the four numbers in it

For a proportion:

n = z^2 * p * (1 - p) / d^2 * deff

Four inputs, and each one is a decision somebody has to make.

• z — the confidence level. 1.96 for 95%. Almost never changed.
• p — the expected proportion. You do not know it, which is the point of the survey. Use the last round, a comparable area, or 0.5 if you genuinely have nothing, because 0.5 maximises the required sample and is therefore the safe guess.
• d — the precision you need, as an absolute margin. This is the number that is actually negotiated, and the one nobody writes down.
• deff — the design effect. The next lesson is entirely about it; for sizing, use the previous round’s value or 2.0 as a placeholder.

import math


def sample_size(p, d, deff=1.0, z=1.96, response=1.0):
    n = z**2 * p * (1 - p) / d**2 * deff
    return math.ceil(n / response)


print(sample_size(0.30, 0.05))            # simple random
print(sample_size(0.30, 0.05, deff=2.4))  # cluster survey
print(sample_size(0.30, 0.03, deff=2.4))  # tighter precision

sample_size <- function(p, d, deff = 1, z = 1.96, response = 1) {
  ceiling(z^2 * p * (1 - p) / d^2 * deff / response)
}

c(sample_size(0.30, 0.05),
  sample_size(0.30, 0.05, deff = 2.4),
  sample_size(0.30, 0.03, deff = 2.4))

Expected p	Precision	deff	Households needed
30%	±5 points	1.0	323
30%	±5 points	2.4	775
30%	±3 points	2.4	2,152

Three things to take from that table.

Clustering more than doubles the requirement. 323 becomes 775 for the same precision. Any sample size that does not mention a design effect is sizing a simple random sample, and nobody runs one of those in this sector.

Precision is brutally expensive. Going from ±5 to ±3 points nearly triples the sample. Precision costs with the square of the improvement, which is the single most useful fact in this lesson when somebody asks for a tighter figure.

Non-response is a divisor, not an afterthought. Expecting 90% response means dividing by 0.9 — and it must be applied to the households selected, not to the households interviewed, which is the same distinction the weighting lesson made.

Reading it backwards

You have 996 interviews and a design effect of 1.60. What precision does that buy?

def precision(n, p, deff=1.0, z=1.96):
    return z * math.sqrt(p * (1 - p) / n * deff)


print(f"+/- {precision(996, 0.291, deff=1.60):.1%}")
print(f"+/- {precision(343, 0.464, deff=1.60):.1%}")   # rural remote alone

precision <- function(n, p, deff = 1, z = 1.96) z * sqrt(p * (1 - p) / n * deff)

c(overall = precision(996, 0.291, deff = 1.60),
  remote  = precision(343, 0.464, deff = 1.60))

About ±3.6 points overall, and about ±5.3 points for rural remote on its own.

Do this before you promise anything. It tells you which comparisons the survey can settle and which it cannot, and it is the calculation behind the last lesson of this course.

The number that is really being negotiated

d is where the argument is, and it is usually conducted in the wrong currency — people argue about the sample size when the thing they disagree about is how precise the answer needs to be.

Reframe it. Precision is only meaningful against a decision:

• A threshold to cross. If the question is whether GAM exceeds 15%, and the estimate is near 14%, you need an interval narrow enough to sit on one side of the line. That is a precision requirement with a number in it.
• A change to detect. Detecting a five-point improvement between rounds needs roughly four times the sample of estimating a level to ±5 points, because two estimates each carry error.
• A comparison between groups. Same problem: two intervals, both of which have to be narrow.

# Detecting a difference between two groups of equal size
def sample_per_group(p1, p2, deff=1.0, power_z=0.84, z=1.96):
    pbar = (p1 + p2) / 2
    n = (z + power_z) ** 2 * 2 * pbar * (1 - pbar) / (p1 - p2) ** 2
    return math.ceil(n * deff)


print(sample_per_group(0.30, 0.25, deff=2.4))

sample_per_group <- function(p1, p2, deff = 1) {
  pbar <- (p1 + p2) / 2
  ceiling((1.96 + 0.84)^2 * 2 * pbar * (1 - pbar) / (p1 - p2)^2 * deff)
}

sample_per_group(0.30, 0.25, deff = 2.4)

Run it and the number is uncomfortable. Say the number out loud early, because a survey sized to estimate a level and then used to claim a change is the single commonest overreach in this sector’s reporting.

Clusters versus households per cluster

For a fixed budget you choose between more clusters and more households in each, and the two are not equivalent.

More clusters is almost always better, because the design effect grows with the number of households per cluster. Thirty clusters of ten beats ten clusters of thirty for the same 300 interviews, by a wide margin.

The counter-pressure is cost: a cluster costs a vehicle and a day, and a household costs an hour. SMART’s convention of about 30 clusters exists exactly at that trade-off, and it is a reasonable default when you have nothing better.

for m in [8, 14, 20, 30]:
    deff = 1 + (m - 1) * 0.11
    clusters = math.ceil(775 * (deff / 2.4) / m)
    print(f"{m:>3} households x {clusters:>3} clusters -> deff {deff:.2f}")

for (m in c(8, 14, 20, 30)) {
  deff <- 1 + (m - 1) * 0.11
  cat(sprintf("%3d households, deff %.2f\n", m, deff))
}

At an intra-cluster correlation of 0.11 — the value measured on this survey — fourteen households per cluster gives a clustering design effect of about 2.4, and thirty would give 4.2. The fourteen was a design decision, not an accident.

Write the calculation down

A sample size is an argument, and it belongs in the survey protocol with its inputs visible.

Indicator        Household food insecurity prevalence
Expected p       0.30 (2023 round, same instrument)
Precision        +/- 5 percentage points, 95% confidence
Design effect    2.0 (2023 round measured 1.9; rounded up)
Expected response 92%
Households       323 * 2.0 / 0.92 = 703, rounded to 25 clusters x 14 = 350 per
                 stratum, 1,050 in total across three strata
Rationale        Equal allocation, because each stratum must support its own
                 estimate. National figures require weighting.

Every number in that block is challengeable, which is the point. A protocol saying “1,050 households were surveyed” invites the question and cannot answer it.

What comes next

Two of the four inputs were the design effect, and it has been a placeholder so far. The next lesson measures it — where it comes from, how to compute it from your own data, and why one outcome in this survey has a design effect below one.