cassionData Analysis

Back to the lessonLesson 3 of 8What the design buys

What precision did the budget buy?

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 19

    What this lesson covers

    • The question is always asked too late
    • The formula, and the four numbers in it
    • Reading it backwards
    • The number that is really being negotiated
    • Clusters versus households per cluster
    • Write the calculation down
    • What comes next
    Speaker notes
    The sample size formula, the four numbers it needs, and why the answer for a cluster survey is roughly double the textbook one. Plus reading it backwards, which is what you usually have to do.

  2. Slide 2 / 19

    The question is always asked too late

    • Sample size is decided in a proposal, months before anyone analyses anything, and usually by whoever is writing the budget.
    Speaker notes
    Sample size is decided in a proposal, months before anyone analyses anything, and usually by whoever is writing the budget. The analyst inherits it. So this lesson runs in both directions. Forwards, to size a survey you are designing. Backwards, to work out what precision the survey you already have can support — which is the version you will need more often, and the one that decides what you are allowed to publish.

  3. Slide 3 / 19

    The formula, and the four numbers in it — Example

    n = z^2 * p * (1 - p) / d^2 * deff
    Speaker notes
    For a proportion:

  4. Slide 4 / 19

    The formula, and the four numbers in it

    • z — the confidence level. 1.96 for 95%. Almost never changed.
    • p — the expected proportion. You do not know it, which is the point of the survey. Use the last round, a…
    • d — the precision you need, as an absolute margin. This is the number that is actually negotiated, and the one…
    • deff — the design effect. The next lesson is entirely about it; for sizing, use the previous round's value or 2.0…
    Speaker notes
    Four inputs, and each one is a decision somebody has to make.

  5. Slide 5 / 19

    The formula, and the four numbers in it — In Python

    import math


def sample_size(p, d, deff=1.0, z=1.96, response=1.0):
    n = z**2 * p * (1 - p) / d**2 * deff
    return math.ceil(n / response)


print(sample_size(0.30, 0.05))            # simple random
print(sample_size(0.30, 0.05, deff=2.4))  # cluster survey
print(sample_size(0.30, 0.03, deff=2.4))  # tighter precision

  6. Slide 6 / 19

    The formula, and the four numbers in it — In R

    sample_size <- function(p, d, deff = 1, z = 1.96, response = 1) {
  ceiling(z^2 * p * (1 - p) / d^2 * deff / response)
}

c(sample_size(0.30, 0.05),
  sample_size(0.30, 0.05, deff = 2.4),
  sample_size(0.30, 0.03, deff = 2.4))

  7. Slide 7 / 19

    The formula, and the four numbers in it

    Expected pPrecisiondeffHouseholds needed
    30%±5 points1.0323
    30%±5 points2.4775
    30%±3 points2.42,152

  8. Slide 8 / 19

    The formula, and the four numbers in it

    • Clustering more than doubles the requirement — 323 becomes 775 for the same precision
    • Precision is brutally expensive — Going from ±5 to ±3 points nearly triples the sample
    • Non-response is a divisor, not an afterthought — Expecting 90% response means dividing by 0.9 — and it must be applied…
    Speaker notes
    Three things to take from that table. Clustering more than doubles the requirement. 323 becomes 775 for the same precision. Any sample size that does not mention a design effect is sizing a simple random sample, and nobody runs one of those in this sector. Precision is brutally expensive. Going from ±5 to ±3 points nearly triples the sample. Precision costs with the square of the improvement, which is the single most useful fact in this lesson when somebody asks for a tighter figure. Non-response is a divisor, not an afterthought. Expecting 90% response means dividing by 0.9 — and it must be applied to the households selected, not to the households interviewed, which is the same distinction the weighting lesson made.

  9. Slide 9 / 19

    Reading it backwards — In Python

    def precision(n, p, deff=1.0, z=1.96):
    return z * math.sqrt(p * (1 - p) / n * deff)


print(f"+/- {precision(996, 0.291, deff=1.60):.1%}")
print(f"+/- {precision(343, 0.464, deff=1.60):.1%}")   # rural remote alone
    Speaker notes
    You have 996 interviews and a design effect of 1.60. What precision does that buy?

  10. Slide 10 / 19

    Reading it backwards — In R

    precision <- function(n, p, deff = 1, z = 1.96) z * sqrt(p * (1 - p) / n * deff)

c(overall = precision(996, 0.291, deff = 1.60),
  remote  = precision(343, 0.464, deff = 1.60))

  11. Slide 11 / 19

    Reading it backwards

    • Do this before you promise anything — It tells you which comparisons the survey can settle and which it cannot, and it…
    Speaker notes
    About ±3.6 points overall, and about ±5.3 points for rural remote on its own. Do this before you promise anything. It tells you which comparisons the survey can settle and which it cannot, and it is the calculation behind the last lesson of this course.

  12. Slide 12 / 19

    The number that is really being negotiated

    • A threshold to cross. If the question is whether GAM exceeds 15%, and the estimate is near 14%, you need an…
    • A change to detect. Detecting a five-point improvement between rounds needs roughly four times the sample of…
    • A comparison between groups. Same problem: two intervals, both of which have to be narrow.
    Speaker notes
    d is where the argument is, and it is usually conducted in the wrong currency — people argue about the sample size when the thing they disagree about is how precise the answer needs to be. Reframe it. Precision is only meaningful against a decision:

  13. Slide 13 / 19

    The number that is really being negotiated — In Python

    # Detecting a difference between two groups of equal size
def sample_per_group(p1, p2, deff=1.0, power_z=0.84, z=1.96):
    pbar = (p1 + p2) / 2
    n = (z + power_z) ** 2 * 2 * pbar * (1 - pbar) / (p1 - p2) ** 2
    return math.ceil(n * deff)


print(sample_per_group(0.30, 0.25, deff=2.4))

  14. Slide 14 / 19

    The number that is really being negotiated — In R

    sample_per_group <- function(p1, p2, deff = 1) {
  pbar <- (p1 + p2) / 2
  ceiling((1.96 + 0.84)^2 * 2 * pbar * (1 - pbar) / (p1 - p2)^2 * deff)
}

sample_per_group(0.30, 0.25, deff = 2.4)
    Speaker notes
    Run it and the number is uncomfortable. Say the number out loud early, because a survey sized to estimate a level and then used to claim a change is the single commonest overreach in this sector's reporting.

  15. Slide 15 / 19

    Clusters versus households per cluster — In Python

    for m in [8, 14, 20, 30]:
    deff = 1 + (m - 1) * 0.11
    clusters = math.ceil(775 * (deff / 2.4) / m)
    print(f"{m:>3} households x {clusters:>3} clusters -> deff {deff:.2f}")
    Speaker notes
    For a fixed budget you choose between more clusters and more households in each, and the two are not equivalent. More clusters is almost always better, because the design effect grows with the number of households per cluster. Thirty clusters of ten beats ten clusters of thirty for the same 300 interviews, by a wide margin. The counter-pressure is cost: a cluster costs a vehicle and a day, and a household costs an hour. SMART's convention of about 30 clusters exists exactly at that trade-off, and it is a reasonable default when you have nothing better.

  16. Slide 16 / 19

    Clusters versus households per cluster — In R

    for (m in c(8, 14, 20, 30)) {
  deff <- 1 + (m - 1) * 0.11
  cat(sprintf("%3d households, deff %.2f\n", m, deff))
}
    Speaker notes
    At an intra-cluster correlation of 0.11 — the value measured on this survey — fourteen households per cluster gives a clustering design effect of about 2.4, and thirty would give 4.2. The fourteen was a design decision, not an accident.

  17. Slide 17 / 19

    Write the calculation down — Example

    Indicator        Household food insecurity prevalence
Expected p       0.30 (2023 round, same instrument)
Precision        +/- 5 percentage points, 95% confidence
Design effect    2.0 (2023 round measured 1.9; rounded up)
Expected response 92%
Households       323 * 2.0 / 0.92 = 703, rounded to 25 clusters x 14 = 350 per
                 stratum, 1,050 in total across three strata
Rationale        Equal allocation, because each stratum must support its own
                 estimate. National figures require weighting.
    Speaker notes
    A sample size is an argument, and it belongs in the survey protocol with its inputs visible. Every number in that block is challengeable, which is the point. A protocol saying "1,050 households were surveyed" invites the question and cannot answer it.

  18. Slide 18 / 19

    What comes next

    • Two of the four inputs were the design effect, and it has been a placeholder so far.
    Speaker notes
    Two of the four inputs were the design effect, and it has been a placeholder so far. The next lesson measures it — where it comes from, how to compute it from your own data, and why one outcome in this survey has a design effect below one.

  19. Slide 19 / 19

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson