cassionData Analysis

Back to the lessonLesson 7 of 8What you may publish

The interval, and the sentence around it

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 25

    What this lesson covers

    • What the interval is for
    • The textbook interval, and where it fails
    • The logit interval
    • Degrees of freedom you actually have
    • Overlapping intervals are not a test
    • Rounding, and false precision
    • The sentence
    • Reading an interval against a threshold
    • What comes next
    Speaker notes
    Why the textbook interval gives 0.88% for a proportion that cannot go below zero, the logit interval that fixes it, degrees of freedom you have 72 of, and why two overlapping intervals do not mean no difference.

  2. Slide 2 / 25

    What the interval is for

    • A survey estimate is a guess about a population, made from a sample.
    Speaker notes
    A survey estimate is a guess about a population, made from a sample. The confidence interval is the honest width of that guess, and publishing the point estimate without it is the single most common misuse of survey data in this sector. The formal reading — "95% of intervals constructed this way would contain the true value" — is correct and unhelpful in a meeting. The operational reading is what you need: the interval is the set of population values that would not be surprising given this sample. If a threshold sits inside your interval, your survey cannot say which side of it the population is on.

  3. Slide 3 / 25

    The textbook interval, and where it fails — Example

    p +/- z * se

  4. Slide 4 / 25

    The textbook interval, and where it fails — In Python

    p, se = 0.0217, 0.0066
print(f"Wald: {p - 1.96 * se:.2%} to {p + 1.96 * se:.2%}")
    Speaker notes
    Fine in the middle of the range and wrong at the ends. Take severe acute malnutrition among the measured children:

  5. Slide 5 / 25

    The textbook interval, and where it fails — In R

    p <- 0.0217; se <- 0.0066
c(p - 1.96 * se, p + 1.96 * se)
    Speaker notes
    2.17%, interval 0.88% to 3.46%. Symmetric, and the symmetry is the problem: a proportion cannot be negative, and with a slightly smaller estimate this interval would extend below zero and be published anyway.

  6. Slide 6 / 25

    The logit interval — In Python

    import math

logit = math.log(p / (1 - p))
se_logit = se / (p * (1 - p))
low, high = (logit - 1.96 * se_logit, logit + 1.96 * se_logit)

print(f"logit: {1 / (1 + math.exp(-low)):.2%} to {1 / (1 + math.exp(-high)):.2%}")
    Speaker notes
    Transform to the log-odds scale, build the interval there, transform back.

  7. Slide 7 / 25

    The logit interval — In R

    library(survey)
svyciprop(~I(child_muac_mm < 115), child_design, method = "logit")

  8. Slide 8 / 25

    The logit interval

    • 1.19% to 3.92% — Asymmetric — further above the estimate than below — which is the correct shape for a small…
    Speaker notes
    1.19% to 3.92%. Asymmetric — further above the estimate than below — which is the correct shape for a small proportion, and it cannot escape the 0–1 range whatever the estimate. Use the logit interval for any proportion below about 10% or above 90%. In the middle the two agree to a decimal place and it does not matter. svyciprop(..., method = "logit") in R does it directly; the Python examples in this course compute it as above.

  9. Slide 9 / 25

    Degrees of freedom you actually have — Example

    df = number of PSUs - number of strata
    Speaker notes
    The 1.96 assumes a normal distribution, which assumes plenty of independent observations. In a cluster survey the independent units are the clusters, not the households.

  10. Slide 10 / 25

    Degrees of freedom you actually have — In Python

    clusters = survey["ea_id"].nunique()
strata = survey["stratum"].nunique()
print(f"{clusters} clusters, {strata} strata, df = {clusters - strata}")

  11. Slide 11 / 25

    Degrees of freedom you actually have — In R

    c(clusters = n_distinct(survey$ea_id), strata = n_distinct(survey$stratum))
degf(design)

  12. Slide 12 / 25

    Degrees of freedom you actually have

    • Use the t multiplier, and report the degrees of freedom — It costs nothing and it tells a reader how many clusters you…
    Speaker notes
    Seventy-five clusters, three strata, 72 degrees of freedom, and t(0.975, 72) is 1.993 rather than 1.96. A difference of 2% in the interval width, which here is irrelevant and is exactly the kind of thing that stops being irrelevant fast: a survey with twelve clusters has nine degrees of freedom and a multiplier of 2.26, which is 15% wider than the normal approximation. Use the t multiplier, and report the degrees of freedom. It costs nothing and it tells a reader how many clusters you had without them having to ask.

  13. Slide 13 / 25

    Overlapping intervals are not a test — In Python

    for stratum in ["urban", "rural-accessible", "rural-remote"]:
    print(stratum)   # displaced households
    Speaker notes
    The most consequential misreading in this lesson.

  14. Slide 14 / 25

    Overlapping intervals are not a test

    StratumDisplaced95% interval
    Urban15.6%12.0 – 19.2%
    Rural accessible8.0%4.7 – 11.3%
    Rural remote15.7%11.0 – 20.4%

  15. Slide 15 / 25

    Overlapping intervals are not a test — In R

    svyttest(I(displacement_status == "displaced") ~ I(stratum == "urban"),
         subset(design, stratum != "rural-remote"))
    Speaker notes
    Urban and rural accessible intervals do not overlap, so those two differ. Urban and rural remote overlap almost entirely, so those two do not differ detectably. Now the trap: two intervals that overlap slightly can still be significantly different. Comparing intervals is a cruder test than comparing the difference, because the difference has its own standard error which is smaller than the sum of the two.

  16. Slide 16 / 25

    Overlapping intervals are not a test — In Python

    # The difference of two estimates, with its own standard error
diff = p_urban - p_rural
se_diff = math.sqrt(se_urban**2 + se_rural**2)
print(f"difference {diff:.1%}, 95% CI {diff - 1.99 * se_diff:.1%} to "
      f"{diff + 1.99 * se_diff:.1%}")
    Speaker notes
    The rule: to compare two estimates, estimate the difference and put an interval on that. Reading two intervals is a screening step, not a conclusion. Note also that the Python line above assumes the two estimates are independent, which is true for separate strata and false for two subgroups within a stratum — in that case the covariance term matters and svyttest or svycontrast handles it properly.

  17. Slide 17 / 25

    Rounding, and false precision — In Python

    print(f"{p:.1%} (95% CI {low:.1%} to {high:.1%})")
    Speaker notes
    An interval of ±3.6 points does not support three decimal places.

  18. Slide 18 / 25

    Rounding, and false precision — In R

    sprintf("%.1f%% (95%% CI %.1f-%.1f)", 100 * p, 100 * low, 100 * high)

  19. Slide 19 / 25

    Rounding, and false precision

    • One decimal place for a percentage from a survey of this size. Two implies a precision of 0.01 points, which is a…
    • Round the interval outwards, never inwards. 25.47–32.71 becomes 25.4–32.8.
    • Never round the point estimate to a threshold. An estimate of 14.96% reported as "15%" beside a 15% emergency…

  20. Slide 20 / 25

    The sentence

    Household food insecurity was estimated at 29.1% (95% CI 25.5–32.7), from 996 households in 75 clusters, weighted to the sampling frame and adjusted for non-response; design effect 1.60, effective sample 623.

  21. Slide 21 / 25

    The sentence

    Food insecurity affects 32.8% of households.
    Speaker notes
    The number, the interval, the denominator, the design. One sentence, and it is the deliverable of this whole course. Compare that with what usually gets published: The second sentence is unweighted, has no interval, no denominator and no design. It is also 3.7 points higher, and it is the one that will be quoted.

  22. Slide 22 / 25

    Reading an interval against a threshold — In Python

    THRESHOLD = 0.15

if high < THRESHOLD:
    verdict = "below the threshold"
elif low > THRESHOLD:
    verdict = "above the threshold"
else:
    verdict = "cannot be distinguished from the threshold"
print(verdict)
    Speaker notes
    The question this sector asks most often, and the one the interval exists for.

  23. Slide 23 / 25

    Reading an interval against a threshold — In R

    dplyr::case_when(
  high < 0.15 ~ "below the threshold",
  low  > 0.15 ~ "above the threshold",
  TRUE        ~ "cannot be distinguished from the threshold"
)
    Speaker notes
    The third branch is the one people delete, and it is the most common honest answer. An estimate of 14.2% with an interval of 11.0–18.1 does not say the situation is below the emergency threshold; it says the survey cannot tell. That sentence is unwelcome, correct, and considerably cheaper than a response scaled on a number that could not support the decision.

  24. Slide 24 / 25

    What comes next

    • The interval tells you what your whole sample supports.
    Speaker notes
    The interval tells you what your whole sample supports. Cut the sample into districts, age groups and sexes and each piece supports far less. The last lesson is where to stop cutting, and how to say so in a table.

  25. Slide 25 / 25

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson