The interval, and the sentence around it

What the interval is for

A survey estimate is a guess about a population, made from a sample. The confidence interval is the honest width of that guess, and publishing the point estimate without it is the single most common misuse of survey data in this sector.

The formal reading — “95% of intervals constructed this way would contain the true value” — is correct and unhelpful in a meeting. The operational reading is what you need: the interval is the set of population values that would not be surprising given this sample. If a threshold sits inside your interval, your survey cannot say which side of it the population is on.

The textbook interval, and where it fails

p +/- z * se

Fine in the middle of the range and wrong at the ends. Take severe acute malnutrition among the measured children:

p, se = 0.0217, 0.0066
print(f"Wald: {p - 1.96 * se:.2%} to {p + 1.96 * se:.2%}")

p <- 0.0217; se <- 0.0066
c(p - 1.96 * se, p + 1.96 * se)

2.17%, interval 0.88% to 3.46%. Symmetric, and the symmetry is the problem: a proportion cannot be negative, and with a slightly smaller estimate this interval would extend below zero and be published anyway.

The logit interval

Transform to the log-odds scale, build the interval there, transform back.

import math

logit = math.log(p / (1 - p))
se_logit = se / (p * (1 - p))
low, high = (logit - 1.96 * se_logit, logit + 1.96 * se_logit)

print(f"logit: {1 / (1 + math.exp(-low)):.2%} to {1 / (1 + math.exp(-high)):.2%}")

library(survey)
svyciprop(~I(child_muac_mm < 115), child_design, method = "logit")

1.19% to 3.92%. Asymmetric — further above the estimate than below — which is the correct shape for a small proportion, and it cannot escape the 0–1 range whatever the estimate.

Use the logit interval for any proportion below about 10% or above 90%. In the middle the two agree to a decimal place and it does not matter. svyciprop(..., method = "logit") in R does it directly; the Python examples in this course compute it as above.

Degrees of freedom you actually have

The 1.96 assumes a normal distribution, which assumes plenty of independent observations. In a cluster survey the independent units are the clusters, not the households.

df = number of PSUs - number of strata

clusters = survey["ea_id"].nunique()
strata = survey["stratum"].nunique()
print(f"{clusters} clusters, {strata} strata, df = {clusters - strata}")

c(clusters = n_distinct(survey$ea_id), strata = n_distinct(survey$stratum))
degf(design)

Seventy-five clusters, three strata, 72 degrees of freedom, and t(0.975, 72) is 1.993 rather than 1.96. A difference of 2% in the interval width, which here is irrelevant and is exactly the kind of thing that stops being irrelevant fast: a survey with twelve clusters has nine degrees of freedom and a multiplier of 2.26, which is 15% wider than the normal approximation.

Use the t multiplier, and report the degrees of freedom. It costs nothing and it tells a reader how many clusters you had without them having to ask.

Overlapping intervals are not a test

The most consequential misreading in this lesson.

for stratum in ["urban", "rural-accessible", "rural-remote"]:
    print(stratum)   # displaced households

Stratum	Displaced	95% interval
Urban	15.6%	12.0 – 19.2%
Rural accessible	8.0%	4.7 – 11.3%
Rural remote	15.7%	11.0 – 20.4%

Urban and rural accessible intervals do not overlap, so those two differ. Urban and rural remote overlap almost entirely, so those two do not differ detectably.

Now the trap: two intervals that overlap slightly can still be significantly different. Comparing intervals is a cruder test than comparing the difference, because the difference has its own standard error which is smaller than the sum of the two.

svyttest(I(displacement_status == "displaced") ~ I(stratum == "urban"),
         subset(design, stratum != "rural-remote"))

# The difference of two estimates, with its own standard error
diff = p_urban - p_rural
se_diff = math.sqrt(se_urban**2 + se_rural**2)
print(f"difference {diff:.1%}, 95% CI {diff - 1.99 * se_diff:.1%} to "
      f"{diff + 1.99 * se_diff:.1%}")

The rule: to compare two estimates, estimate the difference and put an interval on that. Reading two intervals is a screening step, not a conclusion.

Note also that the Python line above assumes the two estimates are independent, which is true for separate strata and false for two subgroups within a stratum — in that case the covariance term matters and svyttest or svycontrast handles it properly.

Rounding, and false precision

An interval of ±3.6 points does not support three decimal places.

print(f"{p:.1%} (95% CI {low:.1%} to {high:.1%})")

sprintf("%.1f%% (95%% CI %.1f-%.1f)", 100 * p, 100 * low, 100 * high)

• One decimal place for a percentage from a survey of this size. Two implies a precision of 0.01 points, which is a hundred times narrower than your interval.
• Round the interval outwards, never inwards. 25.47–32.71 becomes 25.4–32.8.
• Never round the point estimate to a threshold. An estimate of 14.96% reported as “15%” beside a 15% emergency threshold has made a classification decision by rounding.

The sentence

The number, the interval, the denominator, the design. One sentence, and it is the deliverable of this whole course.

Household food insecurity was estimated at 29.1% (95% CI 25.5–32.7), from 996 households in 75 clusters, weighted to the sampling frame and adjusted for non-response; design effect 1.60, effective sample 623.

Compare that with what usually gets published:

Food insecurity affects 32.8% of households.

The second sentence is unweighted, has no interval, no denominator and no design. It is also 3.7 points higher, and it is the one that will be quoted.

Reading an interval against a threshold

The question this sector asks most often, and the one the interval exists for.

THRESHOLD = 0.15

if high < THRESHOLD:
    verdict = "below the threshold"
elif low > THRESHOLD:
    verdict = "above the threshold"
else:
    verdict = "cannot be distinguished from the threshold"
print(verdict)

dplyr::case_when(
  high < 0.15 ~ "below the threshold",
  low  > 0.15 ~ "above the threshold",
  TRUE        ~ "cannot be distinguished from the threshold"
)

The third branch is the one people delete, and it is the most common honest answer. An estimate of 14.2% with an interval of 11.0–18.1 does not say the situation is below the emergency threshold; it says the survey cannot tell. That sentence is unwelcome, correct, and considerably cheaper than a response scaled on a number that could not support the decision.

What comes next

The interval tells you what your whole sample supports. Cut the sample into districts, age groups and sexes and each piece supports far less. The last lesson is where to stop cutting, and how to say so in a table.