The households you did not get

Non-response is not missing data

A missing value is a question that was not answered. Non-response is a household that was selected and never interviewed, and the difference matters because non-response leaves no row at all — the cleaning course’s hardest case, arriving in a setting where the frame tells you exactly how many rows should be there.

selected = frame[frame["selected"] == True]

response = selected.groupby("stratum").agg(
    selected=("households_selected", "sum"),
    interviewed=("households_interviewed", "sum"),
)
response["rate"] = response["interviewed"] / response["selected"]
print(response)

frame |>
  filter(selected) |>
  summarise(selected = sum(households_selected),
            interviewed = sum(households_interviewed), .by = stratum) |>
  mutate(rate = interviewed / selected)

Stratum	Selected	Interviewed	Response
Urban	350	316	90.3%
Rural accessible	350	337	96.3%
Rural remote	350	343	98.0%

996 of 1,050. A 95% overall response rate reads as excellent, and the useful fact is not the average — it is that the shortfall is concentrated in urban areas, which are also the least food insecure by twenty points.

An unadjusted analysis therefore under-represents exactly the stratum that would pull the estimate down. The direction of the bias is predictable from that table alone, before any modelling.

The adjustment, and the assumption under it

The adjustment from lesson 2 inflates each responding household’s weight to carry the non-respondents in its own area.

survey["nr_adjustment"] = 14 / survey["ea_id"].map(
    selected.set_index("ea_id")["households_interviewed"]
)
print(survey.groupby("stratum")["nr_adjustment"].mean().round(3))

survey |> summarise(mean_adjustment = mean(14 / households_interviewed), .by = stratum)

The assumption is that non-respondents resemble respondents in the same area. It is not free and it is not testable from the survey, so it belongs in the limitations section in words:

Non-response was adjusted within enumeration area. Urban response was 90.3% against 96–98% rural. If urban non-respondents are systematically better off than urban respondents — locked compounds and daytime absence both suggest they may be — the adjusted estimate remains slightly too high.

Notice what that paragraph does. It names the direction of the residual bias after adjustment, which is the most a survey can honestly offer.

Adjust at the right level

Three levels are available and they are not equivalent.

• Within area. What this survey does. Best when response varies between areas, which it always does.
• Within stratum. Cruder, and it throws away the information that one area had four refusals and its neighbour none.
• Within a response class. Group areas by something that predicts response — urban density, market day, security — and adjust within the group. Better than stratum where areas are small, because an area with three interviews gives an unstable adjustment factor.

The last one matters here: an area with very few completed interviews gets a large adjustment from a small denominator, and one such area can move a stratum estimate. Check for it.

weights_by_area = survey.groupby("ea_id")["weight"].first().sort_values()
print(weights_by_area.tail(3))
print(f"largest weight / smallest: {weights_by_area.max() / weights_by_area.min():.1f}x")

survey |>
  summarise(weight = first(weight), n = n(), .by = ea_id) |>
  arrange(desc(weight)) |>
  head(3)

Trim or flag any weight far outside the rest. A weight three times its stratum’s base is one area doing the work of three, and it inflates variance without adding information. Trimming is a judgement call with a cost — it introduces a small bias to remove a large variance — and like every other judgement in this course it goes in writing with its threshold.

Replacement clusters

Two rural remote areas could not be reached and were replaced. The frame records it.

replacements = frame[frame["replacement_for"].notna()]
print(replacements[["ea_id", "stratum", "households", "replacement_for"]])

frame |> filter(!is.na(replacement_for)) |>
  select(ea_id, stratum, households, replacement_for)

This is where a design quietly stops being the design that was documented, and there are two defensible treatments.

Treat the replacement as the original. The pragmatic choice, and what almost everyone does. It assumes the replacement is exchangeable with the area it replaced — reasonable when replacement was random within stratum, which the protocol should say and often does not.

Treat the original as non-response at the cluster level. More honest and harder: the stratum effectively has 23 responding clusters, not 25, and the weights for that stratum inflate accordingly.

remote_areas = (frame["stratum"] == "rural-remote") & (frame["selected"] == True)
responding = int(remote_areas.sum()) - len(replacements)
print(f"25 selected, {responding} reached, cluster-level adjustment "
      f"{25 / responding:.3f}")

c(selected = 25, reached = 23, adjustment = 25 / 23)

The difference on this survey is small. The reason to do it explicitly is not the size of the correction; it is that inaccessible areas are never a random subset. An area you cannot reach in February is remote, or insecure, or flooded, and those are precisely the conditions associated with the outcomes being measured. Replacing it with an accessible area systematically removes the worst cases.

Report replacement even when you treat it as exchangeable. “Two of 25 rural remote clusters were replaced for inaccessibility; both replacements were accessible areas, so the rural remote estimate is likely to be conservative” is a sentence a reader needs and cannot reconstruct.

Response rates belong in the report

report = response.reset_index()
report["non_response_bias_risk"] = ["high", "low", "low"]
report["adjustment"] = "within enumeration area"
print(report)

tibble::tribble(
  ~stratum,            ~selected, ~interviewed, ~rate,  ~risk,
  "urban",                   350,          316, 0.903, "high",
  "rural-accessible",        350,          337, 0.963, "low",
  "rural-remote",            350,          343, 0.980, "low"
)

Four columns and a risk assessment. A survey report that gives one overall response rate has hidden the only thing about non-response that affects the estimate — where it fell.

When response is bad enough to stop

There is no universal threshold, and the honest guidance is comparative rather than absolute.

• Above 90% — adjust and report. Residual bias is usually small relative to sampling error.
• 80 to 90% — adjust, report, and describe the likely direction of residual bias. Do not compare against a previous round with a very different response rate without saying so.
• Below 80% — the adjustment is carrying too much. Report the estimate with a prominent caveat, or report by stratum only where response was adequate.
• Below 60% in a stratum — that stratum’s estimate is not usable and saying so is the correct finding.

The comparison that matters most is against the previous round. An estimate that moved five points between rounds, where response also moved fifteen points, has an obvious alternative explanation that must be addressed before any programmatic one.

What comes next

You have an estimate, a design and an honest account of what is missing from it. The next lesson turns that into the thing that gets published: an interval, of the right shape, with the right number of decimal places, and the sentence that goes around it.