Back to the lesson·Lesson 6 of 8·Estimating properly
The households you did not get
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- Non-response is not missing data
- The adjustment, and the assumption under it
- Adjust at the right level
- Replacement clusters
- Response rates belong in the report
- When response is bad enough to stop
- What comes next
Speaker notes
Fifty-four missing interviews concentrated in the stratum least like the others, two replacement clusters that change a probability nobody recomputes, and the assumption every adjustment rests on.Non-response is not missing data — In Python
selected = frame[frame["selected"] == True] response = selected.groupby("stratum").agg( selected=("households_selected", "sum"), interviewed=("households_interviewed", "sum"), ) response["rate"] = response["interviewed"] / response["selected"] print(response)Speaker notes
A missing value is a question that was not answered. Non-response is a household that was selected and never interviewed, and the difference matters because non-response leaves no row at all — the cleaning course's hardest case, arriving in a setting where the frame tells you exactly how many rows should be there.Non-response is not missing data — In R
frame |> filter(selected) |> summarise(selected = sum(households_selected), interviewed = sum(households_interviewed), .by = stratum) |> mutate(rate = interviewed / selected)Non-response is not missing data
Stratum Selected Interviewed Response Urban 350 316 90.3% Rural accessible 350 337 96.3% Rural remote 350 343 98.0% Speaker notes
996 of 1,050. A 95% overall response rate reads as excellent, and the useful fact is not the average — it is that the shortfall is concentrated in urban areas, which are also the least food insecure by twenty points. An unadjusted analysis therefore under-represents exactly the stratum that would pull the estimate down. The direction of the bias is predictable from that table alone, before any modelling.The adjustment, and the assumption under it — In Python
survey["nr_adjustment"] = 14 / survey["ea_id"].map( selected.set_index("ea_id")["households_interviewed"] ) print(survey.groupby("stratum")["nr_adjustment"].mean().round(3))Speaker notes
The adjustment from lesson 2 inflates each responding household's weight to carry the non-respondents in its own area.The adjustment, and the assumption under it — In R
survey |> summarise(mean_adjustment = mean(14 / households_interviewed), .by = stratum)The adjustment, and the assumption under it
- The assumption is that non-respondents resemble respondents in the same area — It is not free and it is not testable…
Speaker notes
The assumption is that non-respondents resemble respondents in the same area. It is not free and it is not testable from the survey, so it belongs in the limitations section in words:The adjustment, and the assumption under it
Non-response was adjusted within enumeration area. Urban response was 90.3% against 96–98% rural. If urban non-respondents are systematically better off than urban respondents — locked compounds and daytime absence both suggest they may be — the adjusted estimate remains slightly too high.
Speaker notes
Notice what that paragraph does. It names the direction of the residual bias after adjustment, which is the most a survey can honestly offer.Adjust at the right level
- Within area. What this survey does. Best when response varies between areas, which it always does.
- Within stratum. Cruder, and it throws away the information that one area had four refusals and its neighbour none.
- Within a response class. Group areas by something that predicts response — urban density, market day, security —…
Speaker notes
Three levels are available and they are not equivalent. The last one matters here: an area with very few completed interviews gets a large adjustment from a small denominator, and one such area can move a stratum estimate. Check for it.Adjust at the right level — In Python
weights_by_area = survey.groupby("ea_id")["weight"].first().sort_values() print(weights_by_area.tail(3)) print(f"largest weight / smallest: {weights_by_area.max() / weights_by_area.min():.1f}x")Adjust at the right level — In R
survey |> summarise(weight = first(weight), n = n(), .by = ea_id) |> arrange(desc(weight)) |> head(3)Adjust at the right level
- Trim or flag any weight far outside the rest — A weight three times its stratum's base is one area doing the work of…
Speaker notes
Trim or flag any weight far outside the rest. A weight three times its stratum's base is one area doing the work of three, and it inflates variance without adding information. Trimming is a judgement call with a cost — it introduces a small bias to remove a large variance — and like every other judgement in this course it goes in writing with its threshold.Replacement clusters — In Python
replacements = frame[frame["replacement_for"].notna()] print(replacements[["ea_id", "stratum", "households", "replacement_for"]])Speaker notes
Two rural remote areas could not be reached and were replaced. The frame records it.Replacement clusters — In R
frame |> filter(!is.na(replacement_for)) |> select(ea_id, stratum, households, replacement_for)Replacement clusters
- Treat the replacement as the original — The pragmatic choice, and what almost everyone does
- Treat the original as non-response at the cluster level — More honest and harder: the stratum effectively has 23…
Speaker notes
This is where a design quietly stops being the design that was documented, and there are two defensible treatments. Treat the replacement as the original. The pragmatic choice, and what almost everyone does. It assumes the replacement is exchangeable with the area it replaced — reasonable when replacement was random within stratum, which the protocol should say and often does not. Treat the original as non-response at the cluster level. More honest and harder: the stratum effectively has 23 responding clusters, not 25, and the weights for that stratum inflate accordingly.Replacement clusters — In Python
remote_areas = (frame["stratum"] == "rural-remote") & (frame["selected"] == True) responding = int(remote_areas.sum()) - len(replacements) print(f"25 selected, {responding} reached, cluster-level adjustment " f"{25 / responding:.3f}")Replacement clusters
Report replacement even when you treat it as exchangeable. "Two of 25 rural remote clusters were replaced for inaccessibility; both replacements were accessible areas, so the rural remote estimate is likely to be conservative" is a sentence a reader needs and cannot reconstruct.
Speaker notes
The difference on this survey is small. The reason to do it explicitly is not the size of the correction; it is that inaccessible areas are never a random subset. An area you cannot reach in February is remote, or insecure, or flooded, and those are precisely the conditions associated with the outcomes being measured. Replacing it with an accessible area systematically removes the worst cases.Response rates belong in the report — In Python
report = response.reset_index() report["non_response_bias_risk"] = ["high", "low", "low"] report["adjustment"] = "within enumeration area" print(report)Response rates belong in the report — In R
tibble::tribble( ~stratum, ~selected, ~interviewed, ~rate, ~risk, "urban", 350, 316, 0.903, "high", "rural-accessible", 350, 337, 0.963, "low", "rural-remote", 350, 343, 0.980, "low" )Speaker notes
Four columns and a risk assessment. A survey report that gives one overall response rate has hidden the only thing about non-response that affects the estimate — where it fell.When response is bad enough to stop
- Above 90% — adjust and report. Residual bias is usually small relative to sampling error.
- 80 to 90% — adjust, report, and describe the likely direction of residual bias. Do not compare against a previous…
- Below 80% — the adjustment is carrying too much. Report the estimate with a prominent caveat, or report by stratum…
- Below 60% in a stratum — that stratum's estimate is not usable and saying so is the correct finding.
Speaker notes
There is no universal threshold, and the honest guidance is comparative rather than absolute. The comparison that matters most is against the previous round. An estimate that moved five points between rounds, where response also moved fifteen points, has an obvious alternative explanation that must be addressed before any programmatic one.What comes next
- You have an estimate, a design and an honest account of what is missing from it.
Speaker notes
You have an estimate, a design and an honest account of what is missing from it. The next lesson turns that into the thing that gets published: an interval, of the right shape, with the right number of decimal places, and the sentence that goes around it.