Back to the lesson·Lesson 8 of 8·What you may publish
Where to stop cutting
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- The request that always arrives
- What the cutting does
- Count clusters, not just households
- Set the rule before you look
- Suppress, but show the cell
- Domain estimation, and the thing that looks like a filter
- Plan the disaggregation before the survey
- The table to publish
- Where this course leaves you
Speaker notes
Three strata support an estimate. Stratum by displacement status gives nine cells with eighteen households in the smallest. A rule set before you look, and the table that shows a reader where the survey ran out.The request that always arrives
- The survey report goes out with three stratum estimates.
Speaker notes
The survey report goes out with three stratum estimates. Within a week somebody asks for it by displacement status. Then by sex of head of household. Then for displaced female-headed households in rural remote areas, because that is the group the next proposal is about. Each request is reasonable and the last one is unanswerable, and the difficulty is that nothing in the software refuses. Every cut produces a percentage.What the cutting does — In Python
for keys in [["stratum"], ["stratum", "displacement_status"], ["stratum", "main_livelihood"]]: cells = survey.groupby(keys).size() print(f"{' x '.join(keys):40} {len(cells):>3} cells, " f"smallest {cells.min():>3}, {(cells < 50).sum()} below 50")What the cutting does — In R
survey |> count(stratum) |> summarise(cells = n(), smallest = min(n)) survey |> count(stratum, displacement_status) |> summarise(cells = n(), smallest = min(n)) survey |> count(stratum, main_livelihood) |> summarise(cells = n(), smallest = min(n))What the cutting does
Disaggregation Cells Smallest Below 50 Stratum 3 316 0 Stratum × displacement 9 18 5 Stratum × livelihood 18 6 9 Speaker notes
One additional variable takes the smallest cell from 316 households to 18. A second takes it to 6. And the household count is the optimistic view, because these are clustered observations. Eighteen households in a cell may come from four enumeration areas, which is four independent units and about nine degrees of freedom — not eighteen.Count clusters, not just households — In Python
cells = survey.groupby(["stratum", "displacement_status"]).agg( households=("household_id", "size"), clusters=("ea_id", "nunique"), ) print(cells.sort_values("clusters").head())Count clusters, not just households — In R
survey |> summarise(households = n(), clusters = n_distinct(ea_id), .by = c(stratum, displacement_status)) |> arrange(clusters)Count clusters, not just households
- A cell drawn from fewer than about ten clusters cannot support a variance estimate you would want to defend — however…
Speaker notes
A cell drawn from fewer than about ten clusters cannot support a variance estimate you would want to defend, however many households are in it. This is the survey-specific version of the minimum cell size the indicator course introduced, and it is stricter, because clustering means the effective sample is smaller than the count. Two cells in this survey come from a single cluster. A variance estimate from one cluster is undefined, and most software will either drop the cell silently or return zero — a standard error of zero on a subgroup estimate is always a bug, and it is always this one.Set the rule before you look — In Python
MIN_HOUSEHOLDS = 50 MIN_CLUSTERS = 10 MAX_CI_WIDTH = 0.20 # 20 percentage points def reportable(cell): return (cell["households"] >= MIN_HOUSEHOLDS and cell["clusters"] >= MIN_CLUSTERS and cell["ci_width"] <= MAX_CI_WIDTH)Speaker notes
Three thresholds, decided in advance and written into the analysis plan.Set the rule before you look — In R
MIN_HOUSEHOLDS <- 50; MIN_CLUSTERS <- 10; MAX_CI_WIDTH <- 0.20 reportable <- function(d) { d$households >= MIN_HOUSEHOLDS & d$clusters >= MIN_CLUSTERS & d$ci_width <= MAX_CI_WIDTH }Speaker notes
The third condition is the one that does the real work, and it is better than the first two because it is about the answer rather than about the input. An interval wider than 20 points cannot distinguish "a serious problem" from "not a problem", so publishing the midpoint invites a decision the number cannot support. Where a threshold in the sector is what matters, set the width against the threshold instead: an interval that has to sit on one side of 15% needs to be narrower than the distance from the estimate to 15%.Suppress, but show the cell — In Python
by_cell = svyby_equivalent(survey, ["stratum", "displacement_status"], "food_insecure") by_cell["reported"] = by_cell.apply(reportable, axis=1) by_cell.loc[~by_cell["reported"], ["estimate", "ci_low", "ci_high"]] = None by_cell["note"] = by_cell["reported"].map( {False: "too few clusters to estimate", True: ""} ) print(by_cell)Suppress, but show the cell — In R
by_cell <- svyby(~I(food_insecure == "true"), ~stratum + displacement_status, design, svymean, vartype = "ci") |> mutate(reported = reportable(cur_data()), note = if_else(reported, "", "too few clusters to estimate"))Suppress, but show the cell
- Never delete the row — A suppressed cell that still shows its household count and a reason tells the reader the group…
Speaker notes
Never delete the row. A suppressed cell that still shows its household count and a reason tells the reader the group was surveyed and the survey could not answer for it. A deleted row reads as though the group does not exist, and the next person to ask will ask again. This is the same rule the DQA course applied to unmeasurable dimensions and the indicator course applied to small cells. It keeps arriving because it is the same underlying discipline: absence of evidence has to look different from absence.Domain estimation, and the thing that looks like a filter — In R
# WRONG: rebuilding the design from a filtered frame displaced <- svydesign(ids = ~ea_id, strata = ~stratum, weights = ~weight, data = filter(survey, displacement_status == "displaced"), nest = TRUE) # RIGHT: a domain estimate keeps the full design in view svyby(~I(food_insecure == "true"), ~displacement_status, design, svymean)Speaker notes
One technical point that changes the answer.Domain estimation, and the thing that looks like a filter — In Python
# The same rule: the variance calculation must still see every cluster, # including those with no displaced households in them.Speaker notes
A subgroup that does not span every cluster is a domain, not a sub-population with its own design. Estimating it from a filtered frame loses the clusters that contain none of the subgroup, and those empty clusters carry real information about the variance of the subgroup's size. The Rsurveypackage handles this correctly throughsvybyon the full design; a filtered rebuild does not. The effect is usually modest and always in the same direction: the filtered version understates the standard error, which is the direction that makes you overconfident.Plan the disaggregation before the survey
- Oversample the subgroup, which is what stratification exists for. This survey oversampled rural remote precisely so…
- Accept a wider interval for that subgroup, stated in advance.
- Drop the requirement, and say in the protocol that the survey will not report on it.
- What you cannot do is decide afterwards — By the time the data exists the cells are whatever they are, and a table…
Speaker notes
The honest fix for all of this is upstream. The indicator course made the point in general; here it has a number attached. If a subgroup is 12% of the population and you need ±10 points on it, the sample size formula run on that subgroup gives what you need — and it is usually far more than the survey was sized for. That is a design conversation, and the options are real: What you cannot do is decide afterwards. By the time the data exists the cells are whatever they are, and a table showing eight suppressed cells out of eighteen is the visible form of a design conversation that did not happen.The table to publish — In Python
final = by_cell[["stratum", "displacement_status", "households", "clusters", "estimate", "ci_low", "ci_high", "note"]] final.to_csv("outputs/tables/food_insecurity_by_stratum_displacement.csv", index=False)The table to publish — In R
readr::write_csv(by_cell, here::here("outputs", "tables", "food_insecurity_by_stratum_displacement.csv"))The table to publish
Stratum Status Households Clusters Estimate 95% CI Note Urban Resident 246 25 13.9% 10.2–18.7 Urban Displaced 49 18 — — too few households Rural remote Resident 279 25 46.1% 40.0–52.3 Rural remote Returnee 24 12 — — too few households Speaker notes
Seven columns, and the two empty ones are as informative as the four filled ones. A reader can see exactly where the survey ran out, which is the difference between a table that answers questions and one that generates them.Where this course leaves you
- The last course in the module, Routine Data and DHIS2, goes back to the
Speaker notes
You can reconstruct weights from a frame and prove they sum to the population, size a survey against a precision requirement and read that requirement backwards from a survey you inherited, measure a design effect rather than assuming one, estimate with the design carried through, handle non-response and replacement explicitly, publish an interval of the right shape, and say where the sample stops supporting a cut. That is the whole of what a survey analyst is judged on, and it is most of module aggregate reporting systems this programme has been borrowing from since module 2 — data elements, category combinations, the org unit hierarchy — and answers the question a coverage figure always provokes: what exactly was counted, and by whom?