Where to stop cutting

The request that always arrives

The survey report goes out with three stratum estimates. Within a week somebody asks for it by displacement status. Then by sex of head of household. Then for displaced female-headed households in rural remote areas, because that is the group the next proposal is about.

Each request is reasonable and the last one is unanswerable, and the difficulty is that nothing in the software refuses. Every cut produces a percentage.

What the cutting does

for keys in [["stratum"],
             ["stratum", "displacement_status"],
             ["stratum", "main_livelihood"]]:
    cells = survey.groupby(keys).size()
    print(f"{' x '.join(keys):40} {len(cells):>3} cells, "
          f"smallest {cells.min():>3}, {(cells < 50).sum()} below 50")

survey |> count(stratum) |> summarise(cells = n(), smallest = min(n))
survey |> count(stratum, displacement_status) |> summarise(cells = n(), smallest = min(n))
survey |> count(stratum, main_livelihood) |> summarise(cells = n(), smallest = min(n))

Disaggregation	Cells	Smallest	Below 50
Stratum	3	316	0
Stratum × displacement	9	18	5
Stratum × livelihood	18	6	9

One additional variable takes the smallest cell from 316 households to 18. A second takes it to 6.

And the household count is the optimistic view, because these are clustered observations. Eighteen households in a cell may come from four enumeration areas, which is four independent units and about nine degrees of freedom — not eighteen.

Count clusters, not just households

cells = survey.groupby(["stratum", "displacement_status"]).agg(
    households=("household_id", "size"),
    clusters=("ea_id", "nunique"),
)
print(cells.sort_values("clusters").head())

survey |>
  summarise(households = n(), clusters = n_distinct(ea_id),
            .by = c(stratum, displacement_status)) |>
  arrange(clusters)

A cell drawn from fewer than about ten clusters cannot support a variance estimate you would want to defend, however many households are in it. This is the survey-specific version of the minimum cell size the indicator course introduced, and it is stricter, because clustering means the effective sample is smaller than the count.

Two cells in this survey come from a single cluster. A variance estimate from one cluster is undefined, and most software will either drop the cell silently or return zero — a standard error of zero on a subgroup estimate is always a bug, and it is always this one.

Set the rule before you look

Three thresholds, decided in advance and written into the analysis plan.

MIN_HOUSEHOLDS = 50
MIN_CLUSTERS = 10
MAX_CI_WIDTH = 0.20      # 20 percentage points

def reportable(cell):
    return (cell["households"] >= MIN_HOUSEHOLDS
            and cell["clusters"] >= MIN_CLUSTERS
            and cell["ci_width"] <= MAX_CI_WIDTH)

MIN_HOUSEHOLDS <- 50; MIN_CLUSTERS <- 10; MAX_CI_WIDTH <- 0.20

reportable <- function(d) {
  d$households >= MIN_HOUSEHOLDS & d$clusters >= MIN_CLUSTERS &
    d$ci_width <= MAX_CI_WIDTH
}

The third condition is the one that does the real work, and it is better than the first two because it is about the answer rather than about the input. An interval wider than 20 points cannot distinguish “a serious problem” from “not a problem”, so publishing the midpoint invites a decision the number cannot support.

Where a threshold in the sector is what matters, set the width against the threshold instead: an interval that has to sit on one side of 15% needs to be narrower than the distance from the estimate to 15%.

Suppress, but show the cell

by_cell = svyby_equivalent(survey, ["stratum", "displacement_status"], "food_insecure")
by_cell["reported"] = by_cell.apply(reportable, axis=1)
by_cell.loc[~by_cell["reported"], ["estimate", "ci_low", "ci_high"]] = None
by_cell["note"] = by_cell["reported"].map(
    {False: "too few clusters to estimate", True: ""}
)
print(by_cell)

by_cell <- svyby(~I(food_insecure == "true"), ~stratum + displacement_status,
                 design, svymean, vartype = "ci") |>
  mutate(reported = reportable(cur_data()),
         note = if_else(reported, "", "too few clusters to estimate"))

Never delete the row. A suppressed cell that still shows its household count and a reason tells the reader the group was surveyed and the survey could not answer for it. A deleted row reads as though the group does not exist, and the next person to ask will ask again.

This is the same rule the DQA course applied to unmeasurable dimensions and the indicator course applied to small cells. It keeps arriving because it is the same underlying discipline: absence of evidence has to look different from absence.

Domain estimation, and the thing that looks like a filter

One technical point that changes the answer.

# WRONG: rebuilding the design from a filtered frame
displaced <- svydesign(ids = ~ea_id, strata = ~stratum, weights = ~weight,
                       data = filter(survey, displacement_status == "displaced"),
                       nest = TRUE)

# RIGHT: a domain estimate keeps the full design in view
svyby(~I(food_insecure == "true"), ~displacement_status, design, svymean)

# The same rule: the variance calculation must still see every cluster,
# including those with no displaced households in them.

A subgroup that does not span every cluster is a domain, not a sub-population with its own design. Estimating it from a filtered frame loses the clusters that contain none of the subgroup, and those empty clusters carry real information about the variance of the subgroup’s size. The R survey package handles this correctly through svyby on the full design; a filtered rebuild does not.

The effect is usually modest and always in the same direction: the filtered version understates the standard error, which is the direction that makes you overconfident.

Plan the disaggregation before the survey

The honest fix for all of this is upstream. The indicator course made the point in general; here it has a number attached.

If a subgroup is 12% of the population and you need ±10 points on it, the sample size formula run on that subgroup gives what you need — and it is usually far more than the survey was sized for. That is a design conversation, and the options are real:

• Oversample the subgroup, which is what stratification exists for. This survey oversampled rural remote precisely so it could be reported on.
• Accept a wider interval for that subgroup, stated in advance.
• Drop the requirement, and say in the protocol that the survey will not report on it.

What you cannot do is decide afterwards. By the time the data exists the cells are whatever they are, and a table showing eight suppressed cells out of eighteen is the visible form of a design conversation that did not happen.

The table to publish

final = by_cell[["stratum", "displacement_status", "households", "clusters",
                 "estimate", "ci_low", "ci_high", "note"]]
final.to_csv("outputs/tables/food_insecurity_by_stratum_displacement.csv", index=False)

readr::write_csv(by_cell,
  here::here("outputs", "tables", "food_insecurity_by_stratum_displacement.csv"))

Stratum	Status	Households	Clusters	Estimate	95% CI	Note
Urban	Resident	246	25	13.9%	10.2–18.7
Urban	Displaced	49	18	—	—	too few households
Rural remote	Resident	279	25	46.1%	40.0–52.3
Rural remote	Returnee	24	12	—	—	too few households

Seven columns, and the two empty ones are as informative as the four filled ones. A reader can see exactly where the survey ran out, which is the difference between a table that answers questions and one that generates them.

Where this course leaves you

You can reconstruct weights from a frame and prove they sum to the population, size a survey against a precision requirement and read that requirement backwards from a survey you inherited, measure a design effect rather than assuming one, estimate with the design carried through, handle non-response and replacement explicitly, publish an interval of the right shape, and say where the sample stops supporting a cut.

That is the whole of what a survey analyst is judged on, and it is most of module 3. The last course in the module, Routine Data and DHIS2, goes back to the aggregate reporting systems this programme has been borrowing from since module 2 — data elements, category combinations, the org unit hierarchy — and answers the question a coverage figure always provokes: what exactly was counted, and by whom?