Back to the lesson·Lesson 4 of 8·Checking against the source
Choosing which six facilities to visit
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- You have a week and thirty-eight facilities
- Random: the only one that generalises
- Risk-based: where the problems are
- Census of the largest
- How many is enough
- LQAS: designed for exactly this
- Combine, and label
- What comes next
Speaker notes
Three sampling strategies that answer three different questions, how many is enough, and the sentence you are allowed to write about the thirty-two facilities you did not visit.You have a week and thirty-eight facilities
Strategy Answers Cannot answer Random "What is the district's data quality?" "Where are the problems?" Risk-based "Are the facilities we suspect actually wrong?" "What is the district's data quality?" Census of the largest "Is the district total right?" Anything about small facilities Speaker notes
A verification visit costs a vehicle, a day, two people and fuel. The budget covers six. Everything in this lesson follows from that arithmetic, and the first thing to be clear about is that which six you choose determines what your finding is allowed to mean. Three strategies, three questions, and mixing them up is the commonest DQA methods failure.Random: the only one that generalises — In Python
import pandas as pd facilities = vax[["facility_id", "facility_type"]].drop_duplicates() sample = facilities.sample(n=6, random_state=20260728) print(sample)Speaker notes
If you want a statement about the district, the sample has to be drawn without reference to what you expect to find.Random: the only one that generalises — In R
set.seed(20260728) sample <- facilities |> dplyr::slice_sample(n = 6)Random: the only one that generalises
- Seed it and record the seed — A DQA whose sample cannot be reproduced invites the question of whether the facilities…
Speaker notes
Seed it and record the seed. A DQA whose sample cannot be reproduced invites the question of whether the facilities were chosen after somebody knew what was in them, and the seed answers it in one line. Stratify where a subgroup matters. Health posts report worst, so a simple random sample of six could easily contain none of them:Random: the only one that generalises — In Python
sample = ( facilities.groupby("facility_type", group_keys=False) .apply(lambda g: g.sample(n=min(2, len(g)), random_state=20260728)) )Random: the only one that generalises — In R
sample <- facilities |> group_by(facility_type) |> slice_sample(n = 2) |> ungroup()Speaker notes
Stratifying by type gives you a statement about each type as well as about the district, and it costs nothing.Risk-based: where the problems are — In Python
risk = ( vax.groupby("facility_id") .agg(reporting_rate=("reported", "mean")) .assign(round_share=lambda d: d.index.map(round_number_share)) ) risk["score"] = (1 - risk["reporting_rate"]) + risk["round_share"] print(risk.sort_values("score", ascending=False).head(6))Speaker notes
If the purpose is to fix things rather than to describe them, choose the facilities the desk analysis already flagged. The next two lessons are entirely about producing that ranking.Risk-based: where the problems are — In R
risk <- vax |> summarise(reporting_rate = mean(report_submitted), .by = facility_id) |> left_join(round_shares, by = "facility_id") |> mutate(score = (1 - reporting_rate) + round_share) |> arrange(desc(score))Risk-based: where the problems are
Six facilities were selected on the basis of desk-review flags. Findings describe those facilities and are not representative of the district.
Speaker notes
This finds more problems per vehicle-day than random sampling, which is why most real DQAs use it. It also cannot be generalised, and the report must say so: Every DQA report that omits that sentence and then quotes a district-level verification factor is making a claim its design does not support.Census of the largest — In Python
volume = ( vax[vax["reported"]] .groupby("facility_id")["doses_administered"].sum() .sort_values(ascending=False) ) share = volume.cumsum() / volume.sum() print(share.head(10).round(3))Speaker notes
A third option, and often the most useful for a total. Order facilities by volume, and visit enough of them to cover most of the numerator.Census of the largest — In R
volume <- vax |> filter(report_submitted) |> summarise(doses = sum(doses_administered), .by = facility_id) |> arrange(desc(doses)) |> mutate(cumulative = cumsum(doses) / sum(doses))Speaker notes
If eight facilities account for half the doses, verifying those eight bounds the error on the district total regardless of what the other thirty do. This is the right design when the question is "can I trust the total I am reporting to the donor" — and the wrong one when the question is about service quality in remote posts, because it systematically ignores them.How many is enough — In Python
p = 0.30 n = 6 print(f"chance of finding none: {(1 - p) ** n:.1%}")Speaker notes
There is no single answer, but there is a way to think about it that survives being challenged. For a yes/no question — is this facility's reporting acceptable — you are estimating a proportion, and precision improves with the square root of the sample. Six facilities out of thirty-eight gives a confidence interval so wide that almost no result is distinguishable from any other. Be honest about that rather than reporting "33% of facilities had findings" from a sample of six. For detecting whether a problem exists at all, small samples are far more capable. If 30% of facilities have a defect, the chance that six randomly chosen facilities contain none of them is about 12% — so a clean sample of six is reasonable evidence that a widespread problem is absent, and no evidence at all about a rare one.How many is enough — In R
(1 - 0.30)^6Speaker notes
That single line is worth putting in the report. It converts "we visited six and found nothing" into a statement with a number attached.LQAS: designed for exactly this
Visit 12 facilities. If 3 or more have a verification factor outside 0.95–1.05, the district's reporting is classified as needing corrective action.
Speaker notes
Lot Quality Assurance Sampling is the formalisation of the paragraph above, and this sector already uses it for coverage surveys. The idea is to stop trying to estimate and start trying to decide: pick a threshold, a sample size and a decision rule, and accept or reject. The virtue is that the sample can be small because you are answering a smaller question. The cost is that you get a verdict rather than a figure, and someone will ask for the figure anyway. Say up front which you designed for.Combine, and label — In Python
plan = pd.DataFrame({ "facility_id": ["FAC012", "FAC033", "FAC004", "FAC019", "FAC007", "FAC021"], "reason": ["random", "random", "risk: reporting 58%", "risk: round numbers", "volume: top 5", "volume: top 5"], })Speaker notes
Real DQAs usually do all three, and that is fine as long as the report separates them.Combine, and label — In R
plan <- tibble::tribble( ~facility_id, ~reason, "FAC012", "random", "FAC033", "random", "FAC004", "risk: reporting 58%", "FAC019", "risk: round numbers", "FAC007", "volume: top 5", "FAC021", "volume: top 5" )Combine, and label
- The
reasoncolumn is the methods section — With it, a reader knows which findings generalise and which do not
Speaker notes
Thereasoncolumn is the methods section. With it, a reader knows which findings generalise and which do not. Without it, the report has one sample and three incompatible claims, and a reviewer who notices will discount all of them.- The
What comes next
- Risk-based sampling needs a risk ranking, and so far you have only reporting rates.
Speaker notes
Risk-based sampling needs a risk ranking, and so far you have only reporting rates. The next two lessons build the rest of it from the extract you already hold — trends that move impossibly, values that repeat, and digits that betray a number nobody measured.