Choosing which six facilities to visit

You have a week and thirty-eight facilities

A verification visit costs a vehicle, a day, two people and fuel. The budget covers six. Everything in this lesson follows from that arithmetic, and the first thing to be clear about is that which six you choose determines what your finding is allowed to mean.

Three strategies, three questions, and mixing them up is the commonest DQA methods failure.

Strategy	Answers	Cannot answer
Random	“What is the district’s data quality?”	“Where are the problems?”
Risk-based	“Are the facilities we suspect actually wrong?”	“What is the district’s data quality?”
Census of the largest	“Is the district total right?”	Anything about small facilities

Random: the only one that generalises

If you want a statement about the district, the sample has to be drawn without reference to what you expect to find.

import pandas as pd

facilities = vax[["facility_id", "facility_type"]].drop_duplicates()

sample = facilities.sample(n=6, random_state=20260728)
print(sample)

set.seed(20260728)

sample <- facilities |> dplyr::slice_sample(n = 6)

Seed it and record the seed. A DQA whose sample cannot be reproduced invites the question of whether the facilities were chosen after somebody knew what was in them, and the seed answers it in one line.

Stratify where a subgroup matters. Health posts report worst, so a simple random sample of six could easily contain none of them:

sample = (
    facilities.groupby("facility_type", group_keys=False)
    .apply(lambda g: g.sample(n=min(2, len(g)), random_state=20260728))
)

sample <- facilities |>
  group_by(facility_type) |>
  slice_sample(n = 2) |>
  ungroup()

Stratifying by type gives you a statement about each type as well as about the district, and it costs nothing.

Risk-based: where the problems are

If the purpose is to fix things rather than to describe them, choose the facilities the desk analysis already flagged. The next two lessons are entirely about producing that ranking.

risk = (
    vax.groupby("facility_id")
    .agg(reporting_rate=("reported", "mean"))
    .assign(round_share=lambda d: d.index.map(round_number_share))
)
risk["score"] = (1 - risk["reporting_rate"]) + risk["round_share"]
print(risk.sort_values("score", ascending=False).head(6))

risk <- vax |>
  summarise(reporting_rate = mean(report_submitted), .by = facility_id) |>
  left_join(round_shares, by = "facility_id") |>
  mutate(score = (1 - reporting_rate) + round_share) |>
  arrange(desc(score))

This finds more problems per vehicle-day than random sampling, which is why most real DQAs use it. It also cannot be generalised, and the report must say so:

Six facilities were selected on the basis of desk-review flags. Findings describe those facilities and are not representative of the district.

Every DQA report that omits that sentence and then quotes a district-level verification factor is making a claim its design does not support.

Census of the largest

A third option, and often the most useful for a total. Order facilities by volume, and visit enough of them to cover most of the numerator.

volume = (
    vax[vax["reported"]]
    .groupby("facility_id")["doses_administered"].sum()
    .sort_values(ascending=False)
)
share = volume.cumsum() / volume.sum()
print(share.head(10).round(3))

volume <- vax |>
  filter(report_submitted) |>
  summarise(doses = sum(doses_administered), .by = facility_id) |>
  arrange(desc(doses)) |>
  mutate(cumulative = cumsum(doses) / sum(doses))

If eight facilities account for half the doses, verifying those eight bounds the error on the district total regardless of what the other thirty do. This is the right design when the question is “can I trust the total I am reporting to the donor” — and the wrong one when the question is about service quality in remote posts, because it systematically ignores them.

How many is enough

There is no single answer, but there is a way to think about it that survives being challenged.

For a yes/no question — is this facility’s reporting acceptable — you are estimating a proportion, and precision improves with the square root of the sample. Six facilities out of thirty-eight gives a confidence interval so wide that almost no result is distinguishable from any other. Be honest about that rather than reporting “33% of facilities had findings” from a sample of six.

For detecting whether a problem exists at all, small samples are far more capable. If 30% of facilities have a defect, the chance that six randomly chosen facilities contain none of them is about 12% — so a clean sample of six is reasonable evidence that a widespread problem is absent, and no evidence at all about a rare one.

p = 0.30
n = 6
print(f"chance of finding none: {(1 - p) ** n:.1%}")

(1 - 0.30)^6

That single line is worth putting in the report. It converts “we visited six and found nothing” into a statement with a number attached.

LQAS: designed for exactly this

Lot Quality Assurance Sampling is the formalisation of the paragraph above, and this sector already uses it for coverage surveys. The idea is to stop trying to estimate and start trying to decide: pick a threshold, a sample size and a decision rule, and accept or reject.

Visit 12 facilities. If 3 or more have a verification factor outside 0.95–1.05, the district’s reporting is classified as needing corrective action.

The virtue is that the sample can be small because you are answering a smaller question. The cost is that you get a verdict rather than a figure, and someone will ask for the figure anyway. Say up front which you designed for.

Combine, and label

Real DQAs usually do all three, and that is fine as long as the report separates them.

plan = pd.DataFrame({
    "facility_id": ["FAC012", "FAC033", "FAC004", "FAC019", "FAC007", "FAC021"],
    "reason": ["random", "random", "risk: reporting 58%", "risk: round numbers",
               "volume: top 5", "volume: top 5"],
})

plan <- tibble::tribble(
  ~facility_id, ~reason,
  "FAC012", "random",
  "FAC033", "random",
  "FAC004", "risk: reporting 58%",
  "FAC019", "risk: round numbers",
  "FAC007", "volume: top 5",
  "FAC021", "volume: top 5"
)

The reason column is the methods section. With it, a reader knows which findings generalise and which do not. Without it, the report has one sample and three incompatible claims, and a reviewer who notices will discount all of them.

What comes next

Risk-based sampling needs a risk ranking, and so far you have only reporting rates. The next two lessons build the rest of it from the extract you already hold — trends that move impossibly, values that repeat, and digits that betray a number nobody measured.