Three coverages that disagree — Public Health and Applied Epidemiology — Data Analysis

Three ways to measure the same thing

Immunisation coverage is the most-reported health indicator in this sector and it is produced three different ways, which routinely disagree.

Method	Numerator	Denominator	Fails when
Administrative	Doses recorded by facilities	Population projection	The projection is wrong, or reporting is incomplete
Survey	Children with a card or recalled dose	Children sampled	Cards are lost; recall is poor; the sample is small
Dropout	First dose minus last dose	First dose	Rarely — and that is the point

The DHIS2 course computed the first. The survey course gave you the machinery for the second. This lesson is why they differ and which to use.

Administrative coverage, and its two dependencies

import pandas as pd

vax = pd.read_csv("vaccination-coverage-2024.v1.csv", parse_dates=["period"])
reported = vax[vax["report_submitted"] == True]

penta3 = reported[reported["antigen"] == "penta3"]
annual_target = (
    vax[vax["antigen"] == "penta3"].groupby("facility_id")["target_population"].first()
)
months = penta3.groupby("facility_id").size()
prorated = (annual_target * months.reindex(annual_target.index).fillna(0) / 12).sum()

print(f"doses {penta3['doses_administered'].sum():,}")
print(f"administrative coverage: "
      f"{penta3['doses_administered'].sum() / prorated:.1%}")

library(dplyr)
# same shape: doses over a denominator pro-rated to months reported

77.5%, among reporting facility-months. That figure carries two dependencies and the DHIS2 course established both.

The denominator is a projection. Surviving infants, from a census projection compounded forward at an assumed growth rate. Nine years at 2.4% is a factor of 1.24, so a quarter of the denominator is an assumption.

The reporting rate was 76.5%. Facilities that did not report contribute no doses, so the figure is a lower bound unless their doses are imputed.

Administrative coverage above 100% is common and always means one of those two is wrong, plus a third possibility — children vaccinated outside their catchment, which inflates a facility numerator against a catchment denominator.

Survey coverage, and its two dependencies

A coverage survey samples children of a given age and asks whether they were vaccinated, verified against a card where one exists.

It removes the projection problem — the denominator is the sample — and introduces two others.

Card retention. Where cards are lost, coverage rests on caregiver recall, which overstates for some antigens and understates for others. Report card-verified and recall-based coverage separately; the gap between them is the measure of how much the estimate depends on memory.

Precision. Survey coverage arrives with a confidence interval, and the survey course showed that a cluster design widens it. A survey that estimates coverage to plus or minus six points cannot settle whether a district crossed an 80% target.

# From the survey course: a proportion with a design-adjusted interval
def coverage_interval(p, n, deff, t=1.99):
    se = (p * (1 - p) / n * deff) ** 0.5
    return p - t * se, p + t * se

print(coverage_interval(0.775, 900, 2.0))

coverage_interval <- function(p, n, deff, t = 1.99) {
  se <- sqrt(p * (1 - p) / n * deff)
  c(p - t * se, p + t * se)
}

Why they disagree

Five reasons, and knowing which applies changes what you do.

The projection is wrong. The commonest, and it moves administrative coverage only. A denominator too small pushes administrative above survey.
Reporting is incomplete. Moves administrative down.
Catchment crossing. Moves facility-level administrative figures in both directions and cancels at district level.
Recall error. Moves survey coverage, usually up.
Different age cohorts. Administrative coverage counts doses given this year; a survey counts children aged 12 to 23 months who were vaccinated at any time. These are not the same children.

The last one is the most common and the least suspected. Two figures for “penta3 coverage” that refer to different cohorts are not two estimates of one quantity; they are two quantities.

Dropout: the figure that survives all of it

doses = reported.groupby("antigen")["doses_administered"].sum()

for start, end in [("penta1", "penta3"), ("mcv1", "mcv2")]:
    dropout = (doses[start] - doses[end]) / doses[start]
    print(f"{start} -> {end}: {dropout:.1%}")

vax |> filter(report_submitted) |>
  summarise(doses = sum(doses_administered), .by = antigen)

Penta1 to penta3: 13.8%. Measles first to second dose: 22.9%.

Dropout is a ratio of two numerators from the same source, the same facilities and the same months. It therefore has:

no population denominator, so no projection to be wrong;
the same reporting bias in both terms, which largely cancels;
no cohort ambiguity, because both doses are counted the same way.

The cost is that it says nothing about level. A district could have 13.8% dropout and 30% coverage — good at retaining the children it starts, bad at starting them. Report dropout beside coverage, never instead of it.

What the two dropouts say together

Measles second-dose dropout is 22.9% against 13.8% for pentavalent, and the difference is informative rather than noise.

The pentavalent doses are given close together in the first months of life, when carers are already attending for other services. The measles second dose is given much later, often after the intensive contact period has ended. A dropout that rises with the interval between doses is a reminder-and-outreach problem, not a supply problem, and the corrective actions differ.

summary = pd.DataFrame({
    "series": ["Pentavalent 1->3", "Measles 1->2"],
    "dropout": [0.138, 0.229],
    "interval": ["weeks", "months"],
    "implication": ["retention within the early contact period",
                    "retention after the early contact period ends"],
})

tibble::tribble(
  ~series,             ~dropout, ~implication,
  "Pentavalent 1->3",     0.138, "retention within early contacts",
  "Measles 1->2",         0.229, "retention after early contacts end"
)

Which to report

Penta3 coverage, administrative     77.5%   among reporting facility-months
                                            (reporting rate 76.5%)
                                            denominator: MoH projection from 2015 census
Penta1 to penta3 dropout            13.8%   same source, both terms
Measles 1 to 2 dropout              22.9%
Survey coverage                     not available this year

Four lines, and the last one is a finding. Where no survey exists, say so — because the reader’s default assumption is that a coverage figure has been validated against one, and here it has not.

What comes next

Coverage compares a programme against a population. The next lesson compares two populations against each other, and finds that the comparison in lesson 4 was partly an artefact of who lives where.