cassionData Analysis

Back to the lessonLesson 5 of 8Comparing places

Three coverages that disagree

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 17

    What this lesson covers

    • Three ways to measure the same thing
    • Administrative coverage, and its two dependencies
    • Survey coverage, and its two dependencies
    • Why they disagree
    • Dropout: the figure that survives all of it
    • What the two dropouts say together
    • Which to report
    • What comes next
    Speaker notes
    Administrative coverage, survey coverage and the dropout between doses. The first depends on a projection, the second on a sample, and only the third is immune to both.

  2. Slide 2 / 17

    Three ways to measure the same thing

    MethodNumeratorDenominatorFails when
    AdministrativeDoses recorded by facilitiesPopulation projectionThe projection is wrong, or reporting is incomplete
    SurveyChildren with a card or recalled doseChildren sampledCards are lost; recall is poor; the sample is small
    DropoutFirst dose minus last doseFirst doseRarely — and that is the point
    Speaker notes
    Immunisation coverage is the most-reported health indicator in this sector and it is produced three different ways, which routinely disagree. The DHIS2 course computed the first. The survey course gave you the machinery for the second. This lesson is why they differ and which to use.

  3. Slide 3 / 17

    Administrative coverage, and its two dependencies — In Python

    import pandas as pd

vax = pd.read_csv("vaccination-coverage-2024.v1.csv", parse_dates=["period"])
reported = vax[vax["report_submitted"] == True]

penta3 = reported[reported["antigen"] == "penta3"]
annual_target = (
    vax[vax["antigen"] == "penta3"].groupby("facility_id")["target_population"].first()
)
months = penta3.groupby("facility_id").size()
prorated = (annual_target * months.reindex(annual_target.index).fillna(0) / 12).sum()

print(f"doses {penta3['doses_administered'].sum():,}")
print(f"administrative coverage: "
      f"{penta3['doses_administered'].sum() / prorated:.1%}")

  4. Slide 4 / 17

    Administrative coverage, and its two dependencies — In R

    library(dplyr)
# same shape: doses over a denominator pro-rated to months reported

  5. Slide 5 / 17

    Administrative coverage, and its two dependencies

    • 77.5% — among reporting facility-months
    • The denominator is a projection — Surviving infants, from a census projection compounded forward at an assumed growth…
    • The reporting rate was 76.5% — Facilities that did not report contribute no doses, so the figure is a lower bound…
    Speaker notes
    77.5%, among reporting facility-months. That figure carries two dependencies and the DHIS2 course established both. The denominator is a projection. Surviving infants, from a census projection compounded forward at an assumed growth rate. Nine years at 2.4% is a factor of 1.24, so a quarter of the denominator is an assumption. The reporting rate was 76.5%. Facilities that did not report contribute no doses, so the figure is a lower bound unless their doses are imputed. Administrative coverage above 100% is common and always means one of those two is wrong, plus a third possibility — children vaccinated outside their catchment, which inflates a facility numerator against a catchment denominator.

  6. Slide 6 / 17

    Survey coverage, and its two dependencies

    • Card retention — Where cards are lost, coverage rests on caregiver recall, which overstates for some antigens and…
    • Precision — Survey coverage arrives with a confidence interval, and the survey course showed that a cluster design…
    Speaker notes
    A coverage survey samples children of a given age and asks whether they were vaccinated, verified against a card where one exists. It removes the projection problem — the denominator is the sample — and introduces two others. Card retention. Where cards are lost, coverage rests on caregiver recall, which overstates for some antigens and understates for others. Report card-verified and recall-based coverage separately; the gap between them is the measure of how much the estimate depends on memory. Precision. Survey coverage arrives with a confidence interval, and the survey course showed that a cluster design widens it. A survey that estimates coverage to plus or minus six points cannot settle whether a district crossed an 80% target.

  7. Slide 7 / 17

    Survey coverage, and its two dependencies — In Python

    # From the survey course: a proportion with a design-adjusted interval
def coverage_interval(p, n, deff, t=1.99):
    se = (p * (1 - p) / n * deff) ** 0.5
    return p - t * se, p + t * se

print(coverage_interval(0.775, 900, 2.0))

  8. Slide 8 / 17

    Survey coverage, and its two dependencies — In R

    coverage_interval <- function(p, n, deff, t = 1.99) {
  se <- sqrt(p * (1 - p) / n * deff)
  c(p - t * se, p + t * se)
}

  9. Slide 9 / 17

    Why they disagree

    • The projection is wrong. The commonest, and it moves administrative coverage only. A denominator too small pushes…
    • Reporting is incomplete. Moves administrative down.
    • Catchment crossing. Moves facility-level administrative figures in both directions and cancels at district level.
    • Recall error. Moves survey coverage, usually up.
    • Different age cohorts. Administrative coverage counts doses given this year; a survey counts children aged 12 to 23…
    • The last one is the most common and the least suspected — Two figures for "penta3 coverage" that refer to different…
    Speaker notes
    Five reasons, and knowing which applies changes what you do. The last one is the most common and the least suspected. Two figures for "penta3 coverage" that refer to different cohorts are not two estimates of one quantity; they are two quantities.

  10. Slide 10 / 17

    Dropout: the figure that survives all of it — In Python

    doses = reported.groupby("antigen")["doses_administered"].sum()

for start, end in [("penta1", "penta3"), ("mcv1", "mcv2")]:
    dropout = (doses[start] - doses[end]) / doses[start]
    print(f"{start} -> {end}: {dropout:.1%}")

  11. Slide 11 / 17

    Dropout: the figure that survives all of it — In R

    vax |> filter(report_submitted) |>
  summarise(doses = sum(doses_administered), .by = antigen)

  12. Slide 12 / 17

    Dropout: the figure that survives all of it

    • Penta1 to penta3: 13.8%. Measles first to second dose: 22.9%
    • no population denominator, so no projection to be wrong;
    • the same reporting bias in both terms, which largely cancels;
    • no cohort ambiguity, because both doses are counted the same way.
    Speaker notes
    Penta1 to penta3: 13.8%. Measles first to second dose: 22.9%. Dropout is a ratio of two numerators from the same source, the same facilities and the same months. It therefore has: The cost is that it says nothing about level. A district could have 13.8% dropout and 30% coverage — good at retaining the children it starts, bad at starting them. Report dropout beside coverage, never instead of it.

  13. Slide 13 / 17

    What the two dropouts say together — In Python

    summary = pd.DataFrame({
    "series": ["Pentavalent 1->3", "Measles 1->2"],
    "dropout": [0.138, 0.229],
    "interval": ["weeks", "months"],
    "implication": ["retention within the early contact period",
                    "retention after the early contact period ends"],
})
    Speaker notes
    Measles second-dose dropout is 22.9% against 13.8% for pentavalent, and the difference is informative rather than noise. The pentavalent doses are given close together in the first months of life, when carers are already attending for other services. The measles second dose is given much later, often after the intensive contact period has ended. A dropout that rises with the interval between doses is a reminder-and-outreach problem, not a supply problem, and the corrective actions differ.

  14. Slide 14 / 17

    What the two dropouts say together — In R

    tibble::tribble(
  ~series,             ~dropout, ~implication,
  "Pentavalent 1->3",     0.138, "retention within early contacts",
  "Measles 1->2",         0.229, "retention after early contacts end"
)

  15. Slide 15 / 17

    Which to report — Example

    Penta3 coverage, administrative     77.5%   among reporting facility-months
                                            (reporting rate 76.5%)
                                            denominator: MoH projection from 2015 census
Penta1 to penta3 dropout            13.8%   same source, both terms
Measles 1 to 2 dropout              22.9%
Survey coverage                     not available this year
    Speaker notes
    Four lines, and the last one is a finding. Where no survey exists, say so — because the reader's default assumption is that a coverage figure has been validated against one, and here it has not.

  16. Slide 16 / 17

    What comes next

    • Coverage compares a programme against a population.
    Speaker notes
    Coverage compares a programme against a population. The next lesson compares two populations against each other, and finds that the comparison in lesson 4 was partly an artefact of who lives where.

  17. Slide 17 / 17

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson