# Methodology — Vaccination coverage and dropout analysis

Technical documentation for the project *Vaccination coverage and dropout
analysis*. It sets out how each figure is computed, why the denominator changed
from the previous dashboard, and what the supervision list does and does not
claim.

- **Source dataset:** `vaccination-coverage-2024.v1.csv` (38 facilities,
  12 months, DHIS2-shaped monthly aggregates)
- **Analysis:** `notebooks/coverage-analysis.python.en.ipynb`
- **Decision informed:** which ten of thirty-eight facilities receive a
  supervision visit next quarter
- **Standards applied:** UNICEF indicator definitions, SDG 3.b.1

Every dataset on this platform is synthetic. Coverage here describes no real
district.

---

## 1. The defect this methodology corrects

The previous dashboard ranked facilities by

```
coverage = Σ penta3 doses / target_population
```

summed over all twelve months **whether or not the facility reported**. A
facility that submitted nothing appears in a DHIS2 extract as rows with zero
doses, so a silent facility became a facility that vaccinated nobody, and
supervision visits went to facilities whose only real problem was a late report.

District reporting completeness is **76.5%**. That is not a rounding error on the
denominator; it is a quarter of the district-months.

---

## 2. Reporting completeness — computed first, because it gates everything else

| | |
| --- | --- |
| **Numerator** | Facility-months with `report_submitted` true |
| **Denominator** | Facility-months expected (38 facilities × 12 months = 456) |
| **Value** | 0.765 |
| **Valid at** | Facility, month, district |
| **Decision informed** | Gates every other indicator; also a supervision criterion in its own right |

Deduplicate to one row per facility-month before computing this — the extract
carries one row per facility-month-antigen, and counting antigen rows would
weight a facility by how many vaccines it reports.

**Monthly completeness**

| Month | Completeness |
| --- | ---: |
| Jan | 92.1% |
| Feb | 81.6% |
| Mar | 73.7% |
| Apr | 86.8% |
| May | 84.2% |
| Jun | 86.8% |
| Jul | 81.6% |
| **Aug** | **28.9%** |
| **Sep** | **44.7%** |
| Oct | 94.7% |
| Nov | 76.3% |
| Dec | 86.8% |

**August and September are a district problem, not a facility problem.** A
supervision list built on those two months alone would visit whoever happened to
be quiet during a district-wide disruption. Any monthly trend crossing that
window must be read against this table, and the annual figures below are used
precisely because they are not dominated by it.

---

## 3. Coverage

| | |
| --- | --- |
| **Numerator** | Penta3 doses administered, **in reported months only** |
| **Denominator** | `target_population × (months reported / 12)` |
| **Valid at** | Facility, annual |
| **Decision informed** | Context for the supervision list; **not itself a ranking criterion** |

The denominator is pro-rated to the months the facility actually reported, so the
indicator measures vaccination rather than reporting.

**Effect of the correction.** Ranking facilities on the two denominators and
comparing positions:

| Facility | Reported-only | All months | Completeness | Rank change |
| --- | ---: | ---: | ---: | ---: |
| FAC019 | 0.866 | 0.505 | 58.3% | +21 |
| FAC021 | 0.876 | 0.511 | 58.3% | +21 |
| FAC033 | 0.680 | 0.680 | 100% | −21 |
| FAC001 | 0.767 | 0.703 | 91.7% | −19 |
| FAC006 | 0.986 | 0.575 | 58.3% | +17 |
| FAC004 | 1.015 | 0.592 | 58.3% | +16 |
| FAC025 | 0.714 | 0.655 | 91.7% | −16 |
| FAC011 | 0.774 | 0.452 | 58.3% | +14 |

Facilities move by up to 21 places, and **they move by how often they reported,
not by how well they vaccinated**. Every facility that gains rank is at 58.3%
completeness; every facility that loses rank is at 91.7% or 100%. That is the
defect, isolated.

**Why coverage is not a ranking criterion.** It rests on `target_population`, an
administrative estimate this dataset takes as given. A facility with an
overstated catchment looks like it is underperforming no matter how many children
it reaches. Coverage above 1.0 (FAC004 at 1.015, FAC035 at 1.121) is the visible
symptom. Coverage is reported for context; the visit list is built on the flags
in §5.

---

## 4. Dropout

| | |
| --- | --- |
| **Numerator** | Penta1 doses − Penta3 doses, annual, reported months only |
| **Denominator** | Penta1 doses, same basis |
| **Valid at** | Facility, annual |
| **Decision informed** | Supervision flag (threshold 25%) |

Distribution: median 14.2%, IQR 4.1% to 18.6%, maximum 24.1%, minimum −29.6%.

**The dropout flag fires zero times**, and that is reported rather than quietly
dropped. No facility exceeds 25% on the annual total, so a supervision list built
on dropout alone would be empty.

Keeping the check in and stating that it fired zero times is more useful to next
quarter's supervisor than removing it: a criterion that disappears from the
methodology cannot be seen to have been tested. **Do not lower the threshold to
make it bind** — 25% is the conventional programmatic action level, and moving it
to produce a list would be fitting the criterion to the desired output.

---

## 5. The three supervision flags

| Flag | Condition | Facilities |
| --- | --- | ---: |
| Over-reporting | Annual penta3 > annual penta1 (dropout < 0) | 6 |
| High dropout | Dropout > 25% | 0 |
| Low completeness | Completeness < 70% | 14 |

14 facilities fail at least one check; 10 visits are available. The list must
therefore be **prioritised, not filtered**.

### 5.1 The priority rule

```
priority = 2 × over_reporting + 1 × low_completeness
```

ties broken by ascending completeness.

**Over-reporting outranks low completeness** because a facility reporting more
third doses than first doses is producing numbers that are *wrong*, while a
facility that did not report is producing numbers that are *absent*. Absent is
recoverable by asking; wrong has already entered the district total and the
denominator of every derived figure.

### 5.2 The shortlist

| Facility | Completeness | Dropout | Coverage | Reason |
| --- | ---: | ---: | ---: | --- |
| FAC021 | 58.3% | −29.6% | 0.876 | penta3 > penta1 on the year; reported 58% of months |
| FAC035 | 58.3% | −4.1% | 1.121 | penta3 > penta1 on the year; reported 58% of months |
| FAC009 | 66.7% | −5.7% | 0.676 | penta3 > penta1 on the year; reported 67% of months |
| FAC015 | 66.7% | −0.8% | 0.692 | penta3 > penta1 on the year; reported 67% of months |
| FAC023 | 66.7% | −11.1% | 0.842 | penta3 > penta1 on the year; reported 67% of months |
| FAC034 | 66.7% | −8.0% | 0.844 | penta3 > penta1 on the year; reported 67% of months |
| FAC004 | 58.3% | 13.3% | 1.015 | reported 58% of months |
| FAC006 | 58.3% | 13.4% | 0.986 | reported 58% of months |
| FAC011 | 58.3% | 6.7% | 0.774 | reported 58% of months |
| FAC019 | 58.3% | 23.8% | 0.866 | reported 58% of months |

**Every facility carries its reason.** A visit with no stated question is a wasted
day, and "you are on the list" is not a question.

### 5.3 The facilities that failed a check and did not make the list

| Facility | Completeness | Dropout | Reason |
| --- | ---: | ---: | --- |
| FAC026 | 66.7% | 18.7% | reported 67% of months |
| FAC028 | 66.7% | 13.9% | reported 67% of months |
| FAC030 | 66.7% | 12.3% | reported 67% of months |
| FAC036 | 66.7% | 19.1% | reported 67% of months |

**Naming these four is the point of a capped shortlist.** They are next quarter's
first candidates, and a supervisor who is not told about them will assume the
list was exhaustive.

---

## 6. What the supervision list claims

It is a list of facilities whose **data** deserves a conversation. Whether the
underlying service is failing is what the visit is for.

Specifically: a facility on this list is not thereby a poorly performing
immunisation service, and a facility off it is not thereby a well performing one.
Facilities with high completeness and genuinely low coverage are invisible to
these three flags — that is a deliberate scope limit, because with an
administrative denominator this dataset cannot distinguish low coverage from an
overstated catchment.

---

## 7. Limitations

1. **`target_population` is taken as given.** Where it is wrong, coverage is
   wrong in the same direction. This is the largest source of error in the
   coverage figure and the reason coverage does not rank the list.
2. **Aggregate data cannot see the child.** Dropout computed on annual totals
   assumes the penta1 and penta3 cohorts correspond. A facility with a genuine
   caseload shift shows dropout that is real arithmetic and not real dropout.
3. **A submitted report is not a correct report.** `report_submitted` records
   that something arrived, not that it was right. The six over-reporting
   facilities are evidence that submission and accuracy are separate properties.
4. **No antigen beyond penta1/penta3 enters the analysis**, so the list says
   nothing about measles, BCG or polio performance.
5. **August–September disruption is unexplained** by this dataset. Whether it was
   a strike, a system outage or a reporting-tool change changes what the
   supervisor should ask, and it is not recorded here.

---

## 8. Reproducing this analysis

```bash
pnpm examples:build
```

Then run `notebooks/coverage-analysis.python.en.ipynb`. It needs pandas and numpy
and reads the CSV over HTTPS.

The dataset is versioned by filename and immutable; a correction ships as
`.v2.csv` with this document revised beside it.

---

## 9. Change log

| Date | Change |
| --- | --- |
| 2026-07-27 | First issue, against dataset v1. |