cassionData Analysis

Back to the lessonLesson 4 of 8Periods and completeness

Reporting rate is a denominator, not a footnote

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 19

    What this lesson covers

    • Four measures, one word
    • Where the system's answer differs from yours
    • Reporting rate as the denominator of everything else
    • The two ways to handle a silent facility
    • Report the pair
    • What comes next
    Speaker notes
    The system computes four completeness measures and they disagree. Which one your report quotes decides whether a coverage figure is an estimate or a lower bound, and August is 29% either way.

  2. Slide 2 / 19

    Four measures, one word

    MeasureNumeratorDenominator
    Reporting rateDatasets marked completeDatasets expected
    Actual reportsDatasets marked complete— (a count)
    Expected reports—Org units assigned x periods
    Reporting rate on timeCompleted by the deadlineDatasets expected
    Speaker notes
    The Data Quality Assessment course established that reporting completeness is an indicator in its own right. This lesson is about what the system means by it, because DHIS2 exposes four related measures under names that get used interchangeably.

  3. Slide 3 / 19

    Four measures, one word — In Python

    import pandas as pd

vax = pd.read_csv("vaccination-coverage-2024.v1.csv", parse_dates=["period"])
vax["reported"] = vax["report_submitted"] == True

facility_months = vax.drop_duplicates(["facility_id", "period"])
print(f"expected: {len(facility_months)}")
print(f"actual:   {facility_months['reported'].sum()}")
print(f"rate:     {facility_months['reported'].mean():.1%}")
    Speaker notes
    The first and last differ only by timeliness, and a report that quotes "reporting rate" without saying which has left a reader unable to tell whether 82% means "submitted" or "submitted on time".

  4. Slide 4 / 19

    Four measures, one word — In R

    library(dplyr)

vax |>
  distinct(facility_id, period, .keep_all = TRUE) |>
  summarise(expected = n(), actual = sum(report_submitted),
            rate = mean(report_submitted))
    Speaker notes
    456 facility-months expected, 349 submitted, 76.5%. Note the deduplication: completeness is registered per dataset per org unit per period, not per data element, so counting the 2,736 element-level rows would answer a different question and give the same percentage by coincidence.

  5. Slide 5 / 19

    Where the system's answer differs from yours

    • Expected reports counts assignment, not existence — A facility that closed in March remains in the denominator for the…
    • Completeness is a click, not a calculation — A facility can enter every value and never mark the form complete; the…
    Speaker notes
    Two differences, and both have caught people out. Expected reports counts assignment, not existence. A facility that closed in March remains in the denominator for the rest of the year unless somebody un-assigns the dataset or closes the org unit. Reporting rates therefore drift downward as facilities close and nobody updates the assignment, and the drift looks like deteriorating performance. Completeness is a click, not a calculation. A facility can enter every value and never mark the form complete; the system counts it as not reporting while the data sits there. The opposite also happens. So the reporting rate and the presence of data are two different questions:

  6. Slide 6 / 19

    Where the system's answer differs from yours — In Python

    has_data = (
    vax.groupby(["facility_id", "period"])["doses_administered"]
    .sum().gt(0).rename("has_data")
)
flags = facility_months.set_index(["facility_id", "period"])["reported"]

crosstab = pd.crosstab(flags, has_data.reindex(flags.index))
print(crosstab)

  7. Slide 7 / 19

    Where the system's answer differs from yours — In R

    vax |>
  summarise(reported = first(report_submitted),
            has_data = sum(doses_administered) > 0,
            .by = c(facility_id, period)) |>
  count(reported, has_data)
    Speaker notes
    On this extract the two agree exactly — every reported month has doses and every unreported month has none — which is the tidy case. Run the crosstab anyway. The off-diagonal cells are where a real instance keeps its most interesting problems, and a cell of "marked complete, no data" is a facility submitting empty forms to meet a target.

  8. Slide 8 / 19

    Reporting rate as the denominator of everything else — In Python

    monthly = (
    vax[vax["antigen"] == "penta3"]
    .groupby(vax["period"].dt.strftime("%Y-%m"))
    .apply(lambda g: pd.Series({
        "reporting_rate": g["reported"].mean(),
        "coverage_reporting": (g.loc[g["reported"], "doses_administered"].sum()
                               / g.loc[g["reported"], "target_population"].sum()),
    }))
)
print((monthly * 100).round(1))
    Speaker notes
    The operational point, and it is the reason this lesson sits in a DHIS2 course rather than only in the DQA one.

  9. Slide 9 / 19

    Reporting rate as the denominator of everything else — In R

    vax |>
  filter(antigen == "penta3") |>
  mutate(month = format(period, "%Y-%m")) |>
  summarise(
    reporting_rate = mean(report_submitted),
    coverage = sum(doses_administered[report_submitted]) /
               sum(target_population[report_submitted]),
    .by = month
  )

  10. Slide 10 / 19

    Reporting rate as the denominator of everything else

    MonthReporting rateCoverage among reporters
    July82%…
    August29%…
    September45%…
    October95%…

  11. Slide 11 / 19

    Reporting rate as the denominator of everything else

    • Three rules that follow, and they are the deliverable of this lesson
    • Below 90% reporting, a coverage figure is a lower bound, not an estimate. Label it "among reporting facilities"…
    • Do not compare two periods with materially different reporting rates without saying so. August against July is a…
    • A trend line needs the reporting rate on the same chart. Plotted alone, a coverage series through August looks like…
    Speaker notes
    Two columns, always adjacent. August's coverage figure is computed on eleven of thirty-eight facilities, and whatever it says, it says it about those eleven. Three rules that follow, and they are the deliverable of this lesson.

  12. Slide 12 / 19

    The two ways to handle a silent facility

    • Exclude it from both numerator and denominator — Coverage among facilities that reported
    • Impute its doses — Fill from the facility's own recent months, and say so
    Speaker notes
    Neither is wrong. They answer different questions and they must be labelled. Exclude it from both numerator and denominator. Coverage among facilities that reported. Honest, and it is what the code above does. Impute its doses. Fill from the facility's own recent months, and say so. This is what national estimates do — WUENIC and similar exercises impute rather than leave gaps, because the question is about children rather than about paperwork.

  13. Slide 13 / 19

    The two ways to handle a silent facility — In Python

    by_facility = (
    vax[(vax["antigen"] == "penta3") & vax["reported"]]
    .groupby("facility_id")["doses_administered"].median()
)
missing = (vax["antigen"] == "penta3") & ~vax["reported"]
imputed_total = (vax.loc[missing, "facility_id"].map(by_facility).sum()
                 + vax.loc[(vax["antigen"] == "penta3") & vax["reported"],
                           "doses_administered"].sum())

annual_target = (vax[vax["antigen"] == "penta3"]
                 .groupby("facility_id")["target_population"].first().sum())
print(f"imputed annual coverage: {imputed_total / annual_target:.1%}")

  14. Slide 14 / 19

    The two ways to handle a silent facility — In R

    median_by_facility <- vax |>
  filter(antigen == "penta3", report_submitted) |>
  summarise(m = median(doses_administered), .by = facility_id)

  15. Slide 15 / 19

    The two ways to handle a silent facility

    • Never impute silently — An imputed figure and a reported one in the same column with no flag is the defect the cleaning…
    Speaker notes
    Never impute silently. An imputed figure and a reported one in the same column with no flag is the defect the cleaning course named: an estimate that has stopped being distinguishable from a measurement.

  16. Slide 16 / 19

    Report the pair — In Python

    summary = pd.DataFrame({
    "period": ["2024-08", "2024-09", "2024-10"],
    "facilities_reporting": [11, 17, 36],
    "facilities_expected": [38, 38, 38],
    "reporting_rate": ["29%", "45%", "95%"],
    "coverage_basis": ["among reporting facilities"] * 3,
})

  17. Slide 17 / 19

    Report the pair — In R

    tibble::tribble(
  ~period,   ~reporting, ~expected, ~basis,
  "2024-08",         11,        38, "among reporting facilities",
  "2024-09",         17,        38, "among reporting facilities",
  "2024-10",         36,        38, "among reporting facilities"
)
    Speaker notes
    Five columns and no coverage figure travels without them. That is the whole argument, and it is cheaper than the meeting where somebody asks why August collapsed.

  18. Slide 18 / 19

    What comes next

    • Everything so far has run on a CSV somebody exported by hand.
    Speaker notes
    Everything so far has run on a CSV somebody exported by hand. The next unit replaces that: the Web API, what to ask it for, and a script that makes the same request every month and records what it asked.

  19. Slide 19 / 19

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson