cassionData Analysis

Back to the lessonLesson 2 of 8What quality means here

The two indicators nobody computes

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 22

    What this lesson covers

    • Two indicators, and neither is in the report
    • Reporting completeness
    • Break it down three ways, always
    • The denominator trap
    • Timeliness needs a field you may not have
    • Read every other figure through these two
    • What comes next
    Speaker notes
    Reporting completeness and timeliness, each with its own denominator. Eleven of thirty-eight facilities reported in August, and every coverage figure for that month is a statement about those eleven.

  2. Slide 2 / 22

    Two indicators, and neither is in the report

    • Open almost any routine health report from this sector and you will find coverage, caseload, cure rates and stock-outs.
    Speaker notes
    Open almost any routine health report from this sector and you will find coverage, caseload, cure rates and stock-outs. You will very rarely find the two numbers that say how much of the district those figures describe. They are cheap. Both come out of the extract you already have — one of them does — and both change how every other number in the report should be read.

  3. Slide 3 / 22

    Reporting completeness — In Python

    import pandas as pd

vax = pd.read_csv("vaccination-coverage-2024.v1.csv", parse_dates=["period"])
vax["reported"] = vax["report_submitted"] == True

print(f"overall reporting rate: {vax['reported'].mean():.1%}")

by_month = vax.groupby(vax["period"].dt.strftime("%Y-%m"))["reported"].mean()
print((by_month * 100).round(0))
    Speaker notes
    The reporting rate is the share of expected reports that were actually submitted. The word doing the work is expected.

  4. Slide 4 / 22

    Reporting completeness — In R

    library(dplyr)
library(readr)

vax <- read_csv("vaccination-coverage-2024.v1.csv")

mean(vax$report_submitted)

vax |>
  mutate(month = format(period, "%Y-%m")) |>
  summarise(reporting_rate = mean(report_submitted), .by = month) |>
  arrange(month)

  5. Slide 5 / 22

    Reporting completeness

    MonthReporting rate
    January92%
    February82%
    March74%
    April–July82–87%
    August29%
    September45%
    October95%
    November–December76–87%
    Speaker notes
    August and September are not a wobble. Eleven of thirty-eight facilities reported in August and seventeen in September. Any August coverage figure is a statement about eleven facilities, and publishing it beside July's without saying so compares two different districts. Note also that the collapse and the recovery are district-wide and simultaneous. That is the signature of a system event — a supply of paper forms, a strike, a server, a supervisor who left — and not of facilities individually failing. A DQA that raises 27 separate facility findings for August has misread the pattern.

  6. Slide 6 / 22

    Break it down three ways, always — In Python

    by_facility = vax.groupby("facility_id")["reported"].mean().sort_values()
print(by_facility.head(6))
print(f"{(by_facility == 1).sum()} facilities reported every month")

by_type = vax.groupby("facility_type")["reported"].mean()
print((by_type * 100).round(1))
    Speaker notes
    One number hides everything worth knowing. Break the reporting rate by time, by reporting unit, and by unit type.

  7. Slide 7 / 22

    Break it down three ways, always — In R

    vax |> summarise(rate = mean(report_submitted), .by = facility_id) |> arrange(rate) |> head(6)

vax |> summarise(rate = mean(report_submitted), .by = facility_type)

  8. Slide 8 / 22

    Break it down three ways, always

    Facility typeReporting rate
    Health centre82.8%
    District hospital75.0%
    Health post71.7%
    Speaker notes
    The worst six facilities all sit at 58% — seven months of twelve — and only two of thirty-eight reported every month. The gradient by type is the actionable finding: health posts report worst, which points at supervision distance, staffing or transport rather than at any individual facility.

  9. Slide 9 / 22

    The denominator trap — In Python

    # WRONG: the share of reports that were submitted, among reports that were submitted
wrong = vax[vax["reported"]]["reported"].mean()          # always 1.0
    Speaker notes
    The one mistake that makes a reporting rate useless:

  10. Slide 10 / 22

    The denominator trap — In R

    # WRONG
vax |> filter(report_submitted) |> summarise(rate = mean(report_submitted))

  11. Slide 11 / 22

    The denominator trap — In Python

    EXPECTED_FACILITIES = pd.read_csv("facility-master-list.csv")["facility_id"]

expected = len(EXPECTED_FACILITIES) * vax["period"].nunique()
submitted = vax.loc[vax["reported"], ["facility_id", "period"]].drop_duplicates().shape[0]
print(f"{submitted} of {expected} expected facility-months")
    Speaker notes
    Obvious written out. It is not obvious when the filter is six lines earlier in a pipeline, and it is exactly what happens when someone reads a DHIS2 export that only contains submitted reports. If the extract has no rows for non-reporting units, the reporting rate cannot be computed from it at all — you need the list of expected units from somewhere else.

  12. Slide 12 / 22

    The denominator trap — In R

    expected  <- nrow(facility_master) * n_distinct(vax$period)
submitted <- vax |> filter(report_submitted) |> distinct(facility_id, period) |> nrow()
    Speaker notes
    The master list is the authority, exactly as the administrative population frame was in the previous course. A facility that closed and one that stopped reporting look identical in the extract and are completely different findings, and only the master list tells them apart.

  13. Slide 13 / 22

    Timeliness needs a field you may not have — In Python

    submissions["days_late"] = (
    submissions["submitted_on"] - submissions["deadline"]
).dt.days

print(f"on time: {(submissions['days_late'] <= 0).mean():.1%}")
print(f"median days late, of those late: "
      f"{submissions.loc[submissions['days_late'] > 0, 'days_late'].median():.0f}")
    Speaker notes
    Timeliness is the share of reports submitted by the deadline, and the median lateness of the rest.

  14. Slide 14 / 22

    Timeliness needs a field you may not have — In R

    submissions <- submissions |>
  mutate(days_late = as.integer(submitted_on - deadline))

c(on_time = mean(submissions$days_late <= 0),
  median_late = median(submissions$days_late[submissions$days_late > 0]))

  15. Slide 15 / 22

    Timeliness needs a field you may not have

    • This extract cannot support that calculation — and saying so is itself a DQA finding
    Speaker notes
    This extract cannot support that calculation, and saying so is itself a DQA finding. There is no submission timestamp in the file — only report_submitted, a flag with no date. DHIS2 records a lastUpdated per data value set and a completeness date per registration, so the field exists in the system and was not carried into the export. Write that up as it stands:

  16. Slide 16 / 22

    Timeliness needs a field you may not have

    Finding. Timeliness cannot be assessed. The monthly extract carries a submission flag but no submission date, although DHIS2 records one. Corrective action. Add the completeness date to the standard extract. Owner. HMIS focal point. By. Next quarterly extract.
    Speaker notes
    A dimension you cannot measure is a finding about the reporting system, not a gap in your assessment. Report it in the same table as the rest.

  17. Slide 17 / 22

    Read every other figure through these two

    • Coverage, at 76.5% reporting. A district total is a lower bound, not an estimate. Say "among reporting facilities"…
    • A month-on-month trend. August's fall is a reporting fall until proven otherwise. Plot the reporting rate on the…
    • A facility ranking. Facilities reporting seven months of twelve cannot be ranked against facilities reporting…
    Speaker notes
    The point of computing these first is what they license you to say afterwards.

  18. Slide 18 / 22

    Read every other figure through these two — In Python

    monthly = (
    vax[vax["reported"]]
    .groupby(vax["period"].dt.strftime("%Y-%m"))
    .apply(lambda g: g["doses_administered"].sum() / g["target_population"].sum())
    .rename("coverage")
    .to_frame()
)
monthly["reporting_rate"] = by_month
print(monthly.round(3))

  19. Slide 19 / 22

    Read every other figure through these two — In R

    vax |>
  mutate(month = format(period, "%Y-%m")) |>
  summarise(
    coverage = sum(doses_administered[report_submitted]) /
               sum(target_population[report_submitted]),
    reporting_rate = mean(report_submitted),
    .by = month
  )

  20. Slide 20 / 22

    Read every other figure through these two

    Two columns, always adjacent. A coverage figure without its reporting rate is an unlabelled number, and the label is the part that stops it being misread.

  21. Slide 21 / 22

    What comes next

    • Completeness and timeliness are about whether a report arrived.
    Speaker notes
    Completeness and timeliness are about whether a report arrived. The next lesson is about whether the report was right — the recount against the source register, the verification factor it produces, and the tolerance band that turns it into a decision.

  22. Slide 22 / 22

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson