Back to the lesson·Lesson 1 of 8·What quality means here
Five dimensions, five measures
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- The sentence to stop saying
- The five, and the measure for each
- Compute all five on one extract
- Weight the dimensions by what they cost
- Do not collapse it to one number
- What comes next
Speaker notes
Calling the data poor is not a finding. Accuracy, completeness, timeliness, consistency and integrity — each with a number attached, computed on the same extract, and a scorecard that survives being disagreed with.The sentence to stop saying
- "The data quality is poor." Everyone nods, nothing happens, and the same sentence is said next quarter.
Speaker notes
"The data quality is poor." Everyone nods, nothing happens, and the same sentence is said next quarter. It fails because it is unactionable in three separate ways. It does not say which quality — a late report and a fabricated report are different problems with different fixes. It does not say how much — poor everywhere or poor in four facilities. And it does not say so what — whether the defect is large enough to change the number anyone is going to act on. The five dimensions exist to fix all three at once. They are not a framework to recite; they are five questions, each of which has a number as its answer.The five, and the measure for each
Dimension The question The measure Completeness Did everyone report, on everything? Reporting rate; missing-value rate per field Timeliness Did the report arrive in time to be used? Share submitted by the deadline; median days late Accuracy Does the reported figure match the source? Verification factor: recount over reported Consistency Do the numbers agree with each other and with last month? Rate of failed internal rules; outlier rate Integrity Was the value produced by measurement or by something else? Digit preference; heaping; implausibly stable series The five, and the measure for each
- Accuracy is the only one that needs the source document — The other four are computable from the extract you already…
- Integrity is not an accusation — It measures whether numbers look like measurements
Speaker notes
Two things about that table are worth arguing over, because both come up. Accuracy is the only one that needs the source document. The other four are computable from the extract you already have, at your desk, this afternoon. That asymmetry drives the whole design of a DQA: you use the four desk dimensions to decide where to spend the expensive fifth. Integrity is not an accusation. It measures whether numbers look like measurements. A heaped age distribution usually means nobody asked for a birth certificate, not that anybody invented anything, and the lesson on it spends most of its time on that distinction.Compute all five on one extract — In Python (cont.)
import pandas as pd vax = pd.read_csv("vaccination-coverage-2024.v1.csv", parse_dates=["period"]) vax["reported"] = vax["report_submitted"] == True scorecard = {} # Completeness scorecard["reporting_rate"] = vax["reported"].mean() scorecard["field_completeness"] = 1 - vax.isna().mean().mean() # Consistency: doses cannot exceed the target population rule_fail = (vax["doses_administered"] > vax["target_population"]).mean() scorecard["rule_failure_rate"] = rule_fail # Integrity: share of reported values ending in zeroSpeaker notes
Take the routine vaccination extract — 38 facilities, twelve months, six antigens.Compute all five on one extract — In Python (cont.)
reported = vax[vax["reported"]] scorecard["round_number_share"] = (reported["doses_administered"] % 10 == 0).mean() print(pd.Series(scorecard).round(3))Compute all five on one extract — In R
library(dplyr) library(readr) vax <- read_csv("vaccination-coverage-2024.v1.csv") scorecard <- tibble::tibble( reporting_rate = mean(vax$report_submitted), field_completeness = 1 - mean(is.na(as.matrix(vax))), rule_failure_rate = mean(vax$doses_administered > vax$target_population), round_number_share = mean( vax$doses_administered[vax$report_submitted] %% 10 == 0 ) ) round(scorecard, 3)Compute all five on one extract
Measure Value Read as Reporting rate 76.5% 642 of 2,736 facility-months carry no report Field completeness 100% no blank cells — which is not the same as no missing data Rule failure rate 0% no facility reports more doses than its target population Round-number share 10.4% what chance produces; nothing to see Speaker notes
Look at rows two and one together. Field completeness is 100% and reporting completeness is 76.5%, and only the second one matters. Every non-reporting facility-month is present in the file as a row of zeroes with a flag, so a blank-cell count says the data is perfect. This is the defect the whole module keeps circling: a missing report that arrives as a zero.Weight the dimensions by what they cost
- For a coverage figure, completeness dominates. A 76.5% reporting rate makes any district total a lower bound, and…
- For a caseload figure used for procurement, accuracy dominates. Ordering therapeutic food against an over-reported…
- For an early warning indicator, timeliness dominates. A perfectly accurate report arriving six weeks after the…
- Say which dimension you weighted and why, in the report — A composite quality score that hides the weighting is the…
Speaker notes
A scorecard with five equal numbers implies the five matter equally. They do not, and which matters most depends entirely on what the data is for. Say which dimension you weighted and why, in the report. A composite quality score that hides the weighting is the same defect as an indicator that hides its denominator.Do not collapse it to one number — In Python
scorecard_table = pd.DataFrame({ "dimension": ["Completeness", "Timeliness", "Accuracy", "Consistency", "Integrity"], "measure": ["Reporting rate", "Submitted by deadline", "Verification factor", "Rule failure rate", "Round-number share"], "value": [0.765, None, None, 0.0, 0.104], "source": ["extract", "submission log", "facility visit", "extract", "extract"], }) print(scorecard_table)Speaker notes
Somebody will ask for a single score out of 100. Resist, and offer the scorecard instead, for a specific reason: the five dimensions have different fixes, and the single number tells you nothing about which one to apply. A district at 82% because it is late is fixed by moving a deadline. A district at 82% because a third of its facilities never report is fixed by finding out why they stopped. Both score 82. Only one of them is fixed by a training workshop, and neither is fixed by the workshop somebody will propose. If a composite is genuinely required — some donor templates demand it — publish it beside the components, never instead of them.Do not collapse it to one number — In R
scorecard_table <- tibble::tribble( ~dimension, ~measure, ~value, ~source, "Completeness", "Reporting rate", 0.765, "extract", "Timeliness", "Submitted by deadline", NA, "submission log", "Accuracy", "Verification factor", NA, "facility visit", "Consistency", "Rule failure rate", 0.000, "extract", "Integrity", "Round-number share", 0.104, "extract" )Do not collapse it to one number
- The
sourcecolumn is the useful one — It says immediately which numbers you can produce today and which need someone…
Speaker notes
Thesourcecolumn is the useful one. It says immediately which numbers you can produce today and which need someone to travel, and that is the shape of every DQA plan.- The
What comes next
- Two of the five dimensions are almost never reported at all, and both are computable from data you already hold.
Speaker notes
Two of the five dimensions are almost never reported at all, and both are computable from data you already hold. The next lesson computes them properly — reporting completeness and timeliness, each with its own denominator, and the trap of dividing by the facilities that reported.