Back to the lesson·Lesson 3 of 8·Periods and completeness
Periods, and the aggregation operator nobody sets
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- Period types, and the identifier that carries them
- Which period does a value belong to?
- The three annual figures
- Why they differ
- The aggregation operator
- Deadlines, expiry and locking
- Completeness is a separate record
- What comes next
Speaker notes
One file, one indicator, three annual coverage figures — 6.5%, 60.8% and 77.5% — all from defensible-sounding sentences. The difference is how the denominator aggregates over time.Period types, and the identifier that carries them — Example
202408 monthly 2024W32 weekly 2024Q3 quarterly 2024 yearly 2024April financial year starting AprilSpeaker notes
Every value belongs to a period, and the period type is a property of the dataset the value was collected on. DHIS2's period identifiers encode the type in the string, which is worth adopting even outside DHIS2:Period types, and the identifier that carries them — In Python
import pandas as pd vax = pd.read_csv("vaccination-coverage-2024.v1.csv", parse_dates=["period"]) vax["period_id"] = vax["period"].dt.strftime("%Y%m") print(sorted(vax["period_id"].unique())[:4])Speaker notes
They sort correctly, they cannot be ambiguous between conventions, and the type is readable without a schema. The joining course made the case for ISO period keys; this is the same argument with the system's own vocabulary.Period types, and the identifier that carries them — In R
library(dplyr) vax <- vax |> mutate(period_id = format(period, "%Y%m"))Which period does a value belong to?
- Right: the period the activity happened in. A dose given on 28 August is August, however late the form arrives.
- Wrong: the period the form was submitted in. Common where data entry is batched, and it shifts activity forward by…
- Wrong: the period the value was keyed in. Same failure, one step further.
Speaker notes
The question has one right answer and two wrong ones in daily use. DHIS2 stores the value against the period the data entry form was opened for, so the system gets this right if the person opened the right form. The failure is human and it shows up as a characteristic pattern: a low month followed by a high one, which is the spike-and-dip the data quality course taught you to check for.The three annual figures — In Python
penta3 = vax[vax["antigen"] == "penta3"] reported = penta3[penta3["report_submitted"] == True] doses = reported["doses_administered"].sum() summed_denominator = reported["target_population"].sum() annual_target = penta3.groupby("facility_id")["target_population"].first().sum() print(f"doses: {doses:,}") print(f"a) doses / summed monthly target : {doses / summed_denominator:.1%}") print(f"b) doses / annual target, all 38 : {doses / annual_target:.1%}")Speaker notes
Now the lesson. Take penta3 for the year and ask for annual coverage.The three annual figures — In R
penta3 <- vax |> filter(antigen == "penta3") reported <- penta3 |> filter(report_submitted) doses <- sum(reported$doses_administered) summed <- sum(reported$target_population) annual <- penta3 |> summarise(t = first(target_population), .by = facility_id) |> summarise(sum(t)) |> pull() c(a = doses / summed, b = doses / annual)The three annual figures
Sentence Figure "Doses over target population, summed across the year" 6.5% "Annual doses over the annual target population" 60.8% "Annual doses over the annual target, among reporting facility-months" 77.5% Speaker notes
Three numbers, one file, one indicator, and each comes out of a sentence somebody would say in a meeting without blinking.Why they differ
- 6.5% is a monthly figure wearing an annual label —
target_populationis an annual cohort — the surviving infants in… - 60.8% counts silent facilities as having vaccinated nobody — The denominator is every facility's full annual cohort,…
- 77.5% is the defensible one — and it needs an explicit denominator adjustment:
Speaker notes
6.5% is a monthly figure wearing an annual label.target_populationis an annual cohort — the surviving infants in the catchment for the year — and the extract repeats it in every month. Summing it across twelve months produces a denominator twelve times too large. The result is the average monthly coverage, and multiplying it by twelve gets you back to 78%. 60.8% counts silent facilities as having vaccinated nobody. The denominator is every facility's full annual cohort, and the numerator is only the doses from facility-months that reported. 107 of 456 penta3 facility-months are missing, and this figure attributes zero doses to all of them. 77.5% is the defensible one, and it needs an explicit denominator adjustment:- 6.5% is a monthly figure wearing an annual label —
Why they differ — In Python
months_reported = reported.groupby("facility_id").size() targets = penta3.groupby("facility_id")["target_population"].first() prorated = (targets * months_reported.reindex(targets.index).fillna(0) / 12).sum() print(f"c) pro-rated denominator: {doses / prorated:.1%}")Why they differ — In R
months <- reported |> summarise(m = n(), .by = facility_id) targets <- penta3 |> summarise(t = first(target_population), .by = facility_id) prorated <- targets |> left_join(months, by = "facility_id") |> mutate(m = coalesce(m, 0)) |> summarise(sum(t * m / 12)) |> pull() doses / proratedSpeaker notes
Each facility contributes the share of its annual cohort matching the months it actually reported. It is coverage among reporting facility-months, and the label must say so.The aggregation operator
Operator Meaning Right for Sum Add across periods Counts of events: doses, admissions, consultations Average Mean across periods Stock levels, staffing, anything that is a state not an event Last value Take the most recent Populations, targets, register sizes Speaker notes
Underneath all three figures is one configuration setting most analysts never see: how a data element aggregates over periods.The aggregation operator
- Ask for the aggregation operator with the metadata — It is one field per data element, it is invisible in the export,…
Speaker notes
doses_administeredsums.target_populationmust not — it is a state, and it sums to twelve times itself. In a real instance the element would be configured as average or last value, and requesting annual coverage would then work. Here it is a repeated column and the correction is yours to make. Ask for the aggregation operator with the metadata. It is one field per data element, it is invisible in the export, and it is the difference between 6.5% and 78%. The same setting exists for aggregation across org units, where "sum" is almost always right and "average" is almost always wrong — a district's doses are the sum of its facilities', not their mean.Deadlines, expiry and locking
- The deadline is when the dataset is due. It drives the timeliness figure the DQA course computes.
- Expiry days lock the form a fixed number of days after the period ends. After that, entry needs an unlock.
- A lock exception unlocks one dataset, one org unit, one period. Every one is a decision, and a system with hundreds…
Speaker notes
Three settings that determine whether a value can still change. The consequence for an analyst: an extract of a recent period is provisional. Pull August in September and again in November and the numbers will differ, legitimately, because late entry is still arriving. That is not a data quality problem and it must not be reported as one — it is the reason the next lesson but one records an extraction date with every pull.Completeness is a separate record
- Completeness in DHIS2 is a registration, not a computation — When a user clicks "complete" on a data entry form, the…
Speaker notes
The last structural point, and it explains a number that otherwise looks inconsistent. Completeness in DHIS2 is a registration, not a computation. When a user clicks "complete" on a data entry form, the system stores a completeness record for that dataset, org unit and period. The reporting rate is built from those records. Which means a dataset can be complete with no values — somebody clicked complete on an empty form — and full of values but not complete — data entered, nobody clicked. Both happen constantly.Completeness is a separate record — In Python
grid = (vax["facility_id"].nunique() * vax["period"].nunique() * vax["antigen"].nunique()) print(f"{grid} rows expected, {len(vax)} present, " f"{(vax['report_submitted'] == True).sum()} flagged reported")Completeness is a separate record — In R
c(expected = n_distinct(vax$facility_id) * n_distinct(vax$period) * n_distinct(vax$antigen), present = nrow(vax), reported = sum(vax$report_submitted))Speaker notes
2,736 rows present and 2,094 flagged as reported. The row exists either way, which is exactly the shape the cleaning course warned about — a missing report arriving as a zero with a flag beside it.What comes next
- The reporting flag is the raw material of the completeness figure, and the next lesson turns it into a denominator.
Speaker notes
The reporting flag is the raw material of the completeness figure, and the next lesson turns it into a denominator. The DQA course already showed why that matters; this one shows how the system computes it, and the two ways its answer differs from yours.