Back to the lesson·Lesson 1 of 8·From intention to indicator
What breaks between the change and the number
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- Three documents, two translations
- The first translation: change to row
- Fix it at the LogFrame, not at the analysis
- The second translation: row to arithmetic
- Assumptions are the part everyone deletes
- What the levels mean, and where people cheat
- Read the chain backwards as a test
- What comes next
Speaker notes
A Theory of Change says how the world is supposed to move; a LogFrame turns that into rows; an indicator turns a row into arithmetic. Something is lost at each translation, and knowing what lets you defend the number.Three documents, two translations
- A Theory of Change — a narrative and usually a diagram, saying how the programme believes the world will move and…
- A LogFrame — a table of goal, outcomes, outputs and activities, each row carrying indicators, means of verification…
- An indicator definition — arithmetic on a dataset.
Speaker notes
Almost every programme in this sector carries three artefacts, produced by different people at different times, and the number you report comes out of the third. Each arrow between them is a translation, and translations lose things. This lesson is about what specifically, because "the indicator does not measure what we care about" is a complaint that arrives too late to fix.The first translation: change to row
If health workers are trained and vaccines are reliably in stock, then caregivers who bring a child for the first dose will complete the schedule, and fewer children will be susceptible to measles at school entry.
The first translation: change to row
Level Indicator Output Number of children receiving penta3 Speaker notes
A Theory of Change for a routine immunisation programme might say: That sentence contains a mechanism, an assumption and a population. The LogFrame row it becomes usually says:The first translation: change to row
- The mechanism — The theory was about completion — children who start the schedule finishing it
- The assumption — "Vaccines reliably in stock" is a condition the theory depends on and no LogFrame row measures
- The population — "Children who are brought for a first dose" is a specific group
Speaker notes
Three things went missing in one step. The mechanism. The theory was about completion — children who start the schedule finishing it. A count of penta3 doses rises if more children start, if more complete, or if a campaign draws children in from a neighbouring district. The indicator cannot distinguish the thing the programme is actually claiming to do. The assumption. "Vaccines reliably in stock" is a condition the theory depends on and no LogFrame row measures. When the indicator underperforms, the argument about why is unresolvable, because nobody was counting the thing the theory said mattered. The population. "Children who are brought for a first dose" is a specific group. A dose count has no denominator at all.Fix it at the LogFrame, not at the analysis — In Python
totals = ( vax[vax["reported"]] .groupby("antigen")["doses_administered"].sum() ) dropout = (totals["penta1"] - totals["penta3"]) / totals["penta1"] print(f"penta1 {totals['penta1']:,} penta3 {totals['penta3']:,} dropout {dropout:.1%}")Speaker notes
The completion mechanism is measurable, and it is a standard indicator:Fix it at the LogFrame, not at the analysis — In R
totals <- vax |> filter(report_submitted) |> summarise(doses = sum(doses_administered), .by = antigen) p1 <- totals$doses[totals$antigen == "penta1"] p3 <- totals$doses[totals$antigen == "penta3"] c(penta1 = p1, penta3 = p3, dropout = (p1 - p3) / p1)Fix it at the LogFrame, not at the analysis
- When the mechanism is measurable, measure the mechanism — A dropout rate and a coverage figure answer different…
Speaker notes
8,302 first doses, 7,160 third doses, a dropout of 13.8%. And the measles series is worse: 7,321 first doses against 5,647 second, a dropout of 22.9%. That single number speaks directly to the theory in a way the dose count does not. It is also robust to the denominator argument — both numerator and denominator come from the same reporting facilities in the same months, so a census projection nine years old cannot move it. When the mechanism is measurable, measure the mechanism. A dropout rate and a coverage figure answer different questions, and a LogFrame carrying only the second is describing the programme's output rather than its claim.The second translation: row to arithmetic
- Children or doses? A dose count and a child count differ whenever a child is recorded twice, and the register does…
- Which period — by date of vaccination or date of report?
- Which facilities — those that reported, or all of them, with non-reporters counted as zero?
- Catchment or residence? A child from the next district vaccinated here.
Speaker notes
The LogFrame says "number of children receiving penta3". The analyst has to turn that into code, and the row does not say: Four questions, each with a defensible answer, and two analysts answering them differently produce two different numbers from the same file. Neither has made an error. This is the defect the whole course exists to prevent, and the fix is the reference sheet in unit 2.Assumptions are the part everyone deletes
- Give the important assumptions an indicator of their own — Stock-out days per quarter, road access days, staff vacancy…
Speaker notes
The right-hand column of a LogFrame is the one that gets emptied first, and it is the column that explains your results. An assumption is a condition outside the programme's control that the causal chain depends on: vaccines arrive, the road is passable, the currency holds, the security situation permits movement. When a target is missed, the honest analysis is usually "the assumption failed", and that sentence is only available if somebody wrote the assumption down beforehand. Give the important assumptions an indicator of their own. Stock-out days per quarter, road access days, staff vacancy rate. They are cheap to collect and they convert an argument into a fact.What the levels mean, and where people cheat
Level Question Attribution Activity What did we do? Fully ours Output What did that produce, directly? Fully ours Outcome What changed for people? Shared with everything else Goal What changed in the population? Almost none of it ours Speaker notes
The universal temptation is to report an output and call it an outcome, because outputs are attributable and outcomes are not. "12,000 children vaccinated" is an output. "Measles susceptibility at school entry fell" is an outcome, it depends on four other actors, and attribution is exactly what the Impact Evaluation Methods course later spends twenty-four hours on. Say which level you are at. A report that lists outputs under an outcome heading will be found out by any evaluator, and the finding is about credibility rather than about numbers.Read the chain backwards as a test
Penta3 dropout rate — measures whether children who start the schedule finish it — which the programme influences through reminders, outreach and stock — which the theory says reduces susceptibility — which is the change we want.
Read the chain backwards as a test
- The indicator does not measure the mechanism. Change the indicator.
- The mechanism is not something the programme influences. Change the programme's claim, or accept that you are…
- Nobody can name the change. Stop and have that conversation before collecting anything.
Speaker notes
The most useful five minutes in indicator design: start from your indicator and walk up. If the chain breaks, you have found something. It usually breaks in one of three places, and each has a different fix:Read the chain backwards as a test
An indicator is a claim about causation compressed into arithmetic. The compression is lossy, and the reference sheet is where you record what was lost.
What comes next
- You now know where an indicator comes from and what it inherits.
Speaker notes
You now know where an indicator comes from and what it inherits. The next lesson takes it apart — what an indicator is made of, why almost every one is a ratio, and the three questions to ask before writing a single line of code against it.