What breaks between the change and the number

Three documents, two translations

Almost every programme in this sector carries three artefacts, produced by different people at different times, and the number you report comes out of the third.

• A Theory of Change — a narrative and usually a diagram, saying how the programme believes the world will move and under what assumptions.
• A LogFrame — a table of goal, outcomes, outputs and activities, each row carrying indicators, means of verification and assumptions.
• An indicator definition — arithmetic on a dataset.

Each arrow between them is a translation, and translations lose things. This lesson is about what specifically, because “the indicator does not measure what we care about” is a complaint that arrives too late to fix.

The first translation: change to row

A Theory of Change for a routine immunisation programme might say:

If health workers are trained and vaccines are reliably in stock, then caregivers who bring a child for the first dose will complete the schedule, and fewer children will be susceptible to measles at school entry.

That sentence contains a mechanism, an assumption and a population. The LogFrame row it becomes usually says:

Level	Indicator
Output	Number of children receiving penta3

Three things went missing in one step.

The mechanism. The theory was about completion — children who start the schedule finishing it. A count of penta3 doses rises if more children start, if more complete, or if a campaign draws children in from a neighbouring district. The indicator cannot distinguish the thing the programme is actually claiming to do.

The assumption. “Vaccines reliably in stock” is a condition the theory depends on and no LogFrame row measures. When the indicator underperforms, the argument about why is unresolvable, because nobody was counting the thing the theory said mattered.

The population. “Children who are brought for a first dose” is a specific group. A dose count has no denominator at all.

Fix it at the LogFrame, not at the analysis

The completion mechanism is measurable, and it is a standard indicator:

totals = (
    vax[vax["reported"]]
    .groupby("antigen")["doses_administered"].sum()
)
dropout = (totals["penta1"] - totals["penta3"]) / totals["penta1"]
print(f"penta1 {totals['penta1']:,}  penta3 {totals['penta3']:,}  dropout {dropout:.1%}")

totals <- vax |>
  filter(report_submitted) |>
  summarise(doses = sum(doses_administered), .by = antigen)

p1 <- totals$doses[totals$antigen == "penta1"]
p3 <- totals$doses[totals$antigen == "penta3"]
c(penta1 = p1, penta3 = p3, dropout = (p1 - p3) / p1)

8,302 first doses, 7,160 third doses, a dropout of 13.8%. And the measles series is worse: 7,321 first doses against 5,647 second, a dropout of 22.9%.

That single number speaks directly to the theory in a way the dose count does not. It is also robust to the denominator argument — both numerator and denominator come from the same reporting facilities in the same months, so a census projection nine years old cannot move it.

When the mechanism is measurable, measure the mechanism. A dropout rate and a coverage figure answer different questions, and a LogFrame carrying only the second is describing the programme’s output rather than its claim.

The second translation: row to arithmetic

The LogFrame says “number of children receiving penta3”. The analyst has to turn that into code, and the row does not say:

• Children or doses? A dose count and a child count differ whenever a child is recorded twice, and the register does not distinguish them.
• Which period — by date of vaccination or date of report?
• Which facilities — those that reported, or all of them, with non-reporters counted as zero?
• Catchment or residence? A child from the next district vaccinated here.

Four questions, each with a defensible answer, and two analysts answering them differently produce two different numbers from the same file. Neither has made an error. This is the defect the whole course exists to prevent, and the fix is the reference sheet in unit 2.

Assumptions are the part everyone deletes

The right-hand column of a LogFrame is the one that gets emptied first, and it is the column that explains your results.

An assumption is a condition outside the programme’s control that the causal chain depends on: vaccines arrive, the road is passable, the currency holds, the security situation permits movement. When a target is missed, the honest analysis is usually “the assumption failed”, and that sentence is only available if somebody wrote the assumption down beforehand.

Give the important assumptions an indicator of their own. Stock-out days per quarter, road access days, staff vacancy rate. They are cheap to collect and they convert an argument into a fact.

What the levels mean, and where people cheat

Level	Question	Attribution
Activity	What did we do?	Fully ours
Output	What did that produce, directly?	Fully ours
Outcome	What changed for people?	Shared with everything else
Goal	What changed in the population?	Almost none of it ours

The universal temptation is to report an output and call it an outcome, because outputs are attributable and outcomes are not. “12,000 children vaccinated” is an output. “Measles susceptibility at school entry fell” is an outcome, it depends on four other actors, and attribution is exactly what the Impact Evaluation Methods course later spends twenty-four hours on.

Say which level you are at. A report that lists outputs under an outcome heading will be found out by any evaluator, and the finding is about credibility rather than about numbers.

Read the chain backwards as a test

The most useful five minutes in indicator design: start from your indicator and walk up.

Penta3 dropout rate — measures whether children who start the schedule finish it — which the programme influences through reminders, outreach and stock — which the theory says reduces susceptibility — which is the change we want.

If the chain breaks, you have found something. It usually breaks in one of three places, and each has a different fix:

• The indicator does not measure the mechanism. Change the indicator.
• The mechanism is not something the programme influences. Change the programme’s claim, or accept that you are monitoring context.
• Nobody can name the change. Stop and have that conversation before collecting anything.

An indicator is a claim about causation compressed into arithmetic. The compression is lossy, and the reference sheet is where you record what was lost.

What comes next

You now know where an indicator comes from and what it inherits. The next lesson takes it apart — what an indicator is made of, why almost every one is a ratio, and the three questions to ask before writing a single line of code against it.