Exercise · Advanced
The slide with one number on it
A board pack says the feeding programme raised literacy by 7 points and recommends doubling it. The number is right, the counterfactual is missing, and you have an hour before the meeting to work out what to say instead.
The board pack goes out tomorrow. Slide 4 reads:
School feeding works. Literacy scores in programme schools rose from 57.8% to 64.4% over the school year — a gain of 6.6 percentage points. We recommend doubling the programme’s coverage in 2025.
Every number on that slide is correct. You have an hour.
The files
learning-assessment-2024.v1.csv and school-roster-2024.v1.csv.
What to do
One: find the missing half. Compute the same change for the schools without the programme. Report both, with the difference between them.
Two: put an interval on the thing the slide is actually claiming. The claim is about the programme, not about the year, so the estimate is the difference-in-differences. Cluster the standard error on school and say why.
Three: work out whether the study could have found what the board wants to know. Doubling coverage is a decision worth making if the effect is around 3 points. Compute the minimum detectable effect for 15 versus 9 schools with about 50 children each, using an intra-cluster correlation of 0.065, and compare.
Four: rewrite the slide. One heading, at most four lines, no number that is not what it appears to be.
Check yourself
| Expected | |
|---|---|
| Programme schools, baseline to endline | 57.75% to 64.41%, +6.66 pts |
| Comparison schools | 54.67% to 62.29%, +7.62 pts |
| Difference-in-differences | −0.96 pts |
| Clustered 95% interval | −3.48 to +1.56 |
| Minimum detectable effect | about 4.9 points |
| Schools needed for a 3-point effect | about 60 |
If your comparison-school gain comes out below the programme gain, check the roster join — two students appear twice and fan out if the duplicates are not dropped.
Three questions
Two sentences each.
1. The slide’s 6.6 points is correct and the comparison schools gained 7.6. Write the single sentence you would say out loud in the meeting, before any statistics, that makes the problem clear to a board member.
2. Your difference-in-differences interval runs from −3.5 to +1.6 and the board cares about 3 points. Explain what that interval does and does not rule out, without using the word “significant”.
3. The recommendation is to double the programme. Say what you would recommend instead, and name the one change to next year’s roll-out that would make the 2026 evaluation able to answer this question.
The point
The number was never wrong. No amount of checking the arithmetic, the sample size, the p-value or the data quality would have caught this, because nothing was wrong with any of them — the slide compares programme schools to themselves a year earlier and calls the school year an effect.
A missing counterfactual is invisible to every statistical check. It is caught by one question, asked before any analysis: compared to what?
The honest slide is shorter than the original and less pleasant to present. It is also the one that survives the first board member who asks what happened in the schools without the programme.