A seven-point gain that was the school year

One question, three answers

import pandas as pd
import numpy as np

assessment = pd.read_csv("learning-assessment-2024.v1.csv")
assessment["pct"] = assessment["raw_score"] / assessment["items"]
literacy = (assessment[assessment["domain"] == "literacy"]
            .pivot_table(index="student_id", columns="assessment_round",
                         values="pct").dropna())

roster = pd.read_csv("school-roster-2024.v1.csv").drop_duplicates(
    "student_id", keep="last")
d = literacy.join(roster.set_index("student_id"), how="inner")

print(d.groupby("feeding_programme")[["baseline", "endline"]].mean().round(4))
print(f"overall gain: {(d['endline'] - d['baseline']).mean():+.4f}")

library(dplyr)

d |> summarise(baseline = mean(baseline), endline = mean(endline),
               .by = feeding_programme)

	Baseline	Endline	Gain	n
Feeding programme	57.75%	64.41%	+6.66 pts	398
No programme	54.67%	62.29%	+7.62 pts	187
All students	56.76%	63.73%	+6.97 pts	585

Three claims can be built from that table and two of them are wrong.

“The programme delivered a 7.0-point gain.” Before-after, paired, 95% CI +6.1 to +7.9, t = 15.2. Decisive, reproducible, and it attributes to the meals a year of schooling that every child had.

“Schools with the programme are 2.1 points ahead.” Endline cross-section, 95% CI −1.2 to +5.4. It compares two groups that were 3.1 points apart before the programme was measured at all.

“The gain was 0.96 points lower in schools with the programme.” Both groups against their own baselines, 95% CI −3.5 to +1.6 once the 24 schools are accounted for. This is the one that is trying to answer the question, and its answer is that there is nothing here.

The counterfactual, stated in one sentence

Every impact claim has a hidden second half.

Literacy in the feeding schools rose 6.66 points compared with what would have happened without the programme.

The second clause is the counterfactual, and it is never observed. The same children in the same year without meals is not a thing any dataset contains, so an evaluation is always an argument about what to substitute for it.

Claim	Counterfactual it assumes	Is that plausible?
Before-after	The same children would not have changed at all	No — a school year happened
Endline cross-section	The two groups would have been identical	No — they differed by 3.1 points at baseline
Difference-in-differences	The two groups would have changed by the same amount	Arguable, and lesson 3 is that argument

Naming the counterfactual is the whole method. Once it is written down, the before-after claim collapses without any statistics: it assumes children learn nothing in a year they are in school.

Why the before-after number is so persuasive

It is precise, it is large, and it is easy to compute — three properties that have nothing to do with being right.

gain = d["endline"] - d["baseline"]
se = gain.std(ddof=1) / np.sqrt(len(gain))
print(f"{gain.mean():+.4f}  95% CI [{gain.mean() - 1.96*se:+.4f},"
      f" {gain.mean() + 1.96*se:+.4f}]  t = {gain.mean()/se:.1f}")

t.test(d$endline - d$baseline)

t = 15.2. Every gate in the statistics course passes. The interval is narrow, the effect size is large, the sample is adequate, and the p-value is far below any threshold.

None of those checks can detect a missing comparison group, and that is the thing worth carrying out of this lesson. Statistical rigour applied to the wrong counterfactual produces a confident wrong answer, and confidence is what gets it into a proposal.

Three things a before-after change contains

Whenever you see one, it is a sum, and only the last term is the programme.

The passage of time. Children get older and are taught. Prices rise. Water points age. Here it is the entire 7 points.

Regression to the mean. Whoever was selected for being worst off will look better next round even if nothing was done, because part of “worst off” was measurement noise. The nutrition course’s admission MUAC is the standing example.

Everything else that happened. A drought, a currency movement, another agency’s programme in the same district, a change in how the indicator is recorded.

by_school = d.groupby(["school_id", "feeding_programme"])[
    ["baseline", "endline"]].mean()
by_school["gain"] = by_school["endline"] - by_school["baseline"]
print(by_school.groupby("feeding_programme")["gain"].describe()[
    ["mean", "min", "max"]].round(4))

by_school |> summarise(mean = mean(gain), min = min(gain), max = max(gain),
                       .by = feeding_programme)

Every one of the 24 schools gained, from +1.2 points to +12.2. A programme that had done nothing at all would have produced a table of positive numbers.

When a before-after comparison is legitimate

It is not always wrong, and the cases where it holds are worth being precise about.

When the counterfactual is genuinely flat and you can show it. Three or more pre-programme rounds with no trend is an argument; one baseline is not.

When the outcome cannot change on its own. A latrine does not build itself. A water point does not become chlorinated without someone doing it. Coverage of a thing only your programme supplies has a defensible zero counterfactual.

When you are monitoring, not evaluating. “Attendance rose from 84% to 88%” is a true statement about the world and a useful one to track. It becomes a claim about the programme only when someone writes “as a result of”.

The sentence to watch for is “as a result of”. It is where a monitoring number becomes an impact claim, and it is usually added by someone who was not in the analysis.

Report it whole

Literacy outcomes, baseline to endline 2024

  All students        56.8% to 63.7%    +6.97 points   95% CI +6.1 to +7.9
    With feeding      57.8% to 64.4%    +6.66 points
    Without feeding   54.7% to 62.3%    +7.62 points

  The overall change is reported as a monitoring result and is not
  attributed to the feeding programme. Schools without the programme gained
  slightly more, so the difference-in-differences estimate of the programme
  effect is -0.96 points (95% CI -3.5 to +1.6, clustered on 24 schools).

  The before-after change assumes children would have learned nothing over
  a school year. That counterfactual is not defensible here and the
  comparison group is what replaces it.

The last paragraph is three lines and it is the difference between a monitoring report and an impact claim. Writing the counterfactual down is what forces the choice, and a report that never writes one has made the choice by default.

What comes next

The difference-in-differences estimate above rests on the two groups of schools being comparable in the ways that matter. The next lesson tests that directly, on the fifteen and the nine, and finds a standardised difference eight times what a randomised trial would tolerate.