A seven-point gain that was the school year

Slide 1 / 21

What this lesson covers

• One question, three answers
• The counterfactual, stated in one sentence
• Why the before-after number is so persuasive
• Three things a before-after change contains
• When a before-after comparison is legitimate
• Report it whole
• What comes next

Slide 2 / 21

One question, three answers — In Python

import pandas as pd
import numpy as np

assessment = pd.read_csv("learning-assessment-2024.v1.csv")
assessment["pct"] = assessment["raw_score"] / assessment["items"]
literacy = (assessment[assessment["domain"] == "literacy"]
            .pivot_table(index="student_id", columns="assessment_round",
                         values="pct").dropna())

roster = pd.read_csv("school-roster-2024.v1.csv").drop_duplicates(
    "student_id", keep="last")
d = literacy.join(roster.set_index("student_id"), how="inner")

print(d.groupby("feeding_programme")[["baseline", "endline"]].mean().round(4))
print(f"overall gain: {(d['endline'] - d['baseline']).mean():+.4f}")

Slide 3 / 21

One question, three answers — In R

library(dplyr)

d |> summarise(baseline = mean(baseline), endline = mean(endline),
               .by = feeding_programme)

Slide 4 / 21

One question, three answers

Slide 5 / 21

One question, three answers

• "The programme delivered a 7.0-point gain." — Before-after, paired, 95% CI +6.1 to +7.9, t = 15.2
• "Schools with the programme are 2.1 points ahead." — Endline cross-section, 95% CI −1.2 to +5.4
• "The gain was 0.96 points lower in schools with the programme." — Both groups against their own baselines, 95% CI −3.5…

Slide 6 / 21

The counterfactual, stated in one sentence

Literacy in the feeding schools rose 6.66 points compared with what would have happened without the programme.

Slide 7 / 21

The counterfactual, stated in one sentence

• The second clause is the counterfactual, and it is never observed — The same children in the same year without meals is…

Slide 8 / 21

The counterfactual, stated in one sentence

Slide 9 / 21

The counterfactual, stated in one sentence

• Naming the counterfactual is the whole method — Once it is written down, the before-after claim collapses without any…

Slide 10 / 21

Why the before-after number is so persuasive — In Python

gain = d["endline"] - d["baseline"]
se = gain.std(ddof=1) / np.sqrt(len(gain))
print(f"{gain.mean():+.4f}  95% CI [{gain.mean() - 1.96*se:+.4f},"
      f" {gain.mean() + 1.96*se:+.4f}]  t = {gain.mean()/se:.1f}")

Slide 11 / 21

Why the before-after number is so persuasive — In R

t.test(d$endline - d$baseline)

Slide 12 / 21

Why the before-after number is so persuasive

• t = 15.2 — Every gate in the statistics course passes
• None of those checks can detect a missing comparison group — and that is the thing worth carrying out of this lesson

Slide 13 / 21

Three things a before-after change contains

• The passage of time — Children get older and are taught
• Regression to the mean — Whoever was selected for being worst off will look better next round even if nothing was done,…
• Everything else that happened — A drought, a currency movement, another agency's programme in the same district, a…

Slide 14 / 21

Three things a before-after change contains — In Python

by_school = d.groupby(["school_id", "feeding_programme"])[
    ["baseline", "endline"]].mean()
by_school["gain"] = by_school["endline"] - by_school["baseline"]
print(by_school.groupby("feeding_programme")["gain"].describe()[
    ["mean", "min", "max"]].round(4))

Slide 15 / 21

Three things a before-after change contains — In R

by_school |> summarise(mean = mean(gain), min = min(gain), max = max(gain),
                       .by = feeding_programme)

Slide 16 / 21

Three things a before-after change contains

• Every one of the 24 schools gained — from +1.2 points to +12.2

Slide 17 / 21

When a before-after comparison is legitimate

• When the counterfactual is genuinely flat and you can show it — Three or more pre-programme rounds with no trend is an…
• When the outcome cannot change on its own — A latrine does not build itself
• When you are monitoring, not evaluating — "Attendance rose from 84% to 88%" is a true statement about the world and a…
• The sentence to watch for is "as a result of" — It is where a monitoring number becomes an impact claim, and it is…

Slide 18 / 21

Report it whole — Example

Literacy outcomes, baseline to endline 2024

  All students        56.8% to 63.7%    +6.97 points   95% CI +6.1 to +7.9
    With feeding      57.8% to 64.4%    +6.66 points
    Without feeding   54.7% to 62.3%    +7.62 points

  The overall change is reported as a monitoring result and is not
  attributed to the feeding programme. Schools without the programme gained
  slightly more, so the difference-in-differences estimate of the programme
  effect is -0.96 points (95% CI -3.5 to +1.6, clustered on 24 schools).

  The before-after change assumes children would have learned nothing over
  a school year. That counterfactual is not defensible here and the
  comparison group is what replaces it.

Slide 19 / 21

Report it whole

• The last paragraph is three lines and it is the difference between a monitoring report and an impact claim — Writing…

Slide 20 / 21

What comes next

• The difference-in-differences estimate above rests on the two groups of schools being comparable in the ways that matter.

Slide 21 / 21

Where this goes next

Read the full lesson, with runnable code Back to the lesson