Better balance, six schools lighter

Matching, in one paragraph

Find, for each treated unit, an untreated unit that looks like it; compare only the pairs. Everything else in the method is a choice about “looks like it” and about what to do when no such unit exists.

The appeal is that it makes the comparison explicit. The cost is that the second half of that sentence — only the pairs — changes who the answer is about, and it does so quietly.

Match the schools

import pandas as pd
import numpy as np

treated = schools[schools["feeding_programme"]].sort_values("baseline")
control = schools[~schools["feeding_programme"]].sort_values("baseline")

used, pairs = set(), []
for _, c in control.iterrows():
    available = treated[~treated["school_id"].isin(used)]
    nearest = (available["baseline"] - c["baseline"]).abs().idxmin()
    used.add(treated.loc[nearest, "school_id"])
    pairs.append({"control": c["school_id"],
                  "treated": treated.loc[nearest, "school_id"],
                  "base_c": c["baseline"], "base_t": treated.loc[nearest, "baseline"],
                  "gain_c": c["gain"], "gain_t": treated.loc[nearest, "gain"]})

matched = pd.DataFrame(pairs)
print(f"{len(matched)} pairs from {len(treated)} treated and {len(control)} control")

library(MatchIt)

m <- matchit(feeding_programme ~ baseline, data = schools,
             method = "nearest", ratio = 1)
summary(m)

Nine pairs from fifteen treated and nine control schools. One-to-one matching without replacement can produce at most as many pairs as the smaller group has units, so six treated schools have no match and leave the analysis.

	Before matching	After matching
Treated schools	15	9
Control schools	9	9
Baseline gap	+3.35 pts	+1.41 pts
Estimate	−0.96 pts	−0.71 pts
Standard error	1.29	1.22

The balance improved and the estimate barely moved, which is the outcome to hope for: it says the difference-in-differences was not being driven by the baseline gap.

Which six schools left, and why it matters

dropped = treated[~treated["school_id"].isin(used)]
print(dropped[["school_id", "baseline", "gain"]].round(4))
print(f"dropped mean baseline: {dropped['baseline'].mean():.4f}")
print(f"kept mean baseline:    {matched['base_t'].mean():.4f}")

# Which units matching threw away is the first table to print, not the last.

The six discarded schools are the six with the highest baseline literacy — 53.9% to 65.5%, against a matched-treated mean of 55.8%. They were dropped because the control group contains no school that scored as highly, so there is nothing to compare them to.

That is not a flaw in the matching; it is the matching telling you something true. The nine control schools cannot speak for the highest-performing programme schools, because no such control school exists.

But it changes the claim. The matched estimate answers “among schools with baseline literacy below about 60%, what did the programme do?” — and that is a narrower question than the one the report set out to answer.

What “looks like it” should mean

Matching on one variable is a teaching simplification. In practice the choice is between three approaches.

Approach	Match on	Use when
Exact	Identical values of a few categorical variables	Few covariates, large samples
Nearest neighbour on a propensity score	The predicted probability of treatment	Many covariates
Coarsened exact	Values binned into ranges	You want a guaranteed balance level

import statsmodels.formula.api as smf

ps = smf.logit("feeding_programme ~ baseline + size + district",
               data=schools).fit(disp=0)
schools["pscore"] = ps.predict(schools)
print(schools.groupby("feeding_programme")["pscore"].describe()[
    ["min", "mean", "max"]].round(3))

glm(feeding_programme ~ baseline + size + district, data = schools,
    family = binomial())

A propensity score is one number summarising many covariates, and matching on it is equivalent to matching on all of them at once — which is what makes it useful when there are more covariates than a small sample can match exactly.

It is not a way to get more data. The score reduces dimensionality; it does not create comparators where none exist.

Check overlap before you match, not after

This is the check the regression course’s exercise found the hard way on the two CMAM programmes, and it applies unchanged here.

t = schools[schools["feeding_programme"]]["pscore"]
c = schools[~schools["feeding_programme"]]["pscore"]
print(f"treated range {t.min():.3f}–{t.max():.3f}")
print(f"control range {c.min():.3f}–{c.max():.3f}")
print(f"treated units above the control maximum: {(t > c.max()).sum()}")

# Plot the two propensity score distributions. The tails are the whole story.

Units outside the other group’s range have no counterfactual in the data. Matching drops them, which is honest; regression adjustment keeps them and extrapolates, which is not. That is the real difference between the two methods, and it is a better reason to prefer matching than any claim about bias.

Four things to report, always

How many units were discarded, and which. The first table, not a footnote.

Balance before and after. Standardised differences on every covariate, both columns, so a reader can see what matching bought.

What population the estimate now describes. One sentence, in the words a programme manager uses — “schools with baseline literacy below 60%”, not “the region of common support”.

The unmatched estimate too. If matching changed the answer materially, that is itself a finding about how much the comparison depended on the units it dropped.

What matching cannot do

It cannot fix an unmeasured confounder. Two schools with identical baseline scores can still differ in why one was selected. Matching balances what you matched on and nothing else — which is exactly the limit of regression adjustment, reached by a different route.

It cannot create a control group. If the untreated units are systematically different, matching will report excellent balance on a handful of pairs and silence about everyone else.

It does not remove the need for a design. Matching plus a baseline is difference-in-differences on a subset; matching alone is a cross-section with better manners. The estimate above is the first of those, and combining the two is usually stronger than either.

Report it whole

School feeding and literacy, matched difference-in-differences

  Nearest-neighbour matching on baseline literacy, 1:1 without replacement.
  9 matched pairs from 15 treated and 9 control schools.

  6 of 15 programme schools were discarded: SCH05, SCH12, SCH14, SCH15,
  SCH19, SCH21. Their mean baseline literacy is 60.7% against 55.8% for the
  matched treated schools; no control school scored high enough to match
  them.

  Baseline imbalance   +3.35 points before matching, +1.41 after.
  Estimate             -0.71 points, SE 1.22, on 9 pairs.
  Unmatched estimate   -0.96 points, 95% CI -3.48 to +1.56.

  The matched estimate describes schools with baseline literacy below about
  60%. It does not describe the six highest-performing programme schools,
  which have no comparator in this data.

  Matching balances the covariates it was given. It does not address why
  these schools were selected for the programme.

The named list of dropped schools is what makes this reportable. A reader can check whether the six that left are the six the programme cares most about, and no summary statistic tells them that.

What comes next

Matching and difference-in-differences both build a comparison group out of units that were not assigned by a rule. The next lesson takes the opposite case — a programme where a rule decided everything, sharply, at a number — and finds that two of the three things such a design needs are already in the data.