A standardised difference of 0.85, where 0.1 is the limit

Slide 1 / 26

What this lesson covers

• Build the balance table before anything else
• What the standardised difference is, and why not a p-value
• What randomisation would have bought
• When randomisation is refused, and what to say
• The three questions to ask before an evaluation design
• Report it whole
• What comes next

Slide 2 / 26

Build the balance table before anything else — In Python (cont.)

import pandas as pd
import numpy as np

enrolment = pd.read_csv("school-enrolment-2024.v1.csv")
current = enrolment[(enrolment["school_year"] == 2024)
                    & (enrolment["age_years"] <= 20)]   # drop the age outliers
roster = pd.read_csv("school-roster-2024.v1.csv").drop_duplicates(
    "student_id", keep="last")
joined = current.merge(roster[["student_id", "school_id", "feeding_programme"]],
                       on="student_id", suffixes=("", "_r"))
joined["over_age"] = joined["age_years"] > joined["grade"] + 5
joined["displaced"] = joined["displacement_status"] != "resident"

by = joined.groupby("school_id")
schools = pd.DataFrame({
    "feeding_programme": by["feeding_programme"].first(),

Slide 3 / 26

Build the balance table before anything else — In Python (cont.)

    "baseline": by["baseline"].mean(),      # from the assessment file
    "age_years": by["age_years"].mean(),
    "size": by.size(),
    "over_age": by["over_age"].mean(),
    "disability": by["disability_reported"].mean(),
    "displaced": by["displaced"].mean(),
    "district": by["admin2"].first(),
}).reset_index()

def std_diff(treated, control):
    pooled = np.sqrt((treated.var(ddof=1) + control.var(ddof=1)) / 2)
    return (treated.mean() - control.mean()) / pooled

for column in ["baseline", "age_years", "size", "over_age",
               "disability", "displaced"]:
    t = schools[schools["feeding_programme"]][column].dropna()

Slide 4 / 26

Build the balance table before anything else — In Python (cont.)

    c = schools[~schools["feeding_programme"]][column].dropna()
    print(f"{column:12} {t.mean():8.4f} {c.mean():8.4f}"
          f"  std.diff {std_diff(t, c):+.3f}")

Slide 5 / 26

Build the balance table before anything else — In R

library(dplyr)

schools |>
  summarise(across(c(baseline, age_years, size, over_age), mean),
            .by = feeding_programme)

Slide 6 / 26

Build the balance table before anything else

Slide 7 / 26

Build the balance table before anything else

• The conventional threshold is 0.10, and every one of the six is above it — four of them by a factor of five or more

Slide 8 / 26

Build the balance table before anything else — In Python

print(pd.crosstab(schools["district"], schools["feeding_programme"]))

Slide 9 / 26

Build the balance table before anything else — In R

table(schools$district, schools$feeding_programme)

Slide 10 / 26

Build the balance table before anything else

Slide 11 / 26

Build the balance table before anything else

• Two districts are five-to-one in favour of the programme and one is two-to-four against — Whatever else differs between…

Slide 12 / 26

What the standardised difference is, and why not a p-value — In Python

t = schools[schools["feeding_programme"]]["baseline"]
c = schools[~schools["feeding_programme"]]["baseline"]
from scipy import stats
print(f"std.diff {std_diff(t, c):+.3f}   p = {stats.ttest_ind(t, c).pvalue:.3f}")

Slide 13 / 26

What the standardised difference is, and why not a p-value — In R

# Both, and only one of them is the right tool.

Slide 14 / 26

What the standardised difference is, and why not a p-value

• Standardised difference +0.85, p = 0.064 — A reviewer reading only the p-value would conclude the schools are balanced…
• A balance test's p-value confounds imbalance with sample size — which is the opposite of what a balance check needs
• Report the standardised difference, which does not depend on n — Anything above 0.10 is imbalance you have to handle;…

Slide 15 / 26

What randomisation would have bought

• That last clause is the whole value — Adjustment can fix imbalance in baseline literacy because it is in the file

Slide 16 / 26

What randomisation would have bought — In Python

rng = np.random.default_rng(20260729)
draws = []
for _ in range(2000):
    assigned = rng.permutation(schools["feeding_programme"].values)
    t = schools["baseline"][assigned]
    c = schools["baseline"][~assigned]
    draws.append(std_diff(t, c))
print(f"under random assignment, |std.diff| > 0.85 in "
      f"{np.mean(np.abs(draws) > 0.85):.1%} of draws")

Slide 17 / 26

What randomisation would have bought — In R

# Permute the assignment 2,000 times and see how unusual the real split is.

Slide 18 / 26

What randomisation would have bought

• Randomly reassigning the same 24 schools produces an imbalance this large in 5.3% of 2,000 draws — So the observed…

Slide 19 / 26

When randomisation is refused, and what to say

Slide 20 / 26

When randomisation is refused, and what to say

• A phased roll-out is the most under-used design in this sector — Every programme that expands over three years has a…
• Where none of that is possible, the balance table is what you owe the reader — It is not a formality: it is the…

Slide 21 / 26

The three questions to ask before an evaluation design

• Who decided who gets the programme, and on what? — If the answer is a rule, you may have a threshold design
• Is there anything measured before the programme started? — Without a baseline the difference-in-differences of the next…
• How many units were assigned? — Not how many people were measured — how many schools, clinics, villages, camps

Slide 22 / 26

Report it whole — Example (cont.)

Comparability of feeding and non-feeding schools

  15 schools with the programme, 9 without. Assignment was not randomised.

  Standardised differences at baseline:
    Baseline literacy     +0.85      School size        +0.68
    Mean age              +0.79      Disability         -0.52
    Displacement          -0.51      Over-age           +0.43

  All six characteristics exceed the 0.10 conventional threshold and all six
  exceed 0.25. Programme schools are concentrated in Centre and
  Nord-Ouest (5 of 6 schools in each) and scarce in Sud (2 of 6).

  Balance tests are not reported as p-values: with 24 units they would show
  balance regardless. The standardised difference does not depend on n.

Slide 23 / 26

Report it whole — Example (cont.)

  The comparison is adjusted for baseline literacy and district. Adjustment
  cannot address unmeasured differences in why these schools were selected,
  and no result in this report should be read as an effect of the programme
  alone.

Slide 24 / 26

Report it whole

• The last paragraph is where an evaluation earns or loses a reviewer — and its length is right: two sentences, stating…

Slide 25 / 26

What comes next

• If the two groups differed at baseline, the way to use the baseline is to subtract it.

Slide 26 / 26

Where this goes next

Read the full lesson, with runnable code Back to the lesson