Course · Advanced · Statistics and Modelling
Impact Evaluation Methods
Counterfactual thinking and the designs that support it — randomisation, difference-in-differences, matching and regression discontinuity — plus how to say what a before-after comparison cannot.
What you will be able to do
- State the counterfactual a claim requires, and say why a before-after change is not one
- Read a balance table and judge whether a comparison group is one
- Fit a difference-in-differences estimate and argue for the assumption it rests on
- Match on a propensity score and report the units matching discarded
- Recognise an eligibility threshold as a design, and check the three things it needs
- Compute a minimum detectable effect before the evaluation, and say what an underpowered study may conclude
Standards and methodologies
Every “report it whole” block in Regression for Programme Data ends with the same sentence: observational, groups were not randomised. This is the course that says what would have to be true to delete it.
The whole of it can be seen in one question asked three ways of the same file.
Literacy scores rose 7.0 points over the school year. Paired, on 585 children, interval +6.1 to +7.9. A programme document would call that the effect of the school feeding programme.
Schools with a feeding programme end the year 2.1 points ahead of schools without. Interval −1.2 to +5.4. A different document would call that the effect.
Both groups gained, and the unfed schools gained more. Fed +6.7, unfed +7.6, so the difference-in-differences estimate is −0.96 points, with a school-clustered interval from −3.5 to +1.6. There is no effect here to report, and the first two numbers are the school year and a pre-existing gap wearing an effect’s clothes.
Three more results shape the course.
Fifteen schools have the programme and nine do not, and they were not comparable to begin with. The standardised difference in baseline literacy is 0.85, against the 0.1 a randomised trial would tolerate. All six characteristics are imbalanced, and Centre and Nord-Ouest run five fed schools to one unfed.
Matching improves the balance by throwing away 40% of the treatment group. Nine control schools can absorb nine treated schools; the six discarded are the six with the highest baseline scores, and the estimate that survives applies only to the schools that were left.
This design could never have detected anything smaller than 4.9 points. With an intra-cluster correlation of 0.065 and fifty children per school, 24 clusters buy a minimum detectable effect of 4.9 points where the same 1,200 children individually randomised would have bought 2.3. That number was fixed before a single child was measured.
Both Python and R throughout. You need Regression for Programme Data first — this course spends no time on how to fit a model and all of it on which comparison the model is allowed to be.
Practice
Reading the lessons is not the same as having done the work. Each of these applies the course to a dataset it did not teach on.
- Lab · 180 minThe evaluation you were not asked forYou are handed the numeracy assessment and asked for the effect of the school feeding programme. Write the one-page plan first, run all four designs, compute what the study could have detected, and deliver the answer the commission did not want.
- Exercise · 60 minThe slide with one number on itA board pack says the feeding programme raised literacy by 7 points and recommends doubling it. The number is right, the counterfactual is missing, and you have an hour before the meeting to work out what to say instead.
Progress
Take it offline
The whole course as a typeset PDF — every lesson, every code example, the data dictionary and the indicator definitions. Generated from the same source as this page.
The LaTeX source ships alongside each PDF, so an organisation can rebrand the handout or fold a lesson into its own training pack.