The evaluation you were not asked for

The commission is one sentence: what effect did the school feeding programme have on numeracy? The lessons answered that question for literacy. Do not read them while doing this — the numeracy answer differs from the literacy answer in one respect that matters, and finding it is the point.

The files

learning-assessment-2024.v1.csv, school-roster-2024.v1.csv and school-enrolment-2024.v1.csv. Same 24 schools, same children, the other domain.

Set up first

An RStudio project, files read-only, an outputs/ directory, one script that runs top to bottom. Every number in your report is printed by it.

Part one: write the plan before you compute anything

This part is graded on being done first. Produce plan.md — one page, dated, with the eight sections from lesson 7: question, design, outcome, threshold of interest, analysis, power, what would change the conclusion, what you will not claim.

Two of those sections you cannot fill in from the file alone and must decide:

The outcome definition. The two rounds use different numbers of items. Decide, in writing, what you will compare and why.

The threshold of interest. Nobody has told you. Choose one, state it, and say what programme decision it corresponds to — then hold yourself to it in part five.

Commit plan.md before writing any analysis code. Its date is the deliverable.

Part two: the three naive answers

Compute all three, each with an interval:

• the before-after change across all children;
• the endline difference between programme and comparison schools;
• the difference-in-differences.

Put them in one table with a column saying what counterfactual each assumes. Two of the three are answers to questions nobody asked, and your table should make that visible without needing a paragraph.

Part three: is the comparison group one?

Build the balance table at school level — baseline numeracy, school size, mean age, disability, displacement, over-age — with standardised differences, and the district crosstab.

Then answer, in your script as a comment: does the imbalance you found make parallel trends more or less plausible? The two are different questions and the answer is not automatic.

Part four: the standard error, and a surprise

Fit the difference-in-differences three ways: naive, clustered on school, and aggregated to 24 school-level means.

One of those will not behave the way the lessons led you to expect. Work out why before reading part five. The relevant question is what a difference-in-differences on gains has already removed from the between-school variance.

Part five: what could you have found?

Compute the minimum detectable effect for this design, using the intra-cluster correlation of the gain rather than of the level. Compare it with the threshold you wrote down in part one.

Then write the summary paragraph from lesson 8 — the one that leads with what the design could and could not have detected.

Check your numbers

	Expected
Children with both rounds	585
Numeracy, baseline and endline	57.6% and 63.7%
Before-after change	+6.14 pts, 95% CI +5.27 to +7.01
Baseline gap, programme minus comparison	+1.58 pts
Endline gap	+1.97 pts, 95% CI −1.33 to +5.27
Difference-in-differences	+0.39 pts
DiD, naive standard error	0.95
DiD, clustered standard error	0.79
DiD, school level (24 units)	+0.07 pts, SE 0.81
SD of the gain	0.107
ICC of the gain	about −0.01

If your before-after comes out near +9 points, you are still on raw scores and the denominator is moving between rounds. If your difference-in-differences is far from +0.4, check that the roster join has not fanned out the two duplicated students.

The surprise, and what to do with it

The clustered standard error is smaller than the naive one. That is not a bug and it is not a rounding artefact: the intra-cluster correlation of the gain is very slightly negative, because differencing each child against their own baseline has already removed the school-level component that made the level clustered.

Clustering does not always widen an interval. It corrects one, and the direction depends on whether observations within a cluster are more alike than observations across clusters on the quantity being analysed. For attendance levels they are; for year-on-year gains here, they are not.

Report the clustered version anyway. It is the one that matches the design, and choosing the naive one because it happens to be wider is the same defect as choosing it because it happens to be narrower.

The questions to answer in prose

Three sentences each.

1. Your three naive answers differ. Name the counterfactual each assumes and say, for the two you reject, what a reader would have concluded from them.

2. The clustered standard error is smaller than the naive one here and larger in the attendance analysis from the regression course. Explain the difference to a colleague who has been told “clustering always widens the interval”.

3. State your threshold of interest from part one, the minimum detectable effect you computed in part five, and what those two numbers together mean for the commission’s question. Say whether you would have accepted this evaluation had you been asked in January.

What to hand in

• plan.md, dated, committed before the analysis
• an R script producing the three-answer table, the balance table, the three standard errors, and the power calculation
• report.md in the shape of lesson 8’s summary — finding, then recommendation, with the design’s detectable effect in the first paragraph
• a deviations table, even if it is empty, in the shape of lesson 7’s
• the three prose answers

How to know you are done

Delete outputs/, rerun, everything regenerates identically. Then change your threshold of interest at the top of the script from 3 points to 1 point: the recommendation paragraph should change and no estimate should. If an estimate moves, the threshold has leaked into the analysis, which is the failure the plan exists to prevent.

What this lab is not

It is not an evaluation of school feeding. Nobody randomised these schools, the comparison group is imbalanced on every characteristic measured, and the design could not detect the effect size a programme would act on — three facts that are conclusions of this lab rather than caveats to it.

It is also not a demonstration that the programme fails. The whole point of the last two lessons is that “we could not tell” is a finding, and producing it clearly, with the number that would have been needed to tell, is the deliverable.