Nine points on a different test

A funding proposal is going out this week. Its headline result reads:

Literacy scores rose from 21.2 to 30.9 over the school year, a gain of 9.7 points (t = 18.0, p < 0.001, n = 1,399). The programme is delivering measurable learning gains.

Every number in that sentence can be reproduced from the committed file. The conclusion does not survive twenty minutes of checking, and your job is to do the checking before the proposal is submitted.

The files

learning-assessment-2024.v1.csv — baseline and endline literacy and numeracy assessments. school-enrolment-2024.v1.csv if you want to know who left.

What to find

Three separate problems, in the order they change the answer.

One: the denominator moved. Look at the items column by round before you look at anything else. Recompute the comparison as percent correct and report how much of the 9.7 points was a change in the test rather than a change in the children.

Two: the sample changed. Count the students with a baseline and no endline, and compute their baseline mean against the baseline mean of the students who stayed. Then produce two versions of the gain — across all students who sat each round, and across the students who sat both — and say how much of the difference between those two numbers is learning and how much is who is left in the room.

Three: the test was the wrong one. The proposal’s t of 18.0 treats baseline and endline as two independent samples of children. They are the same children twice. Run the paired test, report both standard errors, and explain which direction the error went.

What to hand in

A Python script and a short block of prose.

The script prints one table: for each version of the analysis — raw scores unpaired, percent correct unpaired, percent correct paired — the estimate, its 95% confidence interval, the test used, its statistic and its n. Six rows at most.

The prose is the replacement paragraph for the proposal, written to the four-sentence shape from lesson 8: the number and its interval, the comparison and its test, the size in a unit a reader can act on, and what the analysis cannot say. Include the attrition as a stated limitation rather than as a footnote.

Check yourself

	Expected
Items per paper, baseline and endline	40 and 50
Percent correct, baseline and endline	about 53.1% and 61.8%
Students with baseline and no endline	156
Their baseline mean, against those who stayed	about 39.3% versus 56.8%
Paired gain, percent correct	about +7.0 points, 95% CI 6.1 to 7.9
Paired n	585
Correlation between a child’s two scores	about 0.84
Ratio of the unpaired to the paired standard error	about 2.5

If your paired n is 741, you have kept students whose endline is missing and pandas has quietly filled a mean. If the gain comes out near 9.7 points, the scores are still raw and the denominator is still moving.

Three questions

Two sentences each.

1. The correction from raw scores to percent correct changes the headline. Say by how much, and say whether the remaining gain is still worth putting in a proposal.

2. The students who did not sit the endline scored far below those who did. State what that does to the endline mean, and what the honest headline is for a programme that lost its weakest students.

3. Ignoring the pairing made the standard error 2.5 times too large, which is the opposite of the school feeding error in lesson 6. Explain in plain language why one mistake inflates a standard error and the other deflates it.

The point

A p-value below 0.001 survived all three defects. The test was never the thing that was wrong — the comparison was, and no amount of significance rescues a comparison between two different papers taken by two different groups of children.

The gain that survives is real: about seven points of percent correct on the students who sat both papers, with an interval that never comes near zero. It is a smaller, defensible number in place of a larger, indefensible one, and it is the version that will still be true when someone checks it after the money has been spent.