Exercise · Intermediate
The number nobody can reproduce
A report says 12.4% and the analyst has left. The notebook produces 11.8%, the spreadsheet says 12.9%, and the only thing everyone agrees on is that the report is the one the minister saw. You have an afternoon to work out which is right.
The district nutrition report, published in April, says global acute malnutrition in Artibonite is 12.4%. It is quoted in a funding proposal and in a cluster presentation.
The analyst who produced it left in May. What they left behind:
- a notebook that, run today, produces 11.8%;
- a spreadsheet in a shared folder with 12.9% in a cell;
- no environment file, no seed, no record of which export was used.
Nobody can say which number is right, and the proposal deadline is Friday.
The file
muac-screening-artibonite-2024.v1.csv — 4,218 screenings across twelve communes.
Assume this is the export the analyst had.
What to do
One: reproduce every plausible definition. Compute the district GAM under each of these, and say which produces which of the three numbers:
- MUAC below 125 mm only;
- MUAC below 125 mm or oedema present;
- either of the above, restricted to children aged 6–59 months;
- either of the above, with the twelve communes weighted equally rather than by screenings.
Two: identify which choices are defensible. Two of the four are standard, one is defensible with an argument, and one is wrong. Say which is which and why.
Three: say what would have prevented this. Name the three artefacts that, had they existed, would make this afternoon unnecessary — and for each, say which of the three numbers it would have pinned down.
Four: write the correction. One paragraph for the cluster, stating the number, its definition, its denominator, and what happened.
Check yourself
| Expected | |
|---|---|
| Screenings with a MUAC value | 4,218 |
| MUAC below 125 mm | 9.3% |
| MUAC below 125 mm or oedema | 10.3% |
| Communes | 12 |
| Restricted to 6-59 months | 10.2% on 3,992 |
| Unweighted mean of the twelve commune rates | 10.9% |
| Lowest and highest commune | 5.6% and 15.3% |
None of the four definitions produces 12.4%, 11.8% or 12.9% exactly, and that is the finding. Say what it implies about the three numbers and about the export they came from.
Three questions
Two sentences each.
1. The unweighted mean of the twelve commune rates (10.9%) differs from the district figure (10.3%). Explain which one answers “what is GAM in Artibonite” and why the other is a different question.
2. None of your four definitions reproduces the published 12.4%. List what could account for that which is not a definition choice, in the order you would check it.
3. The proposal deadline is Friday and the honest answer is “we cannot reproduce the published figure”. Write the sentence you would send to the person writing the proposal.
The point
This is what irreproducibility costs, in a single afternoon. Not an abstract risk — a published number that nobody can defend, quoted in a document that asks for money, with the person who made it unavailable.
Three artefacts would have made it a five-minute question. A committed lock file, a seed and a stated case definition, and the checksum of the export used. Each is one line of effort at the time and none of them was there.
The correction paragraph is the deliverable. It is uncomfortable to write, and writing it is cheaper than every alternative — including the one where nobody writes it and the 12.4% goes into the next proposal too.