Presenting the finding nobody commissioned

The three unwelcome findings

They are not equally hard and the difference matters.

“The programme did not do what you thought.” Hard, and it is the one people prepare for.

“We cannot tell.” Harder, because it sounds like a failure of the analyst rather than of the design. The impact evaluation course’s whole last unit is this case.

“The number you have been using is wrong.” Hardest, because somebody put it in a proposal, and the room contains the person who did.

All three arrive in this sector regularly, and the difference between a career-limiting meeting and a useful one is almost entirely preparation.

Lead with the decision, not the finding

The instinct is to build to the result. It is the wrong order in a room.

Not this: methodology, then data, then results, then the awkward part at minute eighteen when half the room has stopped listening and the other half has worked out where it is going.

This: the recommendation, then the one sentence that supports it, then the detail for whoever wants it.

  "My recommendation is that you do not expand the programme on the basis of
  this evaluation, and do not discontinue it either.

  Literacy rose seven points in programme schools. It also rose in the
  schools without the programme, by slightly more. The seven points are the
  school year.

  Separately, this evaluation was not large enough to detect the three-point
  difference you would act on, which was knowable before it ran. I can show
  you what a study that could detect it would need."

Sixty seconds, and the room now knows what you are asking for. Everything after that is answering questions rather than delaying the point.

Bring the thing that would have been true

An unwelcome finding lands very differently when it arrives with what a positive finding would have looked like.

“Here is what we would have seen if the programme were working at the size you care about.” Then show that it is not that, on the same chart, with the same axis.

And bring the cost of knowing. “Sixty schools instead of twenty-four” is a concrete, priceable thing, and a committee that cannot act on your finding can act on that.

A recommendation makes an unwelcome finding survivable. Without one the meeting is about whether you are right; with one it is about what to do.

What not to do

Four moves that feel helpful in the moment and are not.

Do not soften the number. “Roughly flat” for a −0.96 with an interval from −3.5 to +1.6 is a kindness that gets quoted as “flat” and then as “slightly positive”.

Do not offer the flattering disaggregation. There is always a subgroup where the programme looks like it worked. Offering it is how the two significant schools out of twenty-four end up in a proposal, and the statistics course showed the generator set that effect to exactly zero.

Do not blame the data. “The data quality was poor” is usually false and always sounds like an excuse. If the design was underpowered, say the design was underpowered — that is a decision somebody made, and naming it is how the next one is made better.

Do not apologise for the finding. You did not cause it. Apologising for it invites the room to treat it as your position rather than as the result.

The questions that will come, and the answers

Question	The answer
“Can we look at it by district?”	Yes, and here is the multiple-comparison count first
“The other agency reports 76%”	Different indicator; here is the definitions panel
“Can you re-run it excluding those schools?”	Yes, as an exploratory analysis, labelled
“So it doesn’t work?”	No — the study could not detect an effect of the size you care about
“What do we tell the donor?”	This, in three sentences, and I will draft it

The last row is the one to volunteer. The committee’s real problem is often not your finding but what they now have to say about it, and offering the three sentences turns you from the bearer of the problem into part of the answer.

The number in the last three proposals

The hardest case, and it needs its own handling.

Establish it privately first. Take it to the person who owns the number before the meeting. Not to protect them — to make sure you are right, because you might not be, and a room is the worst place to discover that.

Separate the error from the person. “The improved-source figure omitted delivered water, which the JMP definition includes” is about a set of strings. “The previous analysis was wrong” is about whoever did it.

Say which direction and how much. 79.7% to 82.3% is a correction that makes coverage look better, and saying so immediately stops the room bracing for the worst.

Bring the corrected version, the change log entry and the check that would have caught it earlier. A correction with a mechanism attached is a process improvement; one without is an accusation.

Change log: 2026-07-29, improved source corrected from 79.7% to 82.3%.
Delivered water was omitted from the JMP improved list. The definitions
document had it right; the analysis code did not. The pipeline now asserts
its computed values against the definitions document, so the two cannot
disagree silently again.

Report it whole

Steering committee, 14 March

  Presented: no detectable effect on literacy (-0.96 points, 95% CI -3.5 to
  +1.6), and the finding that the design's minimum detectable effect was 4.9
  points against a threshold of interest of 3.

  Recommendation: neither expand nor discontinue on this evidence. A design
  able to answer the question needs about 60 schools; randomising the order
  of the next roll-out costs nothing and would give a comparison group that
  does not have to be argued for.

  Agreed: the 7-point gain will not be used in the next proposal. The
  programme team will consider randomising the 2026 expansion order.
  Deferred: whether to fund a larger evaluation.

The “agreed” line is the meeting’s output, and writing it in the room is what stops the seven-point gain reappearing in June.

What comes next

The last lesson turns the course on the seven dashboards this platform ships — what each one is for, what each refuses to show, and the one that exists because a number in this very course was wrong.