The answer was fixed before the first child was measured

Compute what the design can find

import numpy as np

SD = 0.1429        # within-school SD of attendance, from the register
ICC = 0.065        # intra-cluster correlation, from the empty mixed model
M = 50             # children per school
Z_ALPHA, Z_BETA = 1.96, 0.84       # 5% two-sided, 80% power

deff = 1 + (M - 1) * ICC
print(f"design effect {deff:.2f}")

def mde(k_treated, k_control):
    n1, n0 = k_treated * M, k_control * M
    return (Z_ALPHA + Z_BETA) * SD * np.sqrt(deff * (1 / n1 + 1 / n0))

print(f"15 vs 9 schools: {mde(15, 9) * 100:.2f} points")

# clusterPower::cpa.normal(), or the formula above. Either way, before the study.

Design effect 4.19. Minimum detectable effect 4.88 points.

This evaluation could never have found an effect smaller than about five points on attendance, whatever the analysis did afterwards. That is a property of 24 schools, fifty children each, and an intra-cluster correlation of 0.065 — all three known before any child was measured.

Where the number comes from

Four inputs, and only two of them are decisions.

Input	Here	Decision or fact?
Outcome standard deviation	0.143	Fact, from a pilot or an earlier round
Intra-cluster correlation	0.065	Fact, and usually the one nobody has
Clusters per arm	15 and 9	Decision
Children per cluster	50	Decision, and the weaker lever

for k in (12, 24, 40, 60, 100):
    print(f"{k:3} schools: MDE {mde(k // 2, k // 2) * 100:.2f} points")

# The whole planning conversation in five rows.

Schools	Children	MDE
12	600	6.68 pts
24	1,200	4.73 pts
40	2,000	3.66 pts
60	3,000	2.99 pts
100	5,000	2.32 pts

Halving the detectable effect costs roughly four times the schools. Precision improves with the square root, so a design that wants 3 points instead of 4.7 needs 60 schools rather than 24.

Clusters, not children

This is the number that surprises programme managers and it is the whole reason cluster designs are expensive.

individual = (Z_ALPHA + Z_BETA) * SD * np.sqrt(4 / 1200)
print(f"1,200 children, individually randomised: {individual * 100:.2f} points")
print(f"1,200 children in 24 schools:            {mde(12, 12) * 100:.2f} points")
print(f"ratio: {np.sqrt(deff):.2f}")

# Same children, same outcome, same budget. Twice the detectable effect.

2.31 points individually, 4.73 points in clusters — a factor of 2.05, which is the square root of the design effect.

Adding children to existing schools barely helps. With an ICC of 0.065, going from 50 to 100 children per school moves the design effect from 4.19 to 7.44 and the MDE from 4.73 to 4.45 points — a 6% improvement for double the fieldwork.

Adding schools helps in proportion. That is the planning rule: when the programme is assigned by cluster, buy clusters.

The intra-cluster correlation is the input nobody has

It is the hardest of the four to obtain and the one that moves the answer most.

for icc in (0.01, 0.03, 0.065, 0.10, 0.20):
    d = 1 + (M - 1) * icc
    m = (Z_ALPHA + Z_BETA) * SD * np.sqrt(d * (1 / 600 + 1 / 600))
    print(f"ICC {icc:.3f}: design effect {d:5.2f}, MDE {m * 100:.2f} points")

# Vary the ICC across the plausible range and report the MDE for each.

ICC	Design effect	MDE, 24 schools
0.01	1.49	2.82 pts
0.03	2.47	3.63 pts
0.065	4.19	4.73 pts
0.10	5.90	5.61 pts
0.20	10.80	7.59 pts

The MDE ranges from 2.8 to 7.6 points across plausible ICCs. A power calculation quoting one ICC with no range is not a calculation, it is a guess with arithmetic attached.

Where to get one. An earlier round of the same survey, a published study in the same sector and country, or a pilot. Failing all three, report the table above and plan for the pessimistic row — and say in the protocol which row you planned for.

What an underpowered study may conclude

This is the part that changes what gets written, and there are exactly three permitted statements.

“No effect larger than X was detected.” With an MDE of 4.9 points and an observed difference-in-differences of −0.96 with an interval from −3.5 to +1.6, the honest sentence is that effects above about 3.5 points in either direction are ruled out and smaller ones are not.

“The study was not designed to detect an effect of the size that matters.” If the programme would be considered worthwhile at 2 points and the design detects 4.9, that is a finding about the evaluation, and it should have been found before it ran.

“More units are needed, and here is how many.” The table above, with the row a future evaluation should aim at.

Three statements that are not permitted. “The programme had no effect” — the data cannot support it. “The effect was not statistically significant, so the programme should be discontinued” — a decision resting on an absence of evidence. And silence about power, which lets a reader supply the first two for themselves.

Post-hoc power is not a thing

observed = -0.0096
se = 0.0129
print(f"observed effect {observed * 100:+.2f} points, SE {se * 100:.2f}")

# Do not compute power from the observed effect. It is a rearrangement of p.

Power computed from the effect you observed is a monotone function of your p-value — it adds no information and creates a false sense of having checked something. A non-significant result with “low observed power” is exactly a non-significant result.

The useful quantity is the confidence interval, which the statistics course established, and the MDE computed from the design rather than from the result. Both are available here; post-hoc power is not one of them.

Report it whole

Statistical power, school feeding evaluation

  Design    15 treated and 9 control schools, ~50 children each.
  Inputs    outcome SD 0.143, ICC 0.065 (estimated from this study's own
            empty mixed model), 5% two-sided, 80% power.

  Design effect 4.19. Minimum detectable effect 4.88 percentage points.

  The observed difference-in-differences is -0.96 points (95% CI -3.5 to
  +1.6). Effects larger than about 3.5 points in either direction are ruled
  out; smaller effects are not, and the design could not have detected them.

  This is reported as a null result about large effects. It is not evidence
  that the programme has no effect.

  Sensitivity: across ICCs from 0.01 to 0.20 the minimum detectable effect
  ranges from 2.8 to 7.6 points. The ICC used is estimated from 24 clusters
  and is itself imprecise.

  A future evaluation targeting a 3-point effect requires about 60 schools.
  Adding children to the existing 24 does not help: doubling to 100 per
  school moves the minimum detectable effect from 4.73 to 4.45 points.

The last paragraph is the one a programme can act on, and it is the reason to compute power after a null result as well as before a study: it converts “we found nothing” into “here is what finding something would cost”.

What comes next

Every design in this course has been reconstructed after the fact. The next lesson does it in the right order — writing the design, the outcome, the analysis and the threshold of interest down before the data arrives, in a document short enough that someone will actually produce one.