A perfect discontinuity with nothing behind it

A rule is a design

The MUAC screening protocol refers a child to supplementary feeding below 125 mm and takes no action at or above it. That single number is doing something an evaluation usually has to pay for: it assigns treatment on a variable, at a known point, with no discretion.

import pandas as pd
import numpy as np

screening = pd.read_csv("muac-screening-artibonite-2024.v1.csv")
screening["referred"] = screening["outcome"] != "no-action"

window = screening[screening["muac_mm"].between(118, 132)]
print(window.groupby("muac_mm")["referred"].agg(["mean", "size"]).round(3))

library(dplyr)

screening |>
  filter(between(muac_mm, 118, 132)) |>
  summarise(referred = mean(outcome != "no-action"), n = n(), .by = muac_mm)

MUAC (mm)	Referred	n
122	100.0%	44
123	100.0%	47
124	100.0%	45
125	2.0%	51
126	0.0%	78
127	0.0%	70

One millimetre moves the referral probability from 1.00 to 0.02. Children at 124 and 125 mm are, in every way that matters, the same children — the measurement error on a MUAC tape is larger than the gap between them — and one group was treated while the other was not.

That is the closest thing to a randomised experiment that routine data produces, and it arrives free, because the programme was going to apply the rule anyway.

What a regression discontinuity needs

Three conditions. Two of them are checkable in this file and the third is not.

One: the assignment must actually be sharp at the cut-off. Checked above. If the rule were applied loosely — say 60% referral just below and 20% just above — the design still works but becomes fuzzy, and the estimate has to be scaled by the jump in treatment probability rather than read directly.

Two: nobody may manipulate the running variable. If a screener who wants a child treated records 124 instead of 126, the children just below the cut-off are no longer comparable to those just above — they are the ones somebody decided to help.

counts = screening["muac_mm"].value_counts().sort_index()
print(counts.loc[120:130])

below = screening["muac_mm"].between(120, 124).sum()
above = screening["muac_mm"].between(125, 129).sum()
print(f"120-124: {below}   125-129: {above}")

screening["last_digit"] = screening["muac_mm"] % 10
print((screening["last_digit"].value_counts(normalize=True).sort_index() * 100).round(1))

table(screening$muac_mm)[as.character(118:132)]
table(screening$muac_mm %% 10)

The counts rise smoothly through the cut-off: 45 at 124, 51 at 125, 78 at 126. There is no pile-up just below 125, and the density on the treated side is lower than on the untreated side, which is what a genuinely increasing MUAC distribution looks like.

Terminal digits are flat, 9.6% to 10.8% across all ten. No rounding to fives, no avoidance of the cut-off. The data quality course’s digit-preference test, used here for a different purpose: a manipulation check is a digit-preference check asked at one specific number.

Three: an outcome must be measured on both sides. It is not, and that ends the analysis.

The condition this register fails

print(screening.columns.tolist())
print(f"children screened once: {(screening['child_id'].value_counts() == 1).sum()}")
print(f"children screened twice: {(screening['child_id'].value_counts() == 2).sum()}")

names(screening)
table(table(screening$child_id))

The file records a measurement, a referral, and nothing else. 4,194 of 4,206 children appear exactly once. There is no follow-up MUAC, no recovery status, no second screening round — so for the children at 125 mm who were not referred, no outcome exists at all.

The treatment register does not fill the gap. cmam-admissions-2024 holds outcomes for treated children only, and its identifiers do not link to the screening file. Even if they did, it would carry only the referred side, which is precisely the half a discontinuity design already has.

An impact estimate needs the untreated side. Without it, the sharpest assignment rule in the sector produces nothing.

What it would have taken

This is the useful part of the lesson, because it is a data-collection specification someone can act on.

A follow-up measurement on children just above the cut-off. Not all of them — children between 125 and 130 mm, re-measured at eight weeks. A few hundred children, one extra visit, and the design becomes estimable.

The same outcome on both sides. Recovery is defined differently for treated and untreated children, and a comparison needs one definition applied to both: MUAC at eight weeks, measured the same way regardless of what happened in between.

Enough children near the cut-off. The estimate uses a window, and everything outside it is discarded.

for bandwidth in (2, 3, 5, 10):
    band = screening[
        screening["muac_mm"].between(125 - bandwidth, 124 + bandwidth)]
    print(f"±{bandwidth} mm: {len(band)} children"
          f" ({(band['muac_mm'] < 125).sum()} below, "
          f"{(band['muac_mm'] >= 125).sum()} above)")

# The bandwidth trade-off, in four lines.

Bandwidth	Children	Below cut-off	Above
±2 mm	221	92	129
±3 mm	335	136	199
±5 mm	556	187	369
±10 mm	1,210	265	945

A narrow bandwidth is more credible and less precise. Children within 2 mm of the cut-off are almost identical to each other; there are 221 of them. Widen to 10 mm and you have six times the sample and are comparing children who genuinely differ.

Choose the bandwidth before seeing the outcome, and report the estimate at two or three others. A result that appears only at one bandwidth is a result about the bandwidth.

Where thresholds hide in programme data

Once you look for eligibility rules, this sector is full of them.

Threshold	Programme	Running variable
MUAC < 125 mm	Supplementary feeding	Arm circumference
MUAC < 115 mm	Therapeutic feeding	Arm circumference
Poverty score below a cut-off	Cash transfer	Proxy means test
IPC Phase 3	Emergency response	Area classification
Below a coverage target	Facility support	Reported coverage

Each is a natural experiment nobody designed, and each needs the same three checks. The third — an outcome on both sides — is the one that most often fails, because programmes measure what they treat.

That is a data-collection decision, not an analysis one. Deciding to measure a sample of ineligible units costs money in a year when nobody is asking for it, and buys an evaluation three years later that would otherwise be impossible.

Report it whole

Supplementary feeding eligibility: assessment of a regression discontinuity

  Assignment: referral to TSFP below MUAC 125 mm.

  Sharpness    100% referred at 124 mm (n=45), 2.0% at 125 mm (n=51).
               The rule is applied without exception.

  Manipulation No pile-up below the cut-off: counts are 45, 51, 78 at 124,
               125 and 126 mm. Terminal digits are uniform (9.5-10.8%).
               No evidence the running variable was adjusted.

  Outcome      Not available. The register records screening and referral
               only; 4,194 of 4,206 children appear once, and no follow-up
               measurement exists on either side of the cut-off.

  Conclusion   The design is available and cannot be estimated. To make it
               estimable, re-measure MUAC at 8 weeks on a sample of children
               screened between 125 and 130 mm, using the same definition
               applied to referred children. 556 children sit within
               +/-5 mm of the cut-off, of whom 369 were not referred.

A design assessment that concludes “not estimable” is a deliverable, and this one is more useful than a forced estimate: it names the missing measurement, its size and its cost.

What comes next

The threshold design failed on data that does not exist. The next lesson is about a design that ran on data that does exist, and asks a question that should have been asked first — how large an effect could it ever have found.