Back to the lesson·Lesson 8 of 8·The design and the fit
R² of 0.03, and the most useful result in the report
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- A model built to answer a targeting question
- Do not delete it
- Diagnostics that change a decision
- What R² is and is not
- Report it whole
- What comes next
Speaker notes
Twelve household characteristics explain three per cent of the variation in child MUAC. Targeting screening on the model's worst-predicted 20% finds 31% of the malnourished children, against 20% at random. The model failed, and its failure is the operational answer.A model built to answer a targeting question — In Python (cont.)
import pandas as pd import numpy as np import statsmodels.formula.api as smf survey = pd.read_csv("household-survey-2025.v1.csv") d = survey.dropna(subset=["child_muac_mm", "child_age_months", "child_sex", "household_size", "displacement_status", "main_livelihood", "food_insecure", "improved_water_source"]) model = smf.ols( "child_muac_mm ~ child_age_months + child_sex + household_size" " + displacement_status + main_livelihood + food_insecure" " + improved_water_source", data=d).fit() print(f"n = {int(model.nobs)} R2 = {model.rsquared:.4f}"Speaker notes
A programme wants to screen fewer children. If household characteristics predicted which children are malnourished, screening could be targeted at the households most likely to hold them, and the survey carries every variable such a rule would use.A model built to answer a targeting question — In Python (cont.)
f" adjusted R2 = {model.rsquared_adj:.4f}") print(f"residual SD {np.sqrt(model.scale):.2f} outcome SD {d['child_muac_mm'].std():.2f}")A model built to answer a targeting question — In R
model <- lm(child_muac_mm ~ child_age_months + child_sex + household_size + displacement_status + main_livelihood + food_insecure + improved_water_source, data = d) summary(model)A model built to answer a targeting question
- n = 641. R² = 0.030, adjusted R² = 0.011. Residual SD 15.95 mm against an outcome SD of 16.04 mm
Speaker notes
n = 641. R² = 0.030, adjusted R² = 0.011. Residual SD 15.95 mm against an outcome SD of 16.04 mm. The model has removed less than one millimetre of the sixteen it started with. One coefficient is distinguishable from zero.A model built to answer a targeting question
Term Coefficient 95% CI Food insecure −4.70 mm −7.45 to −1.95 Salaried livelihood −3.49 mm −8.44 to +1.45 Farming livelihood −2.76 mm −6.16 to +0.64 Child age, per month −0.03 mm −0.11 to +0.05 Improved water source −1.09 mm −3.76 to +1.58 Do not delete it — In Python
d = d.assign(predicted=model.fittedvalues, mam=(d["child_muac_mm"] < 125)) for frac in (0.20, 0.30, 0.50): cut = d["predicted"].quantile(frac) caught = d.loc[d["predicted"] <= cut, "mam"].sum() print(f"screen worst {frac:.0%}: catches {caught}/{d['mam'].sum()}" f" = {caught / d['mam'].sum():.1%} of cases")Speaker notes
The instinct is to drop the model, try more variables, or reach for something non-linear. All three are wrong here, and the reason is that the question was not "can I model MUAC" — it was "can I target screening".Do not delete it — In R
d |> mutate(predicted = fitted(model), mam = child_muac_mm < 125) |> arrange(predicted) |> summarise(caught = mean(head(mam, n() * 0.2)) * sum(mam))Do not delete it
Screening rule Children screened MUAC < 125 mm found Worst-predicted 20% 129 28 of 90 = 31.1% Worst-predicted 30% 193 41 of 90 = 45.6% Worst-predicted 50% 321 54 of 90 = 60.0% Random 20% 129 20% expected Do not delete it
- Targeting on the model finds 31% of the malnourished children while screening 20% of them. Screening 20% at random…
- That is the finding — Not "the model did not work" — *household characteristics do not identify malnourished children…
Speaker notes
Targeting on the model finds 31% of the malnourished children while screening 20% of them. Screening 20% at random finds 20%. The model is eleven points better than chance and misses seven in ten of the children a programme exists to reach. That is the finding. Not "the model did not work" — household characteristics do not identify malnourished children well enough to target screening, so screening has to be blanket. A programme can act on that sentence, it costs a real amount of money, and it is supported by a model with an R² of 0.03.Diagnostics that change a decision
- Fitted values outside the possible range — For a binary outcome fitted with linear regression, this is immediate
Speaker notes
Four checks, and each one has a decision attached rather than a threshold. Fitted values outside the possible range. For a binary outcome fitted with linear regression, this is immediate.Diagnostics that change a decision — In Python
lpm = smf.ols("completed ~ disability + case_category + age_band + sex" " + service_requested + admin1", data=referrals).fit() print(f"fitted values from {lpm.fittedvalues.min():.3f}" f" to {lpm.fittedvalues.max():.3f}") print(f"outside [0, 1]: {((lpm.fittedvalues < 0) | (lpm.fittedvalues > 1)).sum()}")Diagnostics that change a decision — In R
range(fitted(lm(completed ~ ., data = referrals)))Diagnostics that change a decision
- Eight of 1,581 fitted probabilities fall below zero, the lowest at −0.060 — The logistic model on the same data stays…
- Residuals against fitted values — Look for a fan shape, which means the spread depends on the level
Speaker notes
Eight of 1,581 fitted probabilities fall below zero, the lowest at −0.060. The logistic model on the same data stays within 0.065 and 0.738. The decision: use logistic when predictions matter, and the linear probability model when only the average difference does — the linear model's coefficient is directly a risk difference, which is why it survives in economics. Residuals against fitted values. Look for a fan shape, which means the spread depends on the level.Diagnostics that change a decision — In Python
import matplotlib.pyplot as plt plt.scatter(model.fittedvalues, model.resid, s=6)Diagnostics that change a decision
- The decision: heteroscedasticity does not bias the coefficient, it biases the standard error — so the fix is a robust…
- Influence — A handful of rows can carry a coefficient
Speaker notes
The decision: heteroscedasticity does not bias the coefficient, it biases the standard error — so the fix is a robust standard error, not a transformed outcome. Influence. A handful of rows can carry a coefficient.Diagnostics that change a decision — In Python
influence = model.get_influence().cooks_distance[0] print(f"rows with Cook's D > 4/n: {(influence > 4 / len(d)).sum()}")Diagnostics that change a decision
- The decision: look at the flagged rows before doing anything — The WASH course's eleven households with a unit error…
- Linearity of a continuous predictor — Fit it in bands and see whether the coefficients step evenly
Speaker notes
The decision: look at the flagged rows before doing anything. The WASH course's eleven households with a unit error are exactly the kind of row that turns up here, and the answer is to fix the data, not to down-weight the point. Linearity of a continuous predictor. Fit it in bands and see whether the coefficients step evenly.Diagnostics that change a decision — In Python
d["age_band"] = pd.cut(d["child_age_months"], [0, 12, 24, 36, 48, 60]) print(smf.ols("child_muac_mm ~ C(age_band)", data=d).fit().params.round(2))Diagnostics that change a decision — In R
lm(child_muac_mm ~ cut(child_age_months, c(0, 12, 24, 36, 48, 60)), data = d)Diagnostics that change a decision
- The decision: if the bands do not step evenly, the linear term is answering the wrong question — and banding is usually…
Speaker notes
The decision: if the bands do not step evenly, the linear term is answering the wrong question, and banding is usually a better answer than a polynomial because the bands are interpretable to the people reading the report.What R² is and is not
- R² is the share of variance the model accounts for — In programme data it is routinely small and that is a fact about…
- It is not a measure of whether a coefficient is right — A randomised trial with a decisive treatment effect can have an…
- It is a measure of whether predictions are worth making — That is the use above, and it is the honest one: the…
- Adjusted R² penalises added terms, and it went from 0.030 to 0.011 here — the gap between the two is a measure of how…
Speaker notes
R² is the share of variance the model accounts for. In programme data it is routinely small and that is a fact about people, not about the analyst. It is not a measure of whether a coefficient is right. A randomised trial with a decisive treatment effect can have an R² of 0.02, because individual variation swamps a real average difference. Judging a causal estimate by R² is a category error. It is a measure of whether predictions are worth making. That is the use above, and it is the honest one: the targeting question is a prediction question, and R² answered it. Adjusted R² penalises added terms, and it went from 0.030 to 0.011 here — the gap between the two is a measure of how many of the twelve variables were paying for themselves. Almost none were.Report it whole — Example (cont.)
Predicting child MUAC from household characteristics Linear model, 641 children with complete data. R-squared 0.030, adjusted 0.011. Residual SD 15.95 mm against an outcome SD of 16.04 mm. Only food insecurity is distinguishable from zero: -4.70 mm (95% CI -7.45 to -1.95). Household size, displacement status, water source, child sex and child age are not. Used as a targeting rule, screening the 20% of children the model ranks worst would find 31% of the children with MUAC below 125 mm, against 20% expected from screening 20% at random. Screening half the children would find 60%. Recommendation: do not target screening on household characteristics. TheReport it whole — Example (cont.)
survey's 14.0% prevalence of MUAC below 125 mm is not concentrated in households that a registration form can identify, and blanket screening remains the only rule that reaches the caseload. This model is reported because it does not fit. A negative result about targeting is a budget decision, and dropping the model would have left the proposal to assume targeting works.Report it whole
- The last paragraph is why the section exists — A model that does not fit is evidence, and the file drawer it usually…
Speaker notes
The last paragraph is why the section exists. A model that does not fit is evidence, and the file drawer it usually goes into is where a programme's assumption survives unchallenged.What comes next
- That is the course.
Speaker notes
That is the course. Eight lessons, one idea: a coefficient is a comparison, and the work is saying which comparison, between which units, adjusted for what, with a standard error that matches how the data arrived. The lab puts all four decisions in one analysis. Then Impact Evaluation Methods takes on the question this course has deferred in every lesson's last paragraph — which of these comparisons can be written as a cause.