Back to the lesson·Lesson 5 of 8·Which covariates belong
The covariate that improves every statistic and destroys the answer
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- A covariate that looks obviously right
- Why it removes the effect
- Four kinds of covariate, and what each does
- Drawing it before fitting
- The collider, which is worse
- Report both, and say which is which
- What comes next
Speaker notes
Add whether a referral was made and AIC falls by 595, pseudo-R² quadruples, and 42.5% of the disability gap disappears. Every model-selection criterion says keep it. Every causal consideration says it must come out, and no fit statistic will ever tell you which.A covariate that looks obviously right — In Python
import statsmodels.formula.api as smf BASE = ("completed ~ disability + case_category + age_band + sex" " + service_requested + admin1") adjusted = smf.logit(BASE, data=d).fit(disp=0) plus_gate = smf.logit(BASE + " + referral_made", data=d).fit(disp=0) for name, m in [("adjusted", adjusted), ("+ referral_made", plus_gate)]: print(f"{name:16} AIC {m.aic:7.1f} pseudo-R2 {m.prsquared:.3f}")Speaker notes
The referral pathway has a gate in it. A case is consented, then a referral is made, then it is accepted, then a service is reached.referral_madeis recorded, it is strongly related to completion, and adding it to the model is the natural next step.A covariate that looks obviously right — In R
adjusted <- glm(completed ~ disability + case_category + age_band + sex + service_requested + admin1, data = d, family = binomial()) plus_gate <- update(adjusted, . ~ . + referral_made) AIC(adjusted, plus_gate)A covariate that looks obviously right
Model Odds ratio Average marginal effect AIC Pseudo-R² Crude 0.430 −19.05 pts 2138.5 0.012 Adjusted 0.388 −19.41 pts 1999.6 0.089 + referral_made0.494 −11.15 pts 1404.2 0.365 A covariate that looks obviously right
- AIC improves by 595 points. Pseudo-R² quadruples. And 42.5% of the effect vanishes
Speaker notes
AIC improves by 595 points. Pseudo-R² quadruples. And 42.5% of the effect vanishes. Every automatic model-selection procedure ever written would keep that variable. Stepwise selection keeps it, AIC keeps it, cross-validated accuracy keeps it, and all of them are answering a question nobody asked.Why it removes the effect — In Python
print(d.groupby("disability")["referral_made"].agg(["mean", "size"]).round(4))Why it removes the effect — In R
d |> summarise(made = mean(referral_made), n = n(), .by = disability)Why it removes the effect
- A referral is made for 71.5% of cases with no disability reported and 53.8% of cases with one
- Conditioning on
referral_madeasks: among cases where a referral was made, is there still a gap? — The answer is yes,… - A covariate on the causal path between exposure and outcome is a mediator, and adjusting for it removes exactly the part of the effect that runs through it
Speaker notes
A referral is made for 71.5% of cases with no disability reported and 53.8% of cases with one. That is not a nuisance difference to be controlled away — it is part of the thing being measured. Cases reporting a disability are less likely to reach a service partly because a referral is less often made for them in the first place. Conditioning onreferral_madeasks: among cases where a referral was made, is there still a gap? The answer is yes, 11.2 points — which is a real and useful number about the acceptance stage. It is not the effect of disability on reaching a service, and reporting it as though it were understates that effect by 42%. A covariate on the causal path between exposure and outcome is a mediator, and adjusting for it removes exactly the part of the effect that runs through it.Four kinds of covariate, and what each does
Kind Sits Adjust? If you get it wrong Confounder Causes both exposure and outcome Yes Biased estimate, either direction Mediator Between exposure and outcome No (for a total effect) Effect understated Collider Caused by both exposure and outcome Never Bias created where none existed Competing cause Causes only the outcome Optional Precision only Speaker notes
The decision is about causal structure, not about statistics, and it has to be made before the model is fitted.Four kinds of covariate, and what each does
- Only the first row is a reason to add a variable — The fourth is a reason you may add one, and it buys a smaller…
- A variable's kind is not visible in the data — Confounder, mediator and collider all produce a coefficient that moves…
Speaker notes
Only the first row is a reason to add a variable. The fourth is a reason you may add one, and it buys a smaller standard error rather than a different estimate. The middle two are reasons not to. A variable's kind is not visible in the data. Confounder, mediator and collider all produce a coefficient that moves when you adjust, and all three improve fit if they predict the outcome. The only way to tell them apart is to know which came first and what caused what — which is why the diagram gets drawn before the model is fitted.Drawing it before fitting — Example
disability -------------------------------> completion | ^ +-------> referral made ---------------------+ department --> disability department --> completionSpeaker notes
Three arrows, drawn from what you know about how the pathway works.Drawing it before fitting
- Department is a confounder — where a case arises affects both who is registered with a disability and how well the…
Speaker notes
Department is a confounder — where a case arises affects both who is registered with a disability and how well the service system works — so it belongs in the model. Referral made is a mediator — disability affects it and it affects completion — so it does not.Drawing it before fitting — In Python
TOTAL_EFFECT = ["disability", "case_category", "age_band", "sex", "service_requested", "admin1"] # confounders only MECHANISM = TOTAL_EFFECT + ["referral_made"] # a different questionDrawing it before fitting — In R
# Two named model specifications, two questions, both legitimate.Drawing it before fitting
- Name the models after the question rather than after the variables — A model called
MECHANISMwill not accidentally…
Speaker notes
Name the models after the question rather than after the variables. A model calledMECHANISMwill not accidentally be reported as the total effect, and a reviewer can see which one was intended.- Name the models after the question rather than after the variables — A model called
The collider, which is worse — In Python
closed = d[d["case_status"].isin(["closed-resolved", "closed-lost-contact"])] print(f"closed cases: {len(closed)}") print(closed.groupby("disability")["completed"].agg(["mean", "size"]).round(4))Speaker notes
A mediator understates a real effect. A collider manufactures one that is not there, and the mechanism is subtler. Restricting to closed cases is the most common way it happens here, because a closed case is one where something was resolved — which is downstream of both disability and completion.The collider, which is worse — In R
d |> filter(case_status %in% c("closed-resolved", "closed-lost-contact")) |> summarise(rate = mean(completed), n = n(), .by = disability)The collider, which is worse
Sample n Gap (average marginal effect) All consenting cases 1,581 −19.4 points Closed cases only 860 −21.1 points The collider, which is worse
- Restricting to closed cases moves the gap to 21.1 points — and the movement is produced by the restriction rather than…
- Selecting rows is adjusting for a variable — Dropping open cases, keeping only completed forms, analysing only…
Speaker notes
Restricting to closed cases moves the gap to 21.1 points, and the movement is produced by the restriction rather than by anything about disability. Open cases — the ones still in progress — are excluded on a criterion that both variables influence. Selecting rows is adjusting for a variable. Dropping open cases, keeping only completed forms, analysing only households that answered every question: each is a conditioning step, and each can be a collider. "We restricted the analysis to closed cases for completeness" is a sentence that changes an estimate without anyone noticing it was a modelling decision.Report both, and say which is which — Example
Referral completion by disability status, decomposition Total gap -19.4 points of which runs through referral-making -8.3 points (42.5%) remaining, among referrals made -11.2 points Referrals are made for 53.8% of cases reporting a disability against 71.5% of others. Both gates contribute; neither explains the other away. The -11.2 figure is conditional on a referral having been made and must not be reported as the effect of disability on reaching a service.Speaker notes
Mediation is worth reporting when the mechanism is the programme's question, which here it is: the protection course located the disability gap at referral-making and acceptance, and this decomposition is the quantified version.Report both, and say which is which
- The last line is the one that keeps the decomposition honest — Both numbers are real, they answer different questions,…
Speaker notes
The last line is the one that keeps the decomposition honest. Both numbers are real, they answer different questions, and the failure mode is not computing the wrong one — it is computing the right one and labelling it as the other.What comes next
- Every model in this course so far has assumed each row is an independent observation.
Speaker notes
Every model in this course so far has assumed each row is an independent observation. The next lesson gives the model a term for the thing the rows are grouped inside, and finds three honest routes to the same answer where the naive one was out on its own.