The covariate that improves every statistic and destroys the answer

A covariate that looks obviously right

The referral pathway has a gate in it. A case is consented, then a referral is made, then it is accepted, then a service is reached. referral_made is recorded, it is strongly related to completion, and adding it to the model is the natural next step.

import statsmodels.formula.api as smf

BASE = ("completed ~ disability + case_category + age_band + sex"
        " + service_requested + admin1")

adjusted = smf.logit(BASE, data=d).fit(disp=0)
plus_gate = smf.logit(BASE + " + referral_made", data=d).fit(disp=0)

for name, m in [("adjusted", adjusted), ("+ referral_made", plus_gate)]:
    print(f"{name:16} AIC {m.aic:7.1f}   pseudo-R2 {m.prsquared:.3f}")

adjusted  <- glm(completed ~ disability + case_category + age_band + sex +
                   service_requested + admin1, data = d, family = binomial())
plus_gate <- update(adjusted, . ~ . + referral_made)
AIC(adjusted, plus_gate)

Model	Odds ratio	Average marginal effect	AIC	Pseudo-R²
Crude	0.430	−19.05 pts	2138.5	0.012
Adjusted	0.388	−19.41 pts	1999.6	0.089
+ referral_made	0.494	−11.15 pts	1404.2	0.365

AIC improves by 595 points. Pseudo-R² quadruples. And 42.5% of the effect vanishes.

Every automatic model-selection procedure ever written would keep that variable. Stepwise selection keeps it, AIC keeps it, cross-validated accuracy keeps it, and all of them are answering a question nobody asked.

Why it removes the effect

print(d.groupby("disability")["referral_made"].agg(["mean", "size"]).round(4))

d |> summarise(made = mean(referral_made), n = n(), .by = disability)

A referral is made for 71.5% of cases with no disability reported and 53.8% of cases with one.

That is not a nuisance difference to be controlled away — it is part of the thing being measured. Cases reporting a disability are less likely to reach a service partly because a referral is less often made for them in the first place.

Conditioning on referral_made asks: among cases where a referral was made, is there still a gap? The answer is yes, 11.2 points — which is a real and useful number about the acceptance stage. It is not the effect of disability on reaching a service, and reporting it as though it were understates that effect by 42%.

A covariate on the causal path between exposure and outcome is a mediator, and adjusting for it removes exactly the part of the effect that runs through it.

Four kinds of covariate, and what each does

The decision is about causal structure, not about statistics, and it has to be made before the model is fitted.

Kind	Sits	Adjust?	If you get it wrong
Confounder	Causes both exposure and outcome	Yes	Biased estimate, either direction
Mediator	Between exposure and outcome	No (for a total effect)	Effect understated
Collider	Caused by both exposure and outcome	Never	Bias created where none existed
Competing cause	Causes only the outcome	Optional	Precision only

Only the first row is a reason to add a variable. The fourth is a reason you may add one, and it buys a smaller standard error rather than a different estimate. The middle two are reasons not to.

A variable’s kind is not visible in the data. Confounder, mediator and collider all produce a coefficient that moves when you adjust, and all three improve fit if they predict the outcome. The only way to tell them apart is to know which came first and what caused what — which is why the diagram gets drawn before the model is fitted.

Drawing it before fitting

Three arrows, drawn from what you know about how the pathway works.

   disability -------------------------------> completion
       |                                            ^
       +-------> referral made ---------------------+

   department --> disability        department --> completion

Department is a confounder — where a case arises affects both who is registered with a disability and how well the service system works — so it belongs in the model. Referral made is a mediator — disability affects it and it affects completion — so it does not.

TOTAL_EFFECT = ["disability", "case_category", "age_band", "sex",
                "service_requested", "admin1"]        # confounders only
MECHANISM = TOTAL_EFFECT + ["referral_made"]          # a different question

# Two named model specifications, two questions, both legitimate.

Name the models after the question rather than after the variables. A model called MECHANISM will not accidentally be reported as the total effect, and a reviewer can see which one was intended.

The collider, which is worse

A mediator understates a real effect. A collider manufactures one that is not there, and the mechanism is subtler.

Restricting to closed cases is the most common way it happens here, because a closed case is one where something was resolved — which is downstream of both disability and completion.

closed = d[d["case_status"].isin(["closed-resolved", "closed-lost-contact"])]
print(f"closed cases: {len(closed)}")
print(closed.groupby("disability")["completed"].agg(["mean", "size"]).round(4))

d |> filter(case_status %in% c("closed-resolved", "closed-lost-contact")) |>
  summarise(rate = mean(completed), n = n(), .by = disability)

Sample	n	Gap (average marginal effect)
All consenting cases	1,581	−19.4 points
Closed cases only	860	−21.1 points

Restricting to closed cases moves the gap to 21.1 points, and the movement is produced by the restriction rather than by anything about disability. Open cases — the ones still in progress — are excluded on a criterion that both variables influence.

Selecting rows is adjusting for a variable. Dropping open cases, keeping only completed forms, analysing only households that answered every question: each is a conditioning step, and each can be a collider. “We restricted the analysis to closed cases for completeness” is a sentence that changes an estimate without anyone noticing it was a modelling decision.

Report both, and say which is which

Mediation is worth reporting when the mechanism is the programme’s question, which here it is: the protection course located the disability gap at referral-making and acceptance, and this decomposition is the quantified version.

Referral completion by disability status, decomposition

  Total gap                                    -19.4 points
    of which runs through referral-making      -8.3 points  (42.5%)
    remaining, among referrals made            -11.2 points

  Referrals are made for 53.8% of cases reporting a disability against 71.5%
  of others. Both gates contribute; neither explains the other away.

  The -11.2 figure is conditional on a referral having been made and must not
  be reported as the effect of disability on reaching a service.

The last line is the one that keeps the decomposition honest. Both numbers are real, they answer different questions, and the failure mode is not computing the wrong one — it is computing the right one and labelling it as the other.

What comes next

Every model in this course so far has assumed each row is an independent observation. The next lesson gives the model a term for the thing the rows are grouped inside, and finds three honest routes to the same answer where the naive one was out on its own.