Two coefficients cross the line and one crosses back

Slide 1 / 26

What this lesson covers

• The design the sample was drawn under
• What the weights do to an estimate
• The same model, weighted and not
• Weights, clusters and strata are three separate things
• When to leave the weights out
• Report it whole
• What comes next

Slide 2 / 26

The design the sample was drawn under — In Python

import pandas as pd

survey = pd.read_csv("household-survey-2025.v1.csv")
frame = pd.read_csv("household-survey-frame-2025.v1.csv")

print(frame.groupby("stratum")["households"].sum())
print(survey.groupby("stratum").size())

Slide 3 / 26

The design the sample was drawn under — In R

library(dplyr)
frame |> summarise(households = sum(households), .by = stratum)
survey |> count(stratum)

Slide 4 / 26

The design the sample was drawn under

Slide 5 / 26

The design the sample was drawn under

• Rural remote is 11.0% of the population and 34.4% of the sample — That is a deliberate design choice — the survey…

Slide 6 / 26

What the weights do to an estimate — In Python

stratum_households = frame.groupby("stratum")["households"].sum()
base_weight = stratum_households / (25 * 14)
interviewed = frame[frame["selected"]].set_index("ea_id")["households_interviewed"]

survey["weight"] = (survey["stratum"].map(base_weight)
                    * 14 / survey["ea_id"].map(interviewed))
assert abs(survey["weight"].sum() - frame["households"].sum()) < 1

unweighted = survey["food_insecure"].mean()
weighted = np.average(survey["food_insecure"], weights=survey["weight"])
print(f"unweighted {unweighted:.1%}   weighted {weighted:.1%}")

Slide 7 / 26

What the weights do to an estimate — In R

survey <- survey |>
  mutate(weight = base_weight[stratum] * 14 / households_interviewed)

c(unweighted = mean(survey$food_insecure),
  weighted   = weighted.mean(survey$food_insecure, survey$weight))

Slide 8 / 26

What the weights do to an estimate

• 32.8% unweighted, 29.1% weighted — Rural remote has the highest food insecurity (46.4%), and over-representing it by…
• A regression fitted on this file without weights inherits exactly that problem — and it is not fixed by adding…

Slide 9 / 26

The same model, weighted and not — In Python

import statsmodels.api as sm
import statsmodels.formula.api as smf

FORMULA = ("food_insecure ~ improved_water_source + displacement_status"
           " + main_livelihood + household_size")

plain = smf.glm(FORMULA, data=d, family=sm.families.Binomial()).fit(
    cov_type="cluster", cov_kwds={"groups": d["ea_id"]})

weighted = smf.glm(FORMULA, data=d, family=sm.families.Binomial(),
                   freq_weights=d["weight"]).fit(
    cov_type="cluster", cov_kwds={"groups": d["ea_id"]})

Slide 10 / 26

The same model, weighted and not — In R

library(survey)

design <- svydesign(ids = ~ea_id, strata = ~stratum, weights = ~weight,
                    data = survey, nest = TRUE)
svyglm(food_insecure ~ improved_water_source + displacement_status +
         main_livelihood + household_size, design = design,
       family = quasibinomial())

Slide 11 / 26

The same model, weighted and not

Slide 12 / 26

The same model, weighted and not

• Three coefficients change which side of the line they sit on, in both directions — Weighting is not a correction that…
• Livestock livelihoods are concentrated in the over-sampled remote stratum — Unweighted, they carry a third of the…

Slide 13 / 26

Weights, clusters and strata are three separate things

Slide 14 / 26

Weights, clusters and strata are three separate things

• Weights change the coefficient; clusters and strata change its interval — That is the same distinction as…

Slide 15 / 26

Weights, clusters and strata are three separate things — In Python

print(f"clusters: {d['ea_id'].nunique()} enumeration areas")
print(f"strata:   {d['stratum'].nunique()}")
print(f"weights:  {d['weight'].min():.1f} to {d['weight'].max():.1f}")

Slide 16 / 26

Weights, clusters and strata are three separate things — In R

# survey::svydesign wants all three; a glm with weights= handles only one.

Slide 17 / 26

Weights, clusters and strata are three separate things

• glm(..., weights=) is not survey-weighted regression — It applies the weights and computes standard errors as though…

Slide 18 / 26

When to leave the weights out

• A within-stratum question — "Among rural remote households, does an improved water source go with lower food…
• A model whose covariates include everything the weights are built from — If stratum and selection probability are fully…

Slide 19 / 26

When to leave the weights out — In Python

print(f"unweighted {plain.params['main_livelihood[T.livestock]']:.3f}")
print(f"weighted   {weighted.params['main_livelihood[T.livestock]']:.3f}")

Slide 20 / 26

When to leave the weights out — In R

# A large gap between the two is evidence the design is not in the model.

Slide 21 / 26

When to leave the weights out

• A large difference between the weighted and unweighted fit is itself diagnostic — and it is the check to run before…

Slide 22 / 26

Report it whole — Example (cont.)

Food insecurity, household survey 2025

  976 households with complete covariates (of 996 interviewed),
  75 enumeration areas, 3 strata.
  Survey-weighted logistic regression; weights sum to 56,428 households,
  which is the frame total. Standard errors account for clustering by
  enumeration area and for stratification.

  Weighted prevalence      29.1%   (unweighted 32.8%)

  Adjusted odds ratios (weighted):
    Improved water source          0.74   95% CI 0.51 to 1.08
    Returnee household             0.39   95% CI 0.15 to 0.97
    Farming livelihood             1.69   95% CI 1.04 to 2.75
    Livestock livelihood           1.26   95% CI 0.73 to 2.18
    Salaried livelihood            0.45   95% CI 0.24 to 0.87

Slide 23 / 26

Report it whole — Example (cont.)

    Household size, per person     1.08   95% CI 1.01 to 1.16

  The unweighted model makes livestock livelihoods significant (OR 1.51,
  95% CI 1.04 to 2.18) and household size not. The rural remote stratum is
  11% of households and 34% of interviews, and livestock is concentrated
  there; the unweighted result is an artefact of the sampling design.

  Odds ratios are reported because food insecurity is 29% and the ratios are
  moderate; the weighted risk difference for salaried livelihoods is -14.2
  points against the observed mix of livelihoods, which is the number in
  the summary.

Slide 24 / 26

Report it whole

• The paragraph naming the artefact is what a reviewer will check first — and writing it yourself is better than having…

Slide 25 / 26

What comes next

• Every model so far has been fitted and read.

Slide 26 / 26

Where this goes next

Read the full lesson, with runnable code Back to the lesson