Two gaps, two answers

The two questions module 4 deferred

The education course found boys over-age more often than girls and declined to call it a finding. The protection course found a disability gap in referral completion and called it the most important result in the dataset. Both said “compute the interval first”. This is that computation.

import pandas as pd
import numpy as np
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

enrolment = pd.read_csv("school-enrolment-2024.v1.csv")
clean = enrolment[(enrolment["age_years"] <= 20)
                  & (enrolment["school_year"] == 2024)
                  & enrolment["grade"].between(1, 6)]
clean = clean.assign(over_age=clean["age_years"] > clean["grade"] + 5)

boys = clean[clean["sex"] == "m"]["over_age"]
girls = clean[clean["sex"] == "f"]["over_age"]
print(f"boys  {boys.sum()}/{len(boys)} = {boys.mean():.1%}")
print(f"girls {girls.sum()}/{len(girls)} = {girls.mean():.1%}")

library(dplyr)

enrolment |>
  filter(age_years <= 20, school_year == 2024, between(grade, 1, 6)) |>
  mutate(over_age = age_years > grade + 5) |>
  summarise(k = sum(over_age), n = n(), .by = sex)

Gap one: the one that is not there

Group	Over-age	n	95% interval
Boys	39.2%	510	35.1–43.5%
Girls	33.8%	542	29.9–37.8%

The two intervals overlap. Overlapping intervals are a hint and not a test — the correct thing to compute is an interval on the difference.

count = np.array([boys.sum(), girls.sum()])
nobs = np.array([len(boys), len(girls)])

stat, pvalue = proportions_ztest(count, nobs)
low, high = confint_proportions_2indep(count[0], nobs[0], count[1], nobs[1])
print(f"difference {boys.mean() - girls.mean():+.1%}")
print(f"95% CI [{low:+.1%}, {high:+.1%}]")
print(f"z = {stat:.2f}, p = {pvalue:.3f}")

prop.test(c(200, 183), c(510, 542))

Difference +5.5 points, 95% CI −0.4 to +11.3, p = 0.066.

The interval includes zero. So the honest answer is not “there is no gap” — it is “this survey cannot tell you whether there is a gap, and if there is one it is somewhere between girls being slightly worse off and boys being eleven points worse off”.

That is a genuinely useful sentence and it is not the same as a null result.

Gap two: the one that is

protection = pd.read_csv("protection-referrals-2024.v1.csv")
consenting = protection[protection["consent_to_refer"]]
reached = (consenting["referral_accepted"]
           & consenting["days_to_first_service"].notna())

disability = consenting["disability_reported"].isin([True, "true", "Yes"])
k = np.array([reached[disability].sum(), reached[~disability].sum()])
n = np.array([disability.sum(), (~disability).sum()])

stat, pvalue = proportions_ztest(k, n)
low, high = confint_proportions_2indep(k[0], n[0], k[1], n[1])
print(f"{k[0]}/{n[0]} = {k[0]/n[0]:.1%}   {k[1]}/{n[1]} = {k[1]/n[1]:.1%}")
print(f"difference {k[0]/n[0] - k[1]/n[1]:+.1%}, CI [{low:+.1%}, {high:+.1%}]")
print(f"z = {stat:.2f}, p = {pvalue:.2e}")

prop.test(c(54, 663), c(202, 1436))

Group	Completion	n
Disability reported	26.7%	202
Not reported	46.2%	1,436

Difference −19.4 points, 95% CI −26.1 to −12.8, p < 0.001.

Every value in that interval is a substantial gap. The smallest gap consistent with the data is 12.8 points, which is still large enough to act on. That is what makes this a finding rather than a hint.

The two results side by side

results = pd.DataFrame([
    {"comparison": "over-age, boys vs girls", "diff": 5.5,
     "ci": "-0.4 to +11.3", "n": "510 vs 542", "p": 0.066},
    {"comparison": "completion, disability", "diff": -19.4,
     "ci": "-26.1 to -12.8", "n": "202 vs 1436", "p": 0.0000002},
])
print(results)

# Two rows. The conclusion column is the analyst's, not the test's.

Notice that the significant result has the smaller sample in one arm. 202 cases produced a decisive answer and 510 students did not, because the effect size is nearly four times larger. Sample size and effect size trade off, and it is the combination that determines whether a comparison resolves.

Choosing the test

Three tests cover almost everything a programme report compares, and the choice is made by what the outcome is rather than by what feels sophisticated.

Comparing	Test	In this course
Two proportions	Two-sample z-test, or prop.test	Both gaps above
Two means	Welch’s t-test	Attendance by feeding, lesson 6
A categorical against a categorical	Chi-square	Closure reason by district

from scipy import stats

points = pd.read_csv("water-point-monitoring-2024.v1.csv")
table = pd.crosstab(points["admin2"], points["functional_status"])
chi2, p, dof, expected = stats.chi2_contingency(table)
print(f"chi-square {chi2:.1f}, dof {dof}, p = {p:.4f}")
print(f"smallest expected cell: {expected.min():.1f}")

chisq.test(table(points$admin2, points$functional_status))

Check the assumptions, and say you did

Each test assumes things, and two of them are checkable in one line.

Independence. Every test above assumes each row is an independent observation. It is the assumption most often violated and the hardest to see — lesson 6 is entirely about a case where it fails.

Expected cell counts for chi-square. The test is unreliable when an expected cell falls below about five. Print expected.min() every time; if it is small, collapse categories or use Fisher’s exact test.

Equal variances for the t-test. Do not check it — use Welch’s t-test always, which does not assume it. scipy.stats.ttest_ind(..., equal_var=False) and R’s t.test default. The version that assumes equal variances buys nothing and fails when the groups differ in spread.

girls_rate, boys_rate = girls.mean(), boys.mean()
print("assumptions checked:")
print(f"  independence: one row per student, no student in two rows — verified")
print(f"  sample sizes: {len(boys)} and {len(girls)}, both well above 30")
print(f"  expected counts: smallest is {min(len(boys), len(girls)) * min(girls_rate, 1 - boys_rate):.0f}")

# Write the check into the script, not into your memory of having done it.

Report the difference, not the two numbers

Over-age enrolment by sex, primary, 2024

  Boys    39.2%   n = 510
  Girls   33.8%   n = 542
  Difference +5.5 points, 95% CI -0.4 to +11.3, p = 0.066

  The interval includes zero. This survey cannot establish whether over-age
  enrolment differs by sex; if it does, the gap is between 0 and 11 points
  and boys are the disadvantaged group. Not reported as a finding.

Referral completion by disability status, 1,638 consenting cases

  Disability reported     26.7%   n = 202
  Not reported            46.2%   n = 1,436
  Difference -19.4 points, 95% CI -26.1 to -12.8, p < 0.001

  The smallest gap consistent with the data is 12.8 points. Reported as a
  finding, and the pathway lesson locates it at referral-making and
  acceptance rather than at consent.

Both blocks report the difference and its interval as the headline, with the two group figures beneath. That ordering is deliberate: the difference is the claim, and the two proportions are the evidence for it.

What comes next

One of these gaps is statistically significant. The next lesson asks whether that is the same thing as mattering — and finds a comparison where a p-value below 0.001 describes a difference no programme would act on.