r = 0.039, and everyone expected 0.5

The correlation that is not there

import pandas as pd
import numpy as np

assessment = pd.read_csv("learning-assessment-2024.v1.csv")
attendance = pd.read_csv("school-attendance-2024.v1.csv")
roster = pd.read_csv("school-roster-2024.v1.csv")

MARKS = {"true": True, "Y": True, "false": False, "N": False}
marked = attendance[attendance["present"].isin(MARKS)].copy()
marked["attended"] = marked["present"].map(MARKS)
rate = marked.groupby("student_id")["attended"].mean().rename("attendance")

endline = (assessment[assessment["assessment_round"] == "endline"]
           .pivot_table(index="student_id", columns="domain", values="raw_score"))
joined = endline.join(rate, how="inner").join(roster.set_index("student_id"))

print(joined[["literacy", "numeracy", "attendance"]].corr().round(3))

library(dplyr)

joined |> select(literacy, numeracy, attendance) |> cor(use = "complete.obs")

Pair	r	n
Attendance and endline literacy	0.039	659
Attendance and endline numeracy	−0.006	659
Literacy and numeracy	0.849	659

Attendance and literacy correlate at 0.039. Every education programme document assumes that relationship and most quote a number for it. In this register it is absent.

That is a result, and it needs an interval before it is one.

An interval on a correlation

def fisher_ci(r, n, z=1.96):
    zr = np.arctanh(r)
    se = 1 / np.sqrt(n - 3)
    return np.tanh(zr - z * se), np.tanh(zr + z * se)

for label, r, n in [("attendance vs literacy", 0.039, 659),
                    ("literacy vs numeracy", 0.849, 659)]:
    lo, hi = fisher_ci(r, n)
    print(f"{label:24} r = {r:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]")

cor.test(joined$attendance, joined$literacy)

r = 0.039, 95% CI −0.037 to +0.115. The interval contains zero and every value inside it is negligible, so this is not an underpowered null — 659 students are enough to say that if a relationship exists it is too small to matter.

Contrast the same interval on the second pair: r = 0.849, CI 0.826 to 0.869, where the whole interval is large.

A correlation with no interval is the same defect as a proportion with none, and cor.test gives it for free.

The correlation that is there and says nothing

Literacy and numeracy correlate at 0.849, which is the largest r in this course and the least interesting.

It is one child, one test day, two forty-item papers. A child who was ill, who had not eaten, who could not read the instructions, or who is simply a strong student scores similarly on both. The correlation is a property of the measurement, not a discovery about literacy and numeracy.

Ask what would have to be true for the correlation to be zero. Here the answer is “the same child would have to perform independently on two papers taken half an hour apart”, which nobody believes — so a high r carries no information.

A correlation is only informative when its absence was plausible, and that is a judgement about the world rather than about the data.

Square it before you describe it

for label, r in [("attendance vs literacy", 0.039),
                 ("attendance vs literacy, school level", 0.199),
                 ("literacy vs numeracy", 0.849)]:
    print(f"{label:38} r = {r:+.3f}   r-squared = {r**2:.3f}")

# r^2 is the share of variance, and it is much smaller than r looks.

Pair	r	r²	Reads as
Attendance and literacy	0.039	0.002	Two tenths of one per cent
Attendance and literacy, school level	0.199	0.040	Four per cent
Literacy and numeracy	0.849	0.721	Seventy-two per cent

An r of 0.199 sounds like a relationship and explains four per cent of the variation. Squaring is the cheapest defence against over-reading a middling correlation, and the number to put in a report is usually r² rather than r.

Pearson, Spearman, and eleven bad rows

Lesson 1 found eleven households whose total consumption was entered in a per-person column. Watch what those eleven rows do to a correlation.

wash = pd.read_csv("wash-household-survey-2024.v1.csv")
pair = wash.dropna(subset=["round_trip_minutes", "litres_per_person_day"])
clean = pair[pair["litres_per_person_day"] <= 80]

for label, frame in [("all rows", pair), ("11 rows removed", clean)]:
    p = frame["round_trip_minutes"].corr(frame["litres_per_person_day"])
    s = frame["round_trip_minutes"].corr(frame["litres_per_person_day"],
                                         method="spearman")
    print(f"{label:18} n = {len(frame):5}  Pearson {p:+.3f}  Spearman {s:+.3f}")

cor(pair$round_trip_minutes, pair$litres_per_person_day)                    # Pearson
cor(pair$round_trip_minutes, pair$litres_per_person_day, method = "spearman")

Rows	n	Pearson	Spearman
All	2,403	−0.185	−0.285
Eleven removed	2,392	−0.293	−0.287

Eleven rows in 2,403 cut Pearson by more than a third. Spearman did not move.

Pearson uses the values, so a household recorded at 277 litres per person pulls hard on the coefficient. Spearman uses the ranks, so a wrong value that is still the largest value changes nothing.

Report Spearman when either variable is skewed or has outliers you have not resolved, and Pearson when both are roughly symmetric and clean. A large gap between the two is a signal to go and look at the extremes — the same use the skewness statistic had in lesson 1.

The corrected answer is the useful one: longer trips to water go with less water used, r = −0.29 on 2,392 households, which is the relationship the WASH course predicted and the uncorrected file understated.

Correlated because something else moved both

prices = pd.read_csv("market-prices-2024.v1.csv")
wide = prices.pivot_table(index="period", columns="commodity",
                          values="price_htg", aggfunc="mean").sort_index()

print(f"beans and maize, levels:            {wide['beans-black'].corr(wide['maize']):.3f}")
changes = wide.diff().dropna()
print(f"beans and maize, month-on-month:    {changes['beans-black'].corr(changes['maize']):.3f}")

cor(wide$`beans-black`, wide$maize)
cor(diff(wide$`beans-black`), diff(wide$maize))

Beans and maize correlate at 0.986 in levels and 0.876 in month-on-month changes. Neither number means the price of beans moves the price of maize. Both series carry the same annual inflation and the same lean-season shock, and the correlation is measuring the shock.

Correlating two series over time will nearly always give a large number, because almost everything trends. Differencing removes the shared trend and what survives is the shared shock — which is a common cause, and still not a causal link between the two commodities.

The epidemiology course made this point with confounding. The correlation coefficient has no way to express it, which is why the sentence beneath it has to.

And the clustering from the last lesson

by_school = joined.groupby("school_id")[["attendance", "literacy"]].mean()
print(f"student level  r = {joined['attendance'].corr(joined['literacy']):.3f}  n = {len(joined)}")
print(f"school level   r = {by_school['attendance'].corr(by_school['literacy']):.3f}  n = {len(by_school)}")

# Same two variables, two units of analysis, two answers.

0.039 across 659 students and 0.199 across 24 schools. The school-level figure has an interval from −0.22 to +0.56 on 24 points, so it establishes nothing either — but it is five times larger, and an analyst who aggregated first and reported r without n would have written a different sentence.

Say which unit the correlation is between, every time. “Attendance correlates with literacy at 0.20” is unreadable without knowing whether the rows are children or schools.

Write the sentence

The coefficient is one number and the sentence beneath it is where the analysis is either honest or not.

Not this: “Attendance is correlated with literacy outcomes.” True of the school level, false of the student level, and unfalsifiable as written.

Nor this: “There is no relationship between attendance and literacy.” Stronger than the data supports; the interval reaches +0.12.

This: “Across 659 students, attendance over the February–April term shows no association with endline literacy (r = 0.04, 95% CI −0.04 to +0.12). The register cannot support the assumption that attendance drives learning in this cohort. Note that attendance varies little — the middle half of students sit between 86.7% and 97.8% — so this analysis has limited ability to detect a relationship at the low attendance where one would be expected.”

The third sentence is the one to copy. It states the finding, the interval, what it means for the programme, and the reason the analysis might have missed something real — and it is four lines.

Report it whole

Attendance and learning outcomes, endline 2024

  Attendance vs endline literacy   r = 0.04   95% CI -0.04 to +0.12   n = 659
  Attendance vs endline numeracy   r = -0.01  95% CI -0.08 to +0.07   n = 659

  Between students within the term. No association at student level.
  Aggregated to the 24 schools, r = 0.20 (95% CI -0.22 to +0.56), which is
  also consistent with no relationship.

  Attendance is compressed: the interquartile range is 86.7% to 97.8%, so
  students with the low attendance that a learning effect would show up at
  are rare in this cohort. This is a null result about the range observed,
  not about attendance in general.

The last paragraph is what stops the null being over-read, and it is the counterpart of the “underpowered, not null” sentence from lesson 4. A correlation of zero over a narrow range says nothing about what happens outside it.

What comes next

The last lesson assembles all of this into the statistics section of a report — what goes in, what stays in the appendix, and what a reviewer will ask for that this course has already computed.