88% attendance, 62% of students

One register, two questions

import pandas as pd

attendance = pd.read_csv("school-attendance-2024.v1.csv")
print(f"{len(attendance):,} rows, {attendance['student_id'].nunique():,} students")
print(attendance["present"].value_counts(dropna=False))

library(dplyr)
attendance |> count(present)

The present column has five values, not two. SCH09 recorded Y and N instead of true and false on 873 of its rows, and 271 days were never marked either way. A boolean cast turns those 873 real observations into missing values and silently thins one school out of the denominator.

MARKS = {"true": True, "Y": True, "false": False, "N": False}
marked = attendance[attendance["present"].isin(MARKS)].copy()
marked["attended"] = marked["present"].map(MARKS)
print(f"usable marks: {len(marked):,} of {len(attendance):,}")

attendance |>
  mutate(attended = case_when(present %in% c("true", "Y") ~ TRUE,
                              present %in% c("false", "N") ~ FALSE)) |>
  filter(!is.na(attended))

Now the same file answers two different questions.

Average daily attendance

ada = marked["attended"].mean()
print(f"average daily attendance: {ada:.1%}")

marked |> summarise(ada = mean(attended))

88.4%. Every mark counts once, so a student present 60 days out of 60 and one present 30 out of 60 contribute in proportion to how often they were marked.

This is the number a ministry reports and a system-level indicator should be. It answers how full is the classroom on an average day, which is the right question for staffing, feeding and textbook planning.

The proportion regularly attending

by_student = marked.groupby("student_id")["attended"].agg(["mean", "size"])
eligible = by_student[by_student["size"] >= 20]

for threshold in (0.80, 0.85, 0.90):
    share = (eligible["mean"] >= threshold).mean()
    print(f"attending at least {threshold:.0%} of marked days: {share:.1%}")

marked |>
  summarise(rate = mean(attended), days = n(), .by = student_id) |>
  filter(days >= 20) |>
  summarise(across(everything(), ~ mean(rate >= 0.9)))

Threshold	Students meeting it
At least 80% of days	87.3%
At least 85%	78.9%
At least 90%	62.1%

62.1% at the 90% threshold, against an average daily attendance of 88.4%. The two numbers are twenty-six points apart and neither is wrong.

A mean over marks describes the system; a proportion over students describes children. A school where every child misses one day in eight and a school where seven children in eight attend perfectly while one never comes have the same average daily attendance and completely different problems.

Which one a programme is judged on

Report both, and lead with the one that matches the decision.

The decision	The number
How many meals, desks, textbooks	Average daily attendance
Which children need a follow-up visit	Proportion below the threshold
Whether an intervention worked	Both, because they can move in opposite directions

That last row is the one to internalise. An intervention that brings the worst attenders from 40% to 60% moves average daily attendance barely at all and moves the proportion above 90% not at all — and would be recorded as a failure by either number alone.

struggling = eligible[eligible["mean"] < 0.75]
print(f"students below 75%: {len(struggling)} "
      f"({len(struggling) / len(eligible):.1%})")
print(f"their marks as a share of all marks: "
      f"{struggling['size'].sum() / eligible['size'].sum():.1%}")

# The students furthest behind are a small share of the rows and the whole
# of the problem.

State the threshold, and where it came from

The 90% threshold is a convention, not a standard, and it does most of the work in that 62.1%. Move it to 85% and the figure becomes 78.9%.

sensitivity = {f"{t:.0%}": f"{(eligible['mean'] >= t).mean():.1%}"
               for t in (0.75, 0.80, 0.85, 0.90, 0.95)}
print(sensitivity)

# Print the curve, not the point.

At 95% it is 39.8% and at 75% it is 90.5%. Publish the threshold beside the number, every time. Two reports quoting “the proportion of students regularly attending” at different thresholds produce incomparable figures that both look official — the same failure as the two Food Consumption Score threshold sets, in a different sector.

The denominator decision nobody makes explicitly

by_student above was filtered to students with at least twenty marked days. That is a choice and it has to be declared.

Without it, a student who enrolled in the last week of term and attended three days out of three appears as a 100% attender, and a student marked twice appears in the tail.

With it, you have excluded exactly the late-arriving and early-leaving students, who are the ones an attendance programme most wants to see.

print(f"students with any marks:      {len(by_student)}")
print(f"students with 20 or more:     {len(eligible)}")

# Two denominators, both defensible, and the report says which.

Here the two are nearly identical, so the choice does not move the answer. In a register covering a full year it would move it a great deal, and the habit of declaring it is what makes the figure portable.

Report the pair

Attendance, February to April 2024

  Average daily attendance         88.4%   69,974 usable marks
  Students attending 90%+ of days  62.1%   1,200 students with 20+ marks
  Students attending 85%+          78.9%
  Students below 75%                9.5%   114 students, the follow-up list

  SCH09 recorded 873 marks as Y/N rather than true/false; recoded, not
  dropped. 271 days were never marked and are excluded from both figures.

What comes next

Both numbers in this lesson divide by days that were marked. The next lesson is about the days that are not in the file at all — fifteen of them, at two schools, which turn a strike into a dropout emergency if you let them.