The recount, and the number it produces

Accuracy is the dimension that needs a visit

The other four dimensions come out of the extract. This one does not. Accuracy means “does the reported figure match the source”, and the source is a paper register in a cupboard, a tally sheet, or a tablet that has not synced since March.

The method is the oldest one in this course and it has not been improved on: go and count it again. Then divide.

The verification factor

verification factor = recount / reported

What you counted from the source, over what was reported upward. A VF of 1.00 means the report matched the register. It is a ratio, not a percentage of error, and getting that the right way round matters when you write it down.

Both directions are findings, and they are not the same finding.

• VF above 1 — the register holds more than was reported. Under-reporting. Usually a transcription or aggregation loss: a column not added up, a page missed, an electronic extract that dropped values it did not recognise. The programme did more work than it got credit for.
• VF below 1 — the report claims more than the register supports. Over-reporting. Sometimes double-counting or a definition mismatch; occasionally something worse. This is the direction that gets a grant suspended, so it is the direction to be most careful about asserting.

Compute it on a register you have

Take the school attendance register. The electronic extract carries true and false marks; one school also recorded some days as Y and N, which the extract’s boolean cast did not recognise. A verification visit reading the paper register would count those days as present. The extract did not.

That gives a real recount and a real reported figure, on the same data.

import pandas as pd

attendance = pd.read_csv("school-attendance-2024.v1.csv")
roster = pd.read_csv("school-roster-2024.v1.csv")

# The two students on the roster twice are ambiguous; exclude them from the
# verification and say so, rather than assigning them arbitrarily.
duplicated = roster["student_id"].duplicated(keep=False)
clean_roster = roster[~duplicated]

marks = attendance.merge(
    clean_roster[["student_id", "school_id"]], on="student_id",
    how="inner", validate="many_to_one",
)

vf = (
    marks.assign(
        reported=marks["present"] == "true",
        recount=marks["present"].isin(["true", "Y"]),
    )
    .groupby("school_id")[["recount", "reported"]]
    .sum()
)
vf["verification_factor"] = vf["recount"] / vf["reported"]
print(vf.sort_values("verification_factor", ascending=False).head())

library(dplyr)

clean_roster <- roster |>
  group_by(student_id) |> filter(n() == 1) |> ungroup()

vf <- attendance |>
  inner_join(select(clean_roster, student_id, school_id), by = "student_id",
             relationship = "many-to-one") |>
  summarise(
    recount  = sum(present %in% c("true", "Y")),
    reported = sum(present == "true"),
    .by = school_id
  ) |>
  mutate(verification_factor = recount / reported) |>
  arrange(desc(verification_factor))

School	Recount	Reported	VF
SCH09	2,456	1,734	1.416
SCH01	2,945	2,945	1.000
SCH02	2,569	2,569	1.000
… 22 more			1.000
District	61,764	61,042	1.012

The district figure is the trap

Read the last row first, because it is the row that appears in most reports. The district verification factor is 1.012 — one and a bit percent — and no auditor in the world would raise a finding on that.

Now read the first row. One school in twenty-four under-reported its attendance by 42%, and it is invisible in the district total because the other twenty-three are exact.

This is the same argument the previous course made about averaging verification factors, and it is worth making twice because it is the most common way a DQA produces a clean bill of health for a district with a real problem in it. Report the distribution and the exceptions. The mean is the least informative statistic available.

print(vf["verification_factor"].describe())
outliers = vf[(vf["verification_factor"] - 1).abs() > 0.05]
print(f"{len(outliers)} of {len(vf)} schools outside the tolerance band")

summary(vf$verification_factor)
vf |> filter(abs(verification_factor - 1) > 0.05)

The tolerance band

A DQA needs a threshold decided before you look, or every finding becomes a negotiation.

The convention most donor frameworks use is 0.95 to 1.05 — within 5% is acceptable, outside it is a finding. It is a convention, not a law, and two adjustments are legitimate:

• Widen it for small numbers. A facility reporting 12 cases has a VF of 1.08 if one case was missed. On small counts, use an absolute difference as well as a ratio: “outside 5% and more than 5 units”.
• Narrow it for money. Where the figure drives a payment — results-based financing, a per-beneficiary reimbursement — 5% is a lot of money and the band is usually tighter, set by the contract rather than by you.

TOLERANCE = 0.05
MIN_ABSOLUTE = 5

vf["difference"] = vf["recount"] - vf["reported"]
vf["finding"] = (
    ((vf["verification_factor"] - 1).abs() > TOLERANCE)
    & (vf["difference"].abs() > MIN_ABSOLUTE)
)
print(vf[vf["finding"]])

vf <- vf |>
  mutate(
    difference = recount - reported,
    finding = abs(verification_factor - 1) > 0.05 & abs(difference) > 5
  )

Write the band into the protocol before the visit. A threshold chosen after seeing the numbers is not a threshold, and everyone in the room will know it.

The recount is a protocol, not an afternoon

The arithmetic is trivial. Everything that makes the number trustworthy happens before it.

• Fix the period exactly. “March” is not a period. “1 to 31 March, by date of service, not date of entry” is.
• Fix the definition exactly. Count what the indicator’s numerator says, not what the register’s column heading says. Most VFs far from 1 are definition mismatches, and they are not errors by anyone.
• Recount independently. The person who compiled the report should not do the recount. Not because of dishonesty — because they will reproduce the same interpretation, and the interpretation is what you are testing.
• Record what you counted from. Register, tally sheet, individual cards. Two sources in one facility often disagree, and which one you counted is part of the finding.

What a VF cannot tell you

Three limits worth stating in the report, because someone will over-read the number otherwise.

It cannot tell you the register is right. A VF of 1.00 means the report matches the register. If the register itself was filled in at the end of the week from memory, both are wrong together and the VF is silent.

It does not generalise from the facilities you visited unless you sampled properly. Which is the next lesson.

It is not an accusation. A VF of 1.42 in one school, with twenty-three others at exactly 1.00, is the shape of a systems defect — one school’s data entry convention, unrecognised by the extract — not of one head teacher inflating numbers. Writing it up as the latter guarantees the next round is worse.

What comes next

You cannot visit thirty-eight facilities. The next lesson is how to choose the ones you do visit, how many is enough, and what a sample of six licenses you to say about the other thirty-two.