The panel that caught a wrong number in this course

Slide 1 / 22

What this lesson covers

• What happened
• Why the documentation was the one that was right
• What a definition entry contains
• Where the panel lives
• The check the panel makes possible
• The argument this prevents every month
• Report it whole
• What comes next

Slide 2 / 22

What happened — In Python

# What the lesson had
IMPROVED = {
    "piped-into-dwelling", "piped-into-yard", "public-tap",
    "borehole", "protected-well", "protected-spring",
}

Slide 3 / 22

What happened — In R

# What the JMP definition actually lists
improved <- c("piped-into-dwelling", "piped-into-yard", "public-tap",
              "borehole", "protected-well", "protected-spring",
              "tanker-truck")     # packaged or delivered water

Slide 4 / 22

What happened

• The JMP improved list includes packaged or delivered water — which is tanker-truck in this dataset
• Nothing detected it for months — The lesson was internally consistent, the figure agreed with the lesson because it…

Slide 5 / 22

Why the documentation was the one that was right — Example

### 2.1 Improved source

| Numerator   | Households whose main drinking water source is piped, borehole,
|             | protected well, protected spring, rainwater or packaged water
| Denominator | All surveyed households
| Value       | 0.823
| Valid at    | Household, community, district
| Standard    | WHO/UNICEF JMP service ladders

Slide 6 / 22

Why the documentation was the one that was right

• Writing "or packaged water" in prose made it checkable against the standard — A Python set of six strings does not read…
• That is the whole argument for the panel — It is not documentation of the code — it is the specification the code is…

Slide 7 / 22

What a definition entry contains

Slide 8 / 22

What a definition entry contains

• "Valid at" prevents the commonest misuse — A number computed at district level and quoted at community level is the…
• "Decision informed" is what justifies the indicator existing — The WASH documentation writes "None

Slide 9 / 22

Where the panel lives

Slide 10 / 22

Where the panel lives — In Python

# The JMP improved list, which includes packaged or delivered water —
# `tanker-truck` here. Omitting it understates improved coverage by 2.6
# points and disagrees with the project's own indicator documentation.
IMPROVED = {...}

Slide 11 / 22

Where the panel lives — In R

# The comment is what makes the next reader check rather than assume.

Slide 12 / 22

Where the panel lives

• The tile clause is what a reader sees; the document is what a challenger reads; the comment is what the next analyst…

Slide 13 / 22

The check the panel makes possible

• Compute the indicator twice, from different code, and compare — That is what found the error here, and it is a check…

Slide 14 / 22

The check the panel makes possible — In Python

from_pipeline = summary["basic_service"]
from_documentation = 0.563          # the value the definition panel states
assert abs(from_pipeline - from_documentation) < 0.001, (
    f"pipeline {from_pipeline:.3f} disagrees with the definition panel"
)

Slide 15 / 22

The check the panel makes possible — In R

stopifnot(abs(basic_service - 0.563) < 0.001)

Slide 16 / 22

The check the panel makes possible

• A stated value in the definition panel is a test fixture — Once the panel carries the number, any pipeline that…
• And when they disagree, decide which is authoritative before changing either — Here the standard settled it: JMP lists…

Slide 17 / 22

The argument this prevents every month

"Coverage is 82%." — "We reported 76% last quarter." — "That was a different definition." — "Which one is right?"

Slide 18 / 22

The argument this prevents every month

• Both are right and they are different indicators — Without a panel that conversation takes twenty minutes every quarter…
• The commonest three disagreements in this sector — all of which a panel settles:
• Source against service — Improved source, 82.3%
• Numerator population against denominator population — Gross enrolment of 109.2% is a projected denominator, not more…
• Screening against survey — A screening rate is not a prevalence estimate, and the nutrition dashboard here says so in…

Slide 19 / 22

Report it whole — Example

Indicator definitions

  Full definitions in docs/indicator-definitions.md: numerator, denominator,
  the level it is valid at, the standard applied, and the decision it
  informs, for each of the nine indicators in this report.

  Each dashboard tile carries the definition compressed to one clause.

  The pipeline asserts its computed values against the values stated in the
  definitions document. A disagreement fails the build rather than reaching
  a report.

  Change log: 2026-07-29, improved source corrected from 79.7% to 82.3%.
  Delivered water (tanker truck) was omitted from the JMP improved list.
  The definitions document had it right; the analysis code did not.

Slide 20 / 22

Report it whole

• The change log entry is the part worth copying — An indicator that changed definition needs a dated line saying what…

Slide 21 / 22

What comes next

• The dashboard answers a question.

Slide 22 / 22

Where this goes next

Read the full lesson, with runnable code Back to the lesson