cassionData Analysis

Back to the lessonLesson 2 of 8From intention to indicator

The anatomy of an indicator

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 18

    What this lesson covers

    • What it is made of
    • Four families, and choosing between them
    • Three numbers that look like one
    • The unit of measure belongs in the name
    • Direction: say which way is better
    • Leading and lagging
    • Three questions before any code
    • An indicator you cannot compute is not an indicator
    • What comes next
    Speaker notes
    Four families, one unit of measure, a direction, and the three questions to answer before writing any code. Plus why 4,218 screenings, 4,206 registrations and the number of children are three different indicators.

  2. Slide 2 / 18

    What it is made of

    • A population — who or what is being counted.
    • A condition — what has to be true of them to be in the numerator.
    • A denominator — what they are counted against, unless it is a plain count.
    • A period — over what stretch of time.
    • A unit of measure — people, doses, episodes, percent, litres per person per day.
    Speaker notes
    Strip the framework language away and an indicator has five parts: Miss any one and the indicator is ambiguous. Miss the last and it is unusable in a sentence, which is where it will end up.

  3. Slide 3 / 18

    Four families, and choosing between them

    FamilyShapeExampleWatch for
    CountA number of thingsChildren screenedNo denominator; cannot be compared across places
    ProportionPart over whole, both same unitsShare of children with MUAC under 125 mmNumerator must be a subset of the denominator
    RateEvents over population-timeNew admissions per 1,000 under-fives per monthThe time unit must be in the label
    Ratio / indexTwo quantities not nestedPenta1-to-penta3 dropout; Food Consumption ScoreCan exceed 1; direction is not obvious

  4. Slide 4 / 18

    Four families, and choosing between them — In Python

    screened = len(muac)
referred = muac["outcome"].str.startswith("referred").sum()

print(f"{referred} referred of {screened} screened = {referred / screened:.1%}")
    Speaker notes
    The commonest design error is reporting a count where the audience will read a proportion. "375 children referred" sounds like a lot or a little depending entirely on how many were screened, and the reader will supply their own denominator if you do not.

  5. Slide 5 / 18

    Four families, and choosing between them — In R

    screened <- nrow(muac)
referred <- sum(startsWith(muac$outcome, "referred"))

sprintf("%d referred of %d screened = %.1f%%", referred, screened, 100 * referred / screened)
    Speaker notes
    375 of 4,218, or 8.9%. Report both. The count is what a logistics officer orders supplies against; the proportion is what tells you whether this commune is worse than that one.

  6. Slide 6 / 18

    Three numbers that look like one — In Python

    print("screening events:      ", len(muac))
print("distinct child_id:     ", muac["child_id"].nunique())
    Speaker notes
    Here is the ambiguity that costs the most time in practice. The screening register supports three different "number of children" indicators.

  7. Slide 7 / 18

    Three numbers that look like one — In R

    c(events = nrow(muac), ids = dplyr::n_distinct(muac$child_id))

  8. Slide 8 / 18

    Three numbers that look like one

    • 4,218 screening events. What the community health workers did. The right numerator for workload, supply consumption…
    • 4,206 distinct identifiers. Twelve forms were submitted twice on a poor connection, so twelve events are duplicates…
    • About 4,200 children. Six more children were re-registered under a new identifier — findable only by the record…
    Speaker notes
    Three defensible numbers, one register, and the difference between the largest and the smallest is 0.4%. The size of the gap is not the point. The point is that "children screened" does not say which one, and the moment a donor compares your figure against a partner's, whichever of the three each of you chose determines whether you agree. So the indicator name has to carry it: screening_events, children_screened_deduplicated. Never children_screened on its own.

  9. Slide 9 / 18

    The unit of measure belongs in the name — In Python

    indicators = {
    "children_screened_count": screened,
    "gam_prevalence_percent": round(100 * gam_cases / assessed, 1),
    "admissions_per_1000_under5_per_month": round(1000 * admissions / under5 / 12, 2),
}

  10. Slide 10 / 18

    The unit of measure belongs in the name — In R

    indicators <- list(
  children_screened_count              = screened,
  gam_prevalence_percent               = round(100 * gam_cases / assessed, 1),
  admissions_per_1000_under5_per_month = round(1000 * admissions / under5 / 12, 2)
)
    Speaker notes
    Long names, and they are worth it. A column called rate in a spreadsheet emailed between four organisations will be multiplied by 100 by somebody, divided by 12 by somebody else, and compared against a figure with a different denominator by a third. The name is the cheapest defence available.

  11. Slide 11 / 18

    Direction: say which way is better

    • Higher is better — coverage, completion, attendance.
    • Lower is better — dropout, defaulter rate, prevalence of acute malnutrition.
    • Neither, on its own — referral counts. More referrals can mean better case finding or a deteriorating situation,…
    Speaker notes
    Every indicator needs a stated direction, because a surprising number are ambiguous. That third category is larger than people expect and it is where dashboards go wrong: a red-amber-green traffic light applied to an indicator with no direction produces an alarm whose meaning nobody can state. If you cannot say which way is better, the indicator needs a partner indicator, not a colour.

  12. Slide 12 / 18

    Leading and lagging

    • Lagging — tells you what happened. GAM prevalence, cure rate, annual coverage. Accurate, essential for…
    • Leading — moves early and predicts. Stock-out days, defaulter rate in the first two weeks of treatment,…
    Speaker notes
    A related distinction that decides whether an indicator is any use for management. A monitoring system built entirely of lagging indicators reports faithfully on things nobody can now change. Aim for at least one leading indicator per outcome, and expect it to be noisier — that is the trade you are making.

  13. Slide 13 / 18

    Three questions before any code

    • 1. What decision does this number inform? — If the answer is "the donor asks for it", say so honestly and keep it cheap
    • 2. What would make it move, other than the thing I care about? — Coverage moves with the denominator, with reporting…
    • 3. Who else already computes this, and how? — Almost always somebody does — UNICEF, WHO, the cluster, the ministry
    Speaker notes
    Answer these in writing before opening an editor. They take five minutes and they prevent the rewrite. 1. What decision does this number inform? If the answer is "the donor asks for it", say so honestly and keep it cheap. If two different decisions come out, you have two indicators wearing one name. 2. What would make it move, other than the thing I care about? Coverage moves with the denominator, with reporting completeness, with population movement, and occasionally with vaccination. Listing the alternatives is how you know what to report alongside it. 3. Who else already computes this, and how? Almost always somebody does — UNICEF, WHO, the cluster, the ministry. Lesson 6 is entirely about that, and the answer changes your definition more often than not.

  14. Slide 14 / 18

    An indicator you cannot compute is not an indicator — In Python

    REQUIRED = {"child_id", "commune", "screening_date", "muac_mm", "oedema", "outcome"}
missing = REQUIRED - set(muac.columns)
assert not missing, f"indicator cannot be computed; missing {missing}"
    Speaker notes
    The last check, and it kills more proposed indicators than any other. Before it goes into a LogFrame, confirm the data exists: which system holds it, at what grain, how often, and who extracts it. An indicator whose means of verification is "programme records" is a promise nobody has checked.

  15. Slide 15 / 18

    An indicator you cannot compute is not an indicator — In R

    REQUIRED <- c("child_id", "commune", "screening_date", "muac_mm", "oedema", "outcome")
stopifnot(all(REQUIRED %in% names(muac)))

  16. Slide 16 / 18

    An indicator you cannot compute is not an indicator

    An indicator is a promise that a number can be produced, repeatedly, to the same definition. Everything else in this course is about keeping that promise.
    Speaker notes
    The Data Quality Assessment course found timeliness unmeasurable because a field was not in the export. That is this check, discovered eighteen months late.

  17. Slide 17 / 18

    What comes next

    • You have the anatomy.
    Speaker notes
    You have the anatomy. The next lesson turns it into the artefact this course is built around — the reference sheet, field by field, and the test that tells you whether yours is finished.

  18. Slide 18 / 18

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson