cassionData Analysis

Back to the lessonLesson 7 of 8Turning findings into change

From a defect to the thing that caused it

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 16

    What this lesson covers

    • A finding is not a cause, and only a cause can be fixed
    • Five categories, and they need different money
    • Let the data narrow it before you ask anyone
    • Then ask the form
    • The five whys, with a stopping rule
    • Test the cause against the data
    • Put the cause at the level the fix lives at
    • What comes next
    Speaker notes
    Five categories of cause, a stopping rule for the five whys, and why "the staff need training" is the answer that gets written when nobody looked. Test the cause against the data before you put it in the report.

  2. Slide 2 / 16

    A finding is not a cause, and only a cause can be fixed

    Recommendation. Train staff on data quality.
    Speaker notes
    At this point you have findings. Reporting collapsed to 29% in August. One school under-reported attendance by 42%. One survey team rounds heights to the nearest half centimetre. Twenty-four percent of ages are whole years. Every one of those is a what. None of them is a why, and a corrective action plan written against a what produces the sentence that appears in most DQA reports in this sector: That sentence is written when nobody looked for a cause. It is unfalsifiable, it is expensive, and it will not change August's reporting rate, because nothing about August was caused by staff not knowing what they were doing.

  3. Slide 3 / 16

    Five categories, and they need different money

    CategoryLooks likeFixed by
    DefinitionTwo sites counting different things under one indicator nameA written indicator reference sheet
    ToolA form that cannot express what happened, or a system that drops valuesA form or configuration change
    CapacityThe right tool used wrongly, consistently, by someone who was never shownSupervision and job aids, occasionally training
    IncentiveReporting that improves when it is looked at, or numbers that meet a target exactlyChanging what the number is used for
    SystemEveryone fails at once — no forms, no fuel, no network, no supervisorLogistics, budget, staffing
    Speaker notes
    Almost every data quality cause in routine programme data falls into one of five categories. Naming which one you are in tells you who has to fix it.

  4. Slide 4 / 16

    Five categories, and they need different money

    • Definition is the largest category and the least suspected — A verification factor far from 1 is more often two people…
    • Capacity is the smallest category and the most frequently blamed — because it is the only one where the fix is a…
    Speaker notes
    Two observations about that table, and both come up in every assessment. Definition is the largest category and the least suspected. A verification factor far from 1 is more often two people counting different things than anyone counting wrongly. It is also the cheapest to fix, and the fix — writing the definition down — prevents recurrence permanently. Capacity is the smallest category and the most frequently blamed, because it is the only one where the fix is a workshop and a workshop is procurable.

  5. Slide 5 / 16

    Let the data narrow it before you ask anyone — In Python

    by_month = vax.groupby(vax["period"].dt.strftime("%Y-%m"))["reported"].mean()
by_facility = vax.groupby("facility_id")["reported"].mean()

print("spread across months:  ", round(by_month.max() - by_month.min(), 2))
print("spread across facilities:", round(by_facility.max() - by_facility.min(), 2))
    Speaker notes
    The pattern of a defect tells you which category it is in, and you already have the data to see the pattern.

  6. Slide 6 / 16

    Let the data narrow it before you ask anyone — In R

    by_month <- vax |> mutate(m = format(period, "%Y-%m")) |>
  summarise(r = mean(report_submitted), .by = m)
by_facility <- vax |> summarise(r = mean(report_submitted), .by = facility_id)

c(months = diff(range(by_month$r)), facilities = diff(range(by_facility$r)))

  7. Slide 7 / 16

    Let the data narrow it before you ask anyone

    • Concentrated in time, spread across units. August, everywhere. That is a system cause — something failed for the…
    • Concentrated in one unit, spread across time. SCH09's Y/N coding, every month. That is a tool or definition…
    • Spread across both. Age heaping, every team, all year. That is a method cause, inherent to how the measurement is…
    • Write the shape down before you propose a cause — It rules out three of the five categories for free, and it is the…
    Speaker notes
    Three shapes, three categories: Write the shape down before you propose a cause. It rules out three of the five categories for free, and it is the step that stops a system failure being written up as twenty-seven capacity failures.

  8. Slide 8 / 16

    Then ask the form

    • What values can the form actually accept? SCH09's registers offered Y/N boxes; the extract's boolean cast…
    • Is the field required? An optional field with a 40% missing rate is a form design decision, not a compliance problem.
    • What does the field label say? A column headed "cases" collects whatever the person filling it believes a case is.
    Speaker notes
    For anything local, look at the instrument before you look at the person.

  9. Slide 9 / 16

    The five whys, with a stopping rule

    • Stop when you reach something a named person can change with a budget you can estimate — Anything past that point is…
    Speaker notes
    The technique is standard. What is usually missing is a rule for when to stop, which is why five whys so often terminates at "staff are not motivated". Stop when you reach something a named person can change with a budget you can estimate. Anything past that point is philosophy.

  10. Slide 10 / 16

    The five whys, with a stopping rule

    Attendance was under-reported by 42% in SCH09. — Why? The extract read Y as missing. — Why? The register used Y/N and the form expected true/false. — Why? The paper register printed in 2019 has yes/no boxes; the digital form was designed later, separately. — Stop. The education office can reprint the register or the HMIS team can accept both codings. Both are costed in an afternoon.
    Speaker notes
    Going further — why were they designed separately? — produces true statements about institutional coordination that no one in this assessment can act on.

  11. Slide 11 / 16

    Test the cause against the data — In Python

    # Cause: one school's register uses Y/N. Prediction: Y/N appears in that
# school only, and in most of its months.
yn = attendance[attendance["present"].isin(["Y", "N"])].merge(
    roster[["student_id", "school_id"]], on="student_id", how="left"
)
print(yn["school_id"].value_counts())
print(yn.groupby("school_id")["attendance_date"].nunique())
    Speaker notes
    A proposed cause makes a prediction. Check it before you write it down; it takes one query and it is the difference between a finding and a story.

  12. Slide 12 / 16

    Test the cause against the data — In R

    attendance |>
  filter(present %in% c("Y", "N")) |>
  left_join(select(roster, student_id, school_id), by = "student_id") |>
  count(school_id)
    Speaker notes
    If the Y/N values turned out to be spread across six schools, the cause is not one register — it is a regional convention, and the fix is different. A cause that survives its own test is worth writing; one that was never tested is a guess in the voice of a conclusion.

  13. Slide 13 / 16

    Put the cause at the level the fix lives at

    • A facility-level cause needs a facility-level action. "FAC020's tally sheet is missing" is fixed by delivering a…
    • A district-level cause needs a district-level action. "No facility received forms in August" is not thirty-eight…
    • A system-level cause needs a system-level action, and naming it is often the most valuable thing in the report even…
    Speaker notes
    The last discipline, and it decides whether the report is fair. Attributing a district-level cause to facilities is the single most damaging error in DQA writing. It produces defensive reporting, the numbers get smoothed, and next quarter's assessment finds a cleaner dataset that is less true.

  14. Slide 14 / 16

    Put the cause at the level the fix lives at

    The purpose of naming causes is that the next round is better. A report that assigns blame accurately and changes nothing has failed at the only thing it was for.

  15. Slide 15 / 16

    What comes next

    • You have findings, and now each has a cause with a level and a test behind it.
    Speaker notes
    You have findings, and now each has a cause with a level and a test behind it. The last lesson is the document — what goes in it, in what order, and the four columns that turn a finding into something that is still true when somebody checks in three months.

  16. Slide 16 / 16

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson