The 212 cases that are not a failure

Slide 1 / 19

What this lesson covers

• The pathway has four gates
• Consent is not a performance measure
• Report all three numbers
• Where consent is not the gate
• The contradictions to resolve first
• What comes next

Slide 2 / 19

The pathway has four gates — In Python

import pandas as pd

referrals = pd.read_csv("protection-referrals-2024.v1.csv")

gates = {
    "cases recorded": len(referrals),
    "consented to referral": referrals["consent_to_refer"].sum(),
    "referral made": (referrals["consent_to_refer"] &
                      referrals["referral_made"]).sum(),
    "referral accepted": (referrals["referral_made"] &
                          referrals["referral_accepted"]).sum(),
    "reached a service": (referrals["referral_accepted"] &
                          referrals["days_to_first_service"].notna()).sum(),
}
for name, count in gates.items():
    print(f"{name:24} {count:>5}")

Slide 3 / 19

The pathway has four gates — In R

library(dplyr)

referrals |> summarise(
  cases = n(),
  consented = sum(consent_to_refer),
  made = sum(consent_to_refer & referral_made),
  accepted = sum(referral_made & referral_accepted),
  reached = sum(referral_accepted & !is.na(days_to_first_service))
)

Slide 4 / 19

The pathway has four gates

Slide 5 / 19

Consent is not a performance measure

• As failures — End-to-end completion becomes 717 / 1,850 = 38.8%
• As exclusions — Completion becomes 717 / 1,638 = 43.8% on cases that consented
• As a finding in their own right — 11.5% declined, and why is a service design question worth asking separately

Slide 6 / 19

Consent is not a performance measure — In Python

consenting = referrals["consent_to_refer"]
reached = referrals["referral_accepted"] & referrals["days_to_first_service"].notna()

print(f"on all cases:        {reached.sum() / len(referrals):.1%}")
print(f"on consenting cases: {reached.sum() / consenting.sum():.1%}")
print(f"declined:            {(~consenting).sum()} ({(~consenting).mean():.1%})")

Slide 7 / 19

Consent is not a performance measure — In R

referrals |> summarise(
  all_cases = mean(referral_accepted & !is.na(days_to_first_service)),
  consenting = sum(referral_accepted & !is.na(days_to_first_service)) / sum(consent_to_refer)
)

Slide 8 / 19

Consent is not a performance measure

• Counting a non-consenting case as a pathway failure does two wrong things at once — It misstates performance by five…
• Both matter and the second matters more — An indicator that treats declining as failure creates pressure on caseworkers…

Slide 9 / 19

Report all three numbers — Example

Referral pathway, 1,850 cases

  Consented to referral            88.5%   1,638
  Declined                         11.5%     212     reported separately
  Reached a service, of consenting 43.8%     717
  Reached a service, of all cases  38.8%             for reference only

Slide 10 / 19

Report all three numbers

• Publish the consent rate as its own line — A falling consent rate is a signal about trust in the service, and it is…

Slide 11 / 19

Where consent is not the gate

• Child protection cases involving a young child — operate under a best-interests determination rather than the child's…
• Life-threatening emergencies — proceed to a life-saving referral without waiting for consent to a data transfer

Slide 12 / 19

Where consent is not the gate — In Python

by_category = referrals.groupby("case_category")["consent_to_refer"].agg(
    ["mean", "size"]
)
print((by_category * [100, 1]).round(1))

Slide 13 / 19

Where consent is not the gate — In R

referrals |> summarise(consent = mean(consent_to_refer), n = n(),
                       .by = case_category)

Slide 14 / 19

Where consent is not the gate

• Check whether the consent rate differs by category before applying one rule to all of them — Where it does, the…

Slide 15 / 19

The contradictions to resolve first — In Python

impossible = (referrals["days_to_first_service"].notna() &
              ~referrals["referral_accepted"])
no_referral = impossible & ~referrals["referral_made"]
no_consent = impossible & ~referrals["consent_to_refer"]

print(f"time to service but referral not accepted: {impossible.sum()}")
print(f"  of which no referral was made at all:    {no_referral.sum()}")
print(f"  of which there was no consent:           {no_consent.sum()}")

Slide 16 / 19

The contradictions to resolve first — In R

referrals |> filter(!is.na(days_to_first_service), !referral_accepted) |>
  count(referral_made, consent_to_refer)

Slide 17 / 19

The contradictions to resolve first

• Eleven cases record a time to first service on a referral that was never accepted — Six of them show no referral made…

Slide 18 / 19

What comes next

• The pathway loses cases at every gate.

Slide 19 / 19

Where this goes next

Read the full lesson, with runnable code Back to the lesson