{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Supervision shortlist from coverage and dropout — analysis notebook\n",
    "\n",
    "*Vaccination coverage and dropout analysis · project deliverable*\n",
    "\n",
    "Cassion · data-analysis.cassion.dev\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/thecassion/cassion-learning-platform/blob/main/apps/data-analysis/public/projects/vaccination-coverage-analysis/notebooks/coverage-analysis.python.en.ipynb)"
   ],
   "id": "cell-000"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The decision this produces\n",
    "\n",
    "Which ten of thirty-eight facilities receive a supervision visit next quarter.\n",
    "Ten is what the supervisor's time allows, so this is a shortlist with a hard\n",
    "limit, and the output is the list plus the reason each facility is on it — a\n",
    "visit with no stated question is a wasted day.\n",
    "\n",
    "The problem the project exists to fix: the previous dashboard ranked facilities\n",
    "by reported coverage without adjusting for whether the facility reported at all,\n",
    "so a silent facility looked like a failing one.\n",
    "\n",
    "Audience: the district EPI supervisor.\n",
    "\n",
    "Every dataset on this platform is synthetic. Coverage here describes no real\n",
    "district.\n",
    "\n",
    "## Setup"
   ],
   "id": "cell-001"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "URL = (\n",
    "    \"https://data-analysis.cassion.dev/datasets/files/\"\n",
    "    \"vaccination-coverage-2024.v1.csv\"\n",
    ")\n",
    "\n",
    "MONTHS = 12\n",
    "VISITS = 10\n",
    "\n",
    "epi = pd.read_csv(URL, dtype={\"facility_id\": \"string\"})\n",
    "epi[\"period\"] = pd.to_datetime(epi[\"period\"])\n",
    "epi[\"month\"] = epi[\"period\"].dt.to_period(\"M\")\n",
    "\n",
    "print(f\"{epi['facility_id'].nunique()} facilities, {epi['month'].nunique()} months\")\n",
    "print(f\"supervision visits available: {VISITS}\")"
   ],
   "id": "cell-002"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reporting completeness first, because it gates everything else\n",
    "\n",
    "A facility that did not report appears as a row with zero doses. Sum without\n",
    "filtering and a silent facility becomes a facility that vaccinated nobody."
   ],
   "id": "cell-003"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "facility_months = epi.drop_duplicates([\"facility_id\", \"month\"])\n",
    "\n",
    "completeness = facility_months.groupby(\"facility_id\")[\"report_submitted\"].mean()\n",
    "by_month = facility_months.groupby(\"month\")[\"report_submitted\"].mean()\n",
    "\n",
    "print(f\"district completeness: {facility_months['report_submitted'].mean():.1%}\")\n",
    "by_month.round(3)"
   ],
   "id": "cell-004"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "August and September are the district's problem months, not a facility's. A\n",
    "supervision list built on those two months alone would visit whoever happened to\n",
    "be quiet during a district-wide disruption.\n",
    "\n",
    "## Three signals, each answering a different question"
   ],
   "id": "cell-005"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reported = epi[epi[\"report_submitted\"]]\n",
    "\n",
    "series = (\n",
    "    reported[reported[\"antigen\"].isin([\"penta1\", \"penta3\"])]\n",
    "    .pivot_table(index=\"facility_id\", columns=\"antigen\",\n",
    "                 values=\"doses_administered\", aggfunc=\"sum\")\n",
    "    .dropna()\n",
    ")\n",
    "series[\"dropout\"] = (series[\"penta1\"] - series[\"penta3\"]) / series[\"penta1\"]\n",
    "series[\"completeness\"] = completeness\n",
    "\n",
    "# Coverage on a denominator restricted to the months the facility reported, so\n",
    "# it measures vaccination rather than reporting.\n",
    "target = reported.groupby(\"facility_id\")[\"target_population\"].first()\n",
    "months_reported = facility_months.groupby(\"facility_id\")[\"report_submitted\"].sum()\n",
    "penta3 = reported[reported[\"antigen\"] == \"penta3\"].groupby(\"facility_id\")[\n",
    "    \"doses_administered\"\n",
    "].sum()\n",
    "series[\"coverage\"] = penta3 / (target * months_reported / MONTHS)\n",
    "\n",
    "series[[\"penta1\", \"penta3\", \"dropout\", \"completeness\", \"coverage\"]].describe().round(3)"
   ],
   "id": "cell-006"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The flags, and the one that does not bind"
   ],
   "id": "cell-007"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DROPOUT_LIMIT = 0.25\n",
    "COMPLETENESS_FLOOR = 0.70\n",
    "\n",
    "series[\"over_reporting\"] = series[\"dropout\"] < 0\n",
    "series[\"high_dropout\"] = series[\"dropout\"] > DROPOUT_LIMIT\n",
    "series[\"low_completeness\"] = series[\"completeness\"] < COMPLETENESS_FLOOR\n",
    "\n",
    "flags = pd.Series({\n",
    "    \"penta3 above penta1 on the annual total\": int(series[\"over_reporting\"].sum()),\n",
    "    f\"dropout above {DROPOUT_LIMIT:.0%}\": int(series[\"high_dropout\"].sum()),\n",
    "    f\"completeness below {COMPLETENESS_FLOOR:.0%}\": int(series[\"low_completeness\"].sum()),\n",
    "})\n",
    "flags"
   ],
   "id": "cell-008"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The dropout flag catches nothing here**, and that is worth reporting rather\n",
    "than quietly dropping the criterion. Median dropout sits near 14% and no facility\n",
    "exceeds 25% on the annual total, so a supervision list built on dropout alone\n",
    "would be empty. Keeping the check in and stating that it fired zero times is more\n",
    "useful to next quarter's supervisor than removing it."
   ],
   "id": "cell-009"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "series[\"flags\"] = (\n",
    "    series[\"over_reporting\"].astype(int)\n",
    "    + series[\"high_dropout\"].astype(int)\n",
    "    + series[\"low_completeness\"].astype(int)\n",
    ")\n",
    "int((series[\"flags\"] > 0).sum())"
   ],
   "id": "cell-010"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fourteen facilities fail at least one check and ten visits are available, so the\n",
    "list has to be prioritised rather than simply filtered.\n",
    "\n",
    "## Prioritising, with the reason attached\n",
    "\n",
    "Over-reporting outranks low completeness: a facility reporting more third doses\n",
    "than first doses is producing numbers that are wrong, while a facility that did\n",
    "not report is producing numbers that are absent. Absent is recoverable by asking;\n",
    "wrong has already entered the district total."
   ],
   "id": "cell-011"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reason(row):\n",
    "    reasons = []\n",
    "    if row[\"over_reporting\"]:\n",
    "        reasons.append(\"penta3 > penta1 on the year\")\n",
    "    if row[\"low_completeness\"]:\n",
    "        reasons.append(f\"reported {row['completeness']:.0%} of months\")\n",
    "    if row[\"high_dropout\"]:\n",
    "        reasons.append(f\"dropout {row['dropout']:.0%}\")\n",
    "    return \"; \".join(reasons)\n",
    "\n",
    "series[\"priority\"] = (\n",
    "    series[\"over_reporting\"].astype(int) * 2 + series[\"low_completeness\"].astype(int)\n",
    ")\n",
    "series[\"reason\"] = series.apply(reason, axis=1)\n",
    "\n",
    "shortlist = (\n",
    "    series[series[\"flags\"] > 0]\n",
    "    .sort_values([\"priority\", \"completeness\"], ascending=[False, True])\n",
    "    .head(VISITS)\n",
    ")\n",
    "\n",
    "shortlist[[\"completeness\", \"dropout\", \"coverage\", \"reason\"]].round(3)"
   ],
   "id": "cell-012"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "not_visited = (\n",
    "    series[series[\"flags\"] > 0]\n",
    "    .drop(index=shortlist.index)\n",
    "    [[\"completeness\", \"dropout\", \"reason\"]]\n",
    "    .round(3)\n",
    ")\n",
    "print(f\"failing a check but not visited this quarter: {len(not_visited)}\")\n",
    "not_visited"
   ],
   "id": "cell-013"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Naming the facilities that failed a check and did not make the list is the point\n",
    "of a capped shortlist. They are next quarter's first candidates, and a supervisor\n",
    "who is not told about them will assume the list was exhaustive.\n",
    "\n",
    "## The ranking that was wrong before"
   ],
   "id": "cell-014"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "naive_coverage = penta3 / target          # every month counted, reported or not\n",
    "comparison = pd.DataFrame({\n",
    "    \"reported-only denominator\": series[\"coverage\"],\n",
    "    \"all months counted\": naive_coverage,\n",
    "    \"completeness\": series[\"completeness\"],\n",
    "})\n",
    "comparison[\"rank_correct\"] = comparison[\"reported-only denominator\"].rank(ascending=True)\n",
    "comparison[\"rank_naive\"] = comparison[\"all months counted\"].rank(ascending=True)\n",
    "comparison[\"rank_change\"] = (\n",
    "    comparison[\"rank_correct\"] - comparison[\"rank_naive\"]\n",
    ").astype(int)\n",
    "\n",
    "comparison.sort_values(\"rank_change\", key=abs, ascending=False).head(8).round(3)"
   ],
   "id": "cell-015"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Facilities move by several places between the two rankings, and they move by how\n",
    "often they reported rather than by how well they vaccinated. That is the defect\n",
    "the previous dashboard shipped, and the reason this notebook exists.\n",
    "\n",
    "## What this does not establish\n",
    "\n",
    "Coverage rests on `target_population`, an administrative estimate this dataset\n",
    "takes as given. Where the estimate is wrong the coverage figure is wrong in the\n",
    "same direction, and a facility with an overstated catchment will look like it is\n",
    "underperforming no matter how many children it reaches.\n",
    "\n",
    "The supervision list is therefore a list of facilities whose **data** deserves a\n",
    "conversation. Whether the underlying service is failing is what the visit is for."
   ],
   "id": "cell-016"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "file_extension": ".py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}