{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Screening coverage over the campaign year\n",
    "\n",
    "*MUAC screening — Artibonite, 2024*\n",
    "\n",
    "Cassion · data-analysis.cassion.dev\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/thecassion/cassion-learning-platform/blob/main/apps/data-analysis/public/datasets/examples/muac-screening-artibonite-2024/coverage-trend.python.en.ipynb)"
   ],
   "id": "cell-000"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What this produces\n",
    "\n",
    "Screenings per month and commune across the 2024 campaign, and the data quality\n",
    "pattern hiding inside the volume. It is the counterpart to the prevalence\n",
    "example: that one asks how malnourished the children were, this one asks whether\n",
    "you screened enough of them, in the right places, consistently enough to believe\n",
    "the answer.\n",
    "\n",
    "Every dataset on this platform is synthetic. Nothing here describes a real\n",
    "child.\n",
    "\n",
    "## A warning about the word \"coverage\"\n",
    "\n",
    "What follows is **not** programme coverage. Coverage is cases reached over cases\n",
    "existing, and this register has no denominator of children in the population —\n",
    "only the children who turned up. Calling screening volume \"coverage\" is one of\n",
    "the most common ways a nutrition report overstates what it knows.\n",
    "\n",
    "What volume *can* tell you is where the campaign was interrupted, which is a\n",
    "real and useful question with an honest answer.\n",
    "\n",
    "## Setup"
   ],
   "id": "cell-001"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "URL = (\n",
    "    \"https://data-analysis.cassion.dev/datasets/files/\"\n",
    "    \"muac-screening-artibonite-2024.v1.csv\"\n",
    ")\n",
    "\n",
    "muac = pd.read_csv(\n",
    "    URL,\n",
    "    dtype={\"child_id\": \"string\", \"commune\": \"string\", \"sex\": \"string\"},\n",
    "    na_values={\"muac_mm\": [\"-99\"]},\n",
    ")\n",
    "\n",
    "muac[\"screening_date\"] = pd.to_datetime(muac[\"screening_date\"], format=\"%Y-%m-%d\")\n",
    "muac[\"month\"] = muac[\"screening_date\"].dt.to_period(\"M\")\n",
    "\n",
    "print(f\"{len(muac):,} screenings, {muac['commune'].nunique()} communes\")\n",
    "print(f\"{muac['month'].min()} to {muac['month'].max()}\")"
   ],
   "id": "cell-002"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Volume over the year"
   ],
   "id": "cell-003"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "monthly = muac.groupby(\"month\").size()\n",
    "monthly"
   ],
   "id": "cell-004"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "January and December are visibly thinner than the months between them. Before\n",
    "reading that as a campaign that started slowly and wound down, check the obvious\n",
    "alternative: a register that covers part of a month at each end. Here the first\n",
    "screening is mid-January and the last is mid-December, so the two low months are\n",
    "an artefact of the reporting window rather than a drop in activity.\n",
    "\n",
    "This is the check to run every time a first or last period looks weak."
   ],
   "id": "cell-005"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"first screening:\", muac[\"screening_date\"].min().date())\n",
    "print(\"last screening :\", muac[\"screening_date\"].max().date())"
   ],
   "id": "cell-006"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Volume by commune"
   ],
   "id": "cell-007"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "by_commune = (\n",
    "    muac.groupby(\"commune\")\n",
    "    .size()\n",
    "    .sort_values(ascending=False)\n",
    "    .rename(\"screenings\")\n",
    "    .to_frame()\n",
    ")\n",
    "by_commune[\"share\"] = (by_commune[\"screenings\"] / len(muac)).round(3)\n",
    "by_commune"
   ],
   "id": "cell-008"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gonaïves and Saint-Marc account for a large share of the register. That is not\n",
    "in itself a finding — they are the larger communes — but it means an unweighted\n",
    "department-wide rate is dominated by them, and a commune with 189 screenings\n",
    "carries an interval wide enough that its rank is close to meaningless.\n",
    "\n",
    "## Where the campaign was interrupted\n",
    "\n",
    "The useful pattern is not the total, it is the month a commune's activity\n",
    "departs from its own norm."
   ],
   "id": "cell-009"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid = (\n",
    "    muac.pivot_table(\n",
    "        index=\"commune\", columns=\"month\", values=\"child_id\", aggfunc=\"count\"\n",
    "    )\n",
    "    .fillna(0)\n",
    "    .astype(int)\n",
    ")\n",
    "\n",
    "# Each commune against its own median month, so a big commune and a small one\n",
    "# are comparable.\n",
    "median = grid.median(axis=1)\n",
    "relative = grid.div(median, axis=0).round(2)\n",
    "relative"
   ],
   "id": "cell-010"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stack first, then filter. Filtering the frame and stacking afterwards leaves\n",
    "# the NaN cells the mask produced, and current pandas keeps them — you get a\n",
    "# result with every commune-month in it and no error to tell you why.\n",
    "flat = relative.stack()\n",
    "quiet = flat[flat < 0.5].sort_values()\n",
    "\n",
    "print(f\"{len(quiet)} commune-months below half that commune's median\")\n",
    "quiet"
   ],
   "id": "cell-011"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost all of them are December and January, which is the reporting-window\n",
    "artefact again rather than an interruption. The one that is not — Gros-Morne in\n",
    "January — is a campaign that started late in that commune, and it is the only\n",
    "entry here worth asking about.\n",
    "\n",
    "## The pattern that is not about volume\n",
    "\n",
    "Screening volume held up in June. What did not hold up was completeness — and a\n",
    "count of rows will never show you that, because the rows are there."
   ],
   "id": "cell-012"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "june = muac[muac[\"month\"] == pd.Period(\"2024-06\")]\n",
    "\n",
    "completeness = (\n",
    "    june.assign(missing_age=june[\"age_months\"].isna())\n",
    "    .groupby(\"commune\")[\"missing_age\"]\n",
    "    .agg(rows=\"size\", missing_age_rate=\"mean\")\n",
    "    .sort_values(\"missing_age_rate\", ascending=False)\n",
    "    .round(3)\n",
    ")\n",
    "completeness"
   ],
   "id": "cell-013"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One commune's June is far worse than the rest. Narrow it to the week and the\n",
    "cause becomes obvious:"
   ],
   "id": "cell-014"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "worst = completeness.index[0]\n",
    "\n",
    "weekly = (\n",
    "    muac[muac[\"commune\"] == worst]\n",
    "    .assign(week=lambda d: d[\"screening_date\"].dt.to_period(\"W\"))\n",
    "    .groupby(\"week\")[\"age_months\"]\n",
    "    .agg(rows=\"size\", missing_age_rate=lambda s: s.isna().mean())\n",
    "    .sort_values(\"missing_age_rate\", ascending=False)\n",
    "    .round(3)\n",
    ")\n",
    "weekly.head()"
   ],
   "id": "cell-015"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One team, one week, one tablet form with the age field misconfigured. The\n",
    "screenings happened and the children were measured; only the age went missing.\n",
    "\n",
    "## Why this matters for the prevalence table\n",
    "\n",
    "If you drop incomplete rows before computing prevalence, you remove that commune\n",
    "far more than any other — and then rank communes partly on whose form was\n",
    "broken."
   ],
   "id": "cell-016"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "GAM_MM = 125\n",
    "\n",
    "unit_error = muac[\"muac_mm\"].notna() & (muac[\"muac_mm\"] < 40)\n",
    "muac.loc[unit_error, \"muac_mm\"] = muac.loc[unit_error, \"muac_mm\"] * 10\n",
    "\n",
    "measured = muac[muac[\"muac_mm\"].notna()]\n",
    "complete = measured[measured[\"age_months\"].notna()]\n",
    "\n",
    "comparison = pd.DataFrame({\n",
    "    \"keeping_missing_age\": measured.groupby(\"commune\")[\"muac_mm\"].apply(\n",
    "        lambda s: (s < GAM_MM).mean()\n",
    "    ),\n",
    "    \"dropping_missing_age\": complete.groupby(\"commune\")[\"muac_mm\"].apply(\n",
    "        lambda s: (s < GAM_MM).mean()\n",
    "    ),\n",
    "})\n",
    "comparison[\"difference\"] = (\n",
    "    comparison[\"dropping_missing_age\"] - comparison[\"keeping_missing_age\"]\n",
    ")\n",
    "comparison.sort_values(\"difference\", key=abs, ascending=False).round(4)"
   ],
   "id": "cell-017"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MUAC thresholds for 6 to 59 months are a single band and do not need age at\n",
    "all, so the right decision here is to keep those rows for the MUAC indicator and\n",
    "exclude them only from analyses that genuinely require age. That decision is\n",
    "available only because you looked.\n",
    "\n",
    "## Plot it"
   ],
   "id": "cell-018"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(9, 5))\n",
    "\n",
    "for commune in grid.index:\n",
    "    ax.plot(\n",
    "        range(len(grid.columns)),\n",
    "        grid.loc[commune],\n",
    "        marker=\"o\",\n",
    "        markersize=3,\n",
    "        linewidth=1,\n",
    "        color=\"#9AA8A3\",\n",
    "        alpha=0.7,\n",
    "    )\n",
    "\n",
    "ax.plot(\n",
    "    range(len(grid.columns)),\n",
    "    grid.loc[worst],\n",
    "    marker=\"o\",\n",
    "    markersize=4,\n",
    "    linewidth=2,\n",
    "    color=\"#2F5D50\",\n",
    "    label=worst,\n",
    ")\n",
    "\n",
    "ax.set_xticks(range(len(grid.columns)))\n",
    "ax.set_xticklabels([str(m) for m in grid.columns], rotation=45, ha=\"right\")\n",
    "ax.set_ylabel(\"Screenings\")\n",
    "ax.set_title(\"Screenings per commune per month, 2024\")\n",
    "ax.legend(frameon=False)\n",
    "ax.spines[[\"top\", \"right\"]].set_visible(False)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "id": "cell-019"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What to report\n",
    "\n",
    "State the volume, state the completeness separately, and never let one stand in\n",
    "for the other. A campaign that screened its target number of children with a\n",
    "broken age field has a volume problem of zero and a data problem that changes\n",
    "the ranking — and only one of those is visible in a count of rows."
   ],
   "id": "cell-020"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "file_extension": ".py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}