{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# JMP service ladders for water, sanitation and hygiene\n",
    "\n",
    "*WASH household survey, 2024*\n",
    "\n",
    "Cassion · data-analysis.cassion.dev\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/thecassion/cassion-learning-platform/blob/main/apps/data-analysis/public/datasets/examples/wash-household-survey-2024/jmp-ladders.python.en.ipynb)"
   ],
   "id": "cell-000"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What this produces\n",
    "\n",
    "Every household placed on the three JMP service ladders — drinking water,\n",
    "sanitation and hygiene — and coverage by district. These are the definitions the\n",
    "SDG 6 indicators are reported against, so getting them right is the difference\n",
    "between a figure a cluster will accept and one it will send back.\n",
    "\n",
    "Reference figures from the dataset's quality notes: about 13% of households below\n",
    "the Sphere minimum of 15 litres per person per day, about 39% over a 30-minute\n",
    "round trip, open defecation about 13%, basic hygiene service about 34%.\n",
    "\n",
    "Every dataset on this platform is synthetic. No real household is described.\n",
    "\n",
    "## Setup"
   ],
   "id": "cell-001"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "URL = (\n",
    "    \"https://data-analysis.cassion.dev/datasets/files/\"\n",
    "    \"wash-household-survey-2024.v1.csv\"\n",
    ")\n",
    "\n",
    "wash = pd.read_csv(URL, dtype={\"household_id\": \"string\", \"community\": \"string\"})\n",
    "print(wash.shape)\n",
    "wash.head()"
   ],
   "id": "cell-002"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalise the district before grouping by it\n",
    "\n",
    "One enumerator team wrote the Nord-Ouest district name four different ways. Group\n",
    "without normalising and you get six districts instead of three, splitting the\n",
    "worst-performing one into four pieces small enough to look unremarkable."
   ],
   "id": "cell-003"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(wash[\"district\"].value_counts())"
   ],
   "id": "cell-004"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wash[\"district\"] = (\n",
    "    wash[\"district\"].str.strip().str.lower().str.replace(\" \", \"-\", regex=False)\n",
    ")\n",
    "print(wash[\"district\"].value_counts())"
   ],
   "id": "cell-005"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalise before you group, every time. This is the single most common way a\n",
    "district table quietly hides its worst result.\n",
    "\n",
    "## The drinking water ladder\n",
    "\n",
    "The ladder is **not a property of the source alone.** An improved source more\n",
    "than 30 minutes round trip is *limited* service, not basic — and about a quarter\n",
    "of these households sit on limited service for that reason and no other. An\n",
    "analysis that classifies on source type only misses every one of them."
   ],
   "id": "cell-006"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMPROVED = {\n",
    "    \"piped-into-dwelling\", \"piped-into-yard\", \"public-tap\", \"borehole\",\n",
    "    \"protected-well\", \"protected-spring\", \"tanker-truck\",\n",
    "}\n",
    "UNIMPROVED = {\"unprotected-well\", \"unprotected-spring\"}\n",
    "\n",
    "def water_ladder(row):\n",
    "    source = row[\"water_source\"]\n",
    "    if source == \"surface-water\":\n",
    "        return \"surface water\"\n",
    "    if source in UNIMPROVED:\n",
    "        return \"unimproved\"\n",
    "    if pd.isna(row[\"round_trip_minutes\"]):\n",
    "        return \"improved, time unknown\"\n",
    "    return \"basic\" if row[\"round_trip_minutes\"] <= 30 else \"limited\"\n",
    "\n",
    "wash[\"water_service\"] = wash.apply(water_ladder, axis=1)\n",
    "(wash[\"water_service\"].value_counts(normalize=True) * 100).round(1)"
   ],
   "id": "cell-007"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`safely managed` is deliberately absent. It requires the source to be on\n",
    "premises, available when needed **and** free from contamination, and this survey\n",
    "tests quality on only a third of households — so the top rung cannot be assigned\n",
    "for most of the sample. Reporting \"basic\" and stopping there is honest;\n",
    "inventing a safely-managed figure is not.\n",
    "\n",
    "## The sanitation ladder"
   ],
   "id": "cell-008"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMPROVED_SANITATION = {\n",
    "    \"flush-to-sewer\", \"flush-to-septic\", \"vip-latrine\", \"pit-latrine-with-slab\",\n",
    "}\n",
    "\n",
    "def sanitation_ladder(row):\n",
    "    facility = row[\"sanitation_facility\"]\n",
    "    if facility == \"open-defecation\":\n",
    "        return \"open defecation\"\n",
    "    if facility not in IMPROVED_SANITATION:\n",
    "        return \"unimproved\"\n",
    "    return \"limited\" if row[\"shared_sanitation\"] else \"basic\"\n",
    "\n",
    "wash[\"sanitation_service\"] = wash.apply(sanitation_ladder, axis=1)\n",
    "(wash[\"sanitation_service\"].value_counts(normalize=True) * 100).round(1)"
   ],
   "id": "cell-009"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sharing is what separates basic from limited. A household with a perfectly good\n",
    "VIP latrine shared with three others is on *limited* service, and a table built\n",
    "on facility type alone will report it as basic.\n",
    "\n",
    "## The hygiene ladder"
   ],
   "id": "cell-010"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hygiene_ladder(row):\n",
    "    if row[\"handwashing_facility\"] == \"no-facility\":\n",
    "        return \"no facility\"\n",
    "    return \"basic\" if row[\"soap_observed\"] else \"limited\"\n",
    "\n",
    "wash[\"hygiene_service\"] = wash.apply(hygiene_ladder, axis=1)\n",
    "(wash[\"hygiene_service\"].value_counts(normalize=True) * 100).round(1)"
   ],
   "id": "cell-011"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic hygiene requires a facility **with soap and water present**, observed\n",
    "rather than reported. The distinction matters: a third of these households have\n",
    "a facility and no soap, and asking \"do you wash your hands\" would have counted\n",
    "every one of them as compliant.\n",
    "\n",
    "## Coverage by district"
   ],
   "id": "cell-012"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def coverage(df, column, level):\n",
    "    return (\n",
    "        df.groupby(\"district\")[column]\n",
    "        .apply(lambda s: (s == level).mean() * 100)\n",
    "        .round(1)\n",
    "    )\n",
    "\n",
    "table = pd.DataFrame({\n",
    "    \"basic water\": coverage(wash, \"water_service\", \"basic\"),\n",
    "    \"basic sanitation\": coverage(wash, \"sanitation_service\", \"basic\"),\n",
    "    \"basic hygiene\": coverage(wash, \"hygiene_service\", \"basic\"),\n",
    "    \"open defecation\": coverage(wash, \"sanitation_service\", \"open defecation\"),\n",
    "    \"households\": wash.groupby(\"district\").size(),\n",
    "})\n",
    "table.sort_values(\"basic water\")"
   ],
   "id": "cell-013"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Sphere quantity standard\n",
    "\n",
    "The ladder says nothing about quantity. Sphere sets a minimum of 15 litres per\n",
    "person per day, and it is a separate question from whether the source is\n",
    "improved."
   ],
   "id": "cell-014"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SPHERE_MINIMUM = 15\n",
    "\n",
    "below = wash[\"litres_per_person_day\"] < SPHERE_MINIMUM\n",
    "print(f\"below {SPHERE_MINIMUM} l/p/d: {below.mean():.1%}\")\n",
    "print(f\"over a 30-minute round trip: {(wash['round_trip_minutes'] > 30).mean():.1%}\")\n",
    "\n",
    "pd.crosstab(\n",
    "    wash[\"water_service\"],\n",
    "    below.map({True: \"below Sphere\", False: \"at or above\"}),\n",
    "    normalize=\"index\",\n",
    ").round(3)"
   ],
   "id": "cell-015"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Households on *basic* service still fall below the Sphere minimum. Access and\n",
    "quantity are different indicators and neither substitutes for the other.\n",
    "\n",
    "## The unit errors nobody notices\n",
    "\n",
    "Eleven records hold collection time in hours rather than minutes, and fourteen\n",
    "hold litres for the whole household rather than per person. Both look entirely\n",
    "plausible in isolation — a round trip of 2, or 40 litres a day — and only stand\n",
    "out against household size."
   ],
   "id": "cell-016"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "suspect_litres = (\n",
    "    wash[\"litres_per_person_day\"] > 60\n",
    ") & wash[\"household_size\"].notna()\n",
    "\n",
    "wash.loc[suspect_litres, [\n",
    "    \"household_id\", \"household_size\", \"litres_per_person_day\",\n",
    "]].assign(\n",
    "    implied_household_total=lambda d: d[\"litres_per_person_day\"] * d[\"household_size\"],\n",
    "    as_if_household_total=lambda d: d[\"litres_per_person_day\"] / d[\"household_size\"],\n",
    ").head(10)"
   ],
   "id": "cell-017"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the last column: divided by household size, these become ordinary values.\n",
    "That is the signature of a per-household figure entered in a per-person column.\n",
    "Flag them; do not silently rescale, because you cannot prove which reading the\n",
    "enumerator meant.\n",
    "\n",
    "## What to report\n",
    "\n",
    "State the ladder rung, the denominator it rests on, and the quantity indicator\n",
    "separately. And say which households could not be classified — the ones with no\n",
    "collection time recorded are not \"basic\", they are unknown, and rolling them into\n",
    "the basic count is how a coverage figure drifts upward without anyone deciding\n",
    "that it should."
   ],
   "id": "cell-018"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "file_extension": ".py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}