{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Targeting the cash caseload — analysis notebook\n",
    "\n",
    "*Food security classification and assistance targeting · project deliverable*\n",
    "\n",
    "Cassion · data-analysis.cassion.dev\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/thecassion/cassion-learning-platform/blob/main/apps/data-analysis/public/projects/food-security-classification/notebooks/composite-indicators.python.en.ipynb)"
   ],
   "id": "cell-000"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The decision this produces\n",
    "\n",
    "Which households enter the next cash distribution caseload, and on what stated\n",
    "criteria. The caseload is fixed at roughly a fifth of surveyed households — the\n",
    "budget decided that, not the data — so this is a ranking problem with a hard cut,\n",
    "and the honest output is the rule plus who it excludes.\n",
    "\n",
    "Audience: the food security cluster and the cash working group.\n",
    "\n",
    "Every dataset on this platform is synthetic. No real household is described.\n",
    "\n",
    "## Setup"
   ],
   "id": "cell-001"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "URL = (\n",
    "    \"https://data-analysis.cassion.dev/datasets/files/\"\n",
    "    \"food-security-survey-2024.v1.csv\"\n",
    ")\n",
    "\n",
    "fs = pd.read_csv(URL, dtype={\"household_id\": \"string\"})\n",
    "CASELOAD_SHARE = 0.20\n",
    "caseload = int(round(CASELOAD_SHARE * len(fs)))\n",
    "\n",
    "print(f\"households surveyed: {len(fs)}\")\n",
    "print(f\"caseload the budget allows: {caseload}\")"
   ],
   "id": "cell-002"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring, with the exclusions the indicators require\n",
    "\n",
    "Applied here; the reasoning is in `docs/threshold-justification.md`. Two rules matter for\n",
    "targeting: a consumption value above seven days is impossible and must not\n",
    "inflate a score, and a partial Household Hunger Scale must be excluded rather\n",
    "than zero-filled — zero-filling would score a hungry household as food secure\n",
    "and drop it out of the caseload."
   ],
   "id": "cell-003"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FCS_W = {\n",
    "    \"fcs_cereals_tubers\": 2, \"fcs_pulses\": 3, \"fcs_vegetables\": 1, \"fcs_fruit\": 1,\n",
    "    \"fcs_meat_fish_eggs\": 4, \"fcs_dairy\": 4, \"fcs_oils_fats\": 0.5, \"fcs_sugar\": 0.5,\n",
    "}\n",
    "RCSI_W = {\n",
    "    \"rcsi_less_preferred_food\": 1, \"rcsi_borrowed_food\": 2,\n",
    "    \"rcsi_limit_portion_size\": 1, \"rcsi_restrict_adult_consumption\": 3,\n",
    "    \"rcsi_reduce_meal_numbers\": 1,\n",
    "}\n",
    "HHS = [\"hhs_no_food_in_house\", \"hhs_sleep_hungry\", \"hhs_day_and_night_without_eating\"]\n",
    "\n",
    "for column in FCS_W:\n",
    "    fs[column] = fs[column].where(fs[column].between(0, 7))\n",
    "\n",
    "fs[\"fcs\"] = sum(fs[c] * w for c, w in FCS_W.items())\n",
    "fs[\"rcsi\"] = sum(fs[c] * w for c, w in RCSI_W.items())\n",
    "fs[\"hhs\"] = fs[HHS].sum(axis=1).where(fs[HHS].notna().all(axis=1))\n",
    "\n",
    "fs[\"sex_head\"] = (\n",
    "    fs[\"sex_head_of_household\"].str.strip().str.lower().str[0]\n",
    "    .map({\"f\": \"female\", \"m\": \"male\"})\n",
    ")\n",
    "\n",
    "pd.Series({\n",
    "    \"incomplete FCS (excluded from FCS rules)\": int(fs[\"fcs\"].isna().sum()),\n",
    "    \"partial HHS (excluded from HHS rules)\": int(fs[\"hhs\"].isna().sum()),\n",
    "    \"duplicate enumerations\": int(fs.drop(columns=[\"household_id\"]).duplicated().sum()),\n",
    "})"
   ],
   "id": "cell-004"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why no single indicator gives the caseload\n",
    "\n",
    "The obvious approach is one threshold. It does not work, and seeing why is the\n",
    "argument for the composite."
   ],
   "id": "cell-005"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rules = {\n",
    "    \"FCS poor (<= 21)\": fs[\"fcs\"] <= 21,\n",
    "    \"FCS poor or borderline (<= 35)\": fs[\"fcs\"] <= 35,\n",
    "    \"HHS moderate or worse (>= 2)\": fs[\"hhs\"] >= 2,\n",
    "    \"HHS severe (>= 4)\": fs[\"hhs\"] >= 4,\n",
    "    \"rCSI at or above median\": fs[\"rcsi\"] >= fs[\"rcsi\"].median(),\n",
    "}\n",
    "\n",
    "pd.DataFrame({\n",
    "    \"households\": {k: int(v.sum()) for k, v in rules.items()},\n",
    "    \"share\": {k: round(v.mean(), 3) for k, v in rules.items()},\n",
    "    \"vs caseload\": {k: int(v.sum()) - caseload for k, v in rules.items()},\n",
    "})"
   ],
   "id": "cell-006"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`FCS <= 21` selects far too few to spend the budget; `HHS >= 2` selects twice the\n",
    "caseload. `FCS <= 35` lands close by accident, and choosing it *because* it lands\n",
    "close is fitting the criterion to the budget rather than to need — which is the\n",
    "thing a cluster review will ask about.\n",
    "\n",
    "## The composite, and what it assumes\n",
    "\n",
    "Three indicators, standardised so none dominates by scale, averaged. FCS enters\n",
    "negatively because a **higher** score is better food consumption."
   ],
   "id": "cell-007"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def standardise(series):\n",
    "    return (series - series.mean()) / series.std()\n",
    "\n",
    "fs[\"need_score\"] = (\n",
    "    -standardise(fs[\"fcs\"].fillna(fs[\"fcs\"].median()))\n",
    "    + standardise(fs[\"rcsi\"])\n",
    "    + standardise(fs[\"hhs\"].fillna(0))\n",
    ") / 3\n",
    "\n",
    "fs[\"need_score\"].describe().round(3)"
   ],
   "id": "cell-008"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two choices are visible in that cell and both should be in the report. Equal\n",
    "weights say the three indicators matter equally, which is an assumption rather\n",
    "than a finding. And the fills — median FCS for an incomplete score, zero HHS for\n",
    "a partial one — are conservative in opposite directions, so a household missing\n",
    "data is neither pushed into nor out of the caseload by the gap itself."
   ],
   "id": "cell-009"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selected = fs.nlargest(caseload, \"need_score\").copy()\n",
    "selected[\"selected\"] = True\n",
    "fs[\"selected\"] = fs[\"household_id\"].isin(selected[\"household_id\"])\n",
    "\n",
    "pd.Series({\n",
    "    \"caseload\": int(fs[\"selected\"].sum()),\n",
    "    \"of which FCS poor\": int((selected[\"fcs\"] <= 21).sum()),\n",
    "    \"of which FCS poor or borderline\": int((selected[\"fcs\"] <= 35).sum()),\n",
    "    \"of which HHS severe\": int((selected[\"hhs\"] >= 4).sum()),\n",
    "})"
   ],
   "id": "cell-010"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Who the rule includes, and who it leaves out\n",
    "\n",
    "The part a cash working group will actually interrogate."
   ],
   "id": "cell-011"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def composition(column):\n",
    "    overall = fs[column].value_counts(normalize=True)\n",
    "    chosen = selected[column].value_counts(normalize=True)\n",
    "    out = pd.DataFrame({\"population\": overall, \"caseload\": chosen}).fillna(0)\n",
    "    out[\"difference\"] = (out[\"caseload\"] - out[\"population\"]).round(3)\n",
    "    return out.round(3).sort_values(\"difference\", ascending=False)\n",
    "\n",
    "composition(\"displacement_status\")"
   ],
   "id": "cell-012"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Displaced households are over-represented in the caseload relative to the\n",
    "surveyed population. That is what the indicators say, not a targeting choice —\n",
    "but it is the sentence that has to appear in the report, because a district\n",
    "authority will read the caseload as a statement about who deserves assistance."
   ],
   "id": "cell-013"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "composition(\"main_livelihood\")"
   ],
   "id": "cell-014"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "composition(\"sex_head\")"
   ],
   "id": "cell-015"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The households just outside the line\n",
    "\n",
    "A fixed caseload means some households score barely below the cut. Naming them is\n",
    "more useful than pretending the boundary is meaningful."
   ],
   "id": "cell-016"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordered = fs.sort_values(\"need_score\", ascending=False).reset_index(drop=True)\n",
    "boundary = ordered.iloc[caseload - 20 : caseload + 20]\n",
    "\n",
    "print(f\"need score at the cut: {ordered.loc[caseload - 1, 'need_score']:.3f}\")\n",
    "print(f\"score range across the 40 households either side: \"\n",
    "      f\"{boundary['need_score'].min():.3f} to {boundary['need_score'].max():.3f}\")\n",
    "\n",
    "boundary[[\"household_id\", \"district\", \"displacement_status\", \"fcs\", \"rcsi\", \"hhs\",\n",
    "          \"need_score\", \"selected\"]].head(12).round(3)"
   ],
   "id": "cell-017"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The forty households either side of the cut span under a tenth of a point, and\n",
    "adjacent pairs differ by thousandths. **Nothing in the data distinguishes the\n",
    "last household included from the first excluded**, and the\n",
    "report should say so rather than implying the cut is a finding. If the programme\n",
    "can extend the caseload at the margin, this is the list to extend it with.\n",
    "\n",
    "## The output the cash working group receives"
   ],
   "id": "cell-018"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "caseload_list = (\n",
    "    selected.sort_values(\"need_score\", ascending=False)\n",
    "    [[\"household_id\", \"district\", \"household_size\", \"displacement_status\",\n",
    "      \"main_livelihood\", \"fcs\", \"rcsi\", \"hhs\", \"need_score\"]]\n",
    "    .round(3)\n",
    ")\n",
    "caseload_list.head(15)"
   ],
   "id": "cell-019"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "by_district = fs.groupby(\"district\").agg(\n",
    "    surveyed=(\"household_id\", \"size\"),\n",
    "    selected=(\"selected\", \"sum\"),\n",
    ")\n",
    "by_district[\"share_selected\"] = (\n",
    "    by_district[\"selected\"] / by_district[\"surveyed\"]\n",
    ").round(3)\n",
    "by_district"
   ],
   "id": "cell-020"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What this does not establish\n",
    "\n",
    "The survey is of households reached by the enumeration, not of the district\n",
    "population, so the caseload is drawn from who was surveyed. Households the\n",
    "survey did not reach — displaced people outside settlements, households absent\n",
    "on survey day — are not in this ranking and cannot be targeted from it.\n",
    "\n",
    "And the caseload size is a budget decision. This notebook ranks need; it does not\n",
    "say that a fifth of households is the right number to assist, and the report\n",
    "should not let the two be confused."
   ],
   "id": "cell-021"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "file_extension": ".py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}