{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Food insecurity by displacement status and livelihood\n",
    "\n",
    "*Food security survey, 2024*\n",
    "\n",
    "Cassion · data-analysis.cassion.dev\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/thecassion/cassion-learning-platform/blob/main/apps/data-analysis/public/datasets/examples/food-security-survey-2024/disaggregation.python.en.ipynb)"
   ],
   "id": "cell-000"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What this produces\n",
    "\n",
    "Food consumption groups cut by displacement status, livelihood and sex of\n",
    "household head — and, more usefully, an honest account of **which cuts are large\n",
    "enough to act on**. Disaggregation is where a survey stops being a headline and\n",
    "starts being a targeting decision, and it is also where a small sample quietly\n",
    "runs out.\n",
    "\n",
    "Every dataset on this platform is synthetic. No real household is described.\n",
    "\n",
    "## Setup and scoring\n",
    "\n",
    "The scoring is the previous example condensed: range-check, exclude incomplete,\n",
    "weight."
   ],
   "id": "cell-001"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "URL = (\n",
    "    \"https://data-analysis.cassion.dev/datasets/files/\"\n",
    "    \"food-security-survey-2024.v1.csv\"\n",
    ")\n",
    "\n",
    "fs = pd.read_csv(URL, dtype={\"household_id\": \"string\"})\n",
    "\n",
    "FCS_WEIGHTS = {\n",
    "    \"fcs_cereals_tubers\": 2, \"fcs_pulses\": 3, \"fcs_vegetables\": 1,\n",
    "    \"fcs_fruit\": 1, \"fcs_meat_fish_eggs\": 4, \"fcs_dairy\": 4,\n",
    "    \"fcs_oils_fats\": 0.5, \"fcs_sugar\": 0.5,\n",
    "}\n",
    "components = list(FCS_WEIGHTS)\n",
    "\n",
    "for column in components:\n",
    "    fs[column] = fs[column].where(fs[column].between(0, 7))\n",
    "\n",
    "fs[\"fcs\"] = sum(fs[c] * w for c, w in FCS_WEIGHTS.items())\n",
    "fs[\"consumption_group\"] = pd.cut(\n",
    "    fs[\"fcs\"], [-np.inf, 21, 35, np.inf],\n",
    "    labels=[\"poor\", \"borderline\", \"acceptable\"],\n",
    ")\n",
    "\n",
    "valid = fs[fs[\"fcs\"].notna()].copy()\n",
    "print(f\"{len(valid)} households with a complete FCS, of {len(fs)} surveyed\")"
   ],
   "id": "cell-002"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean the disaggregation variable first\n",
    "\n",
    "One team in Sud recorded the head of household's sex as `Female` and `Male`\n",
    "rather than `f` and `m`. A sex-disaggregated table built without normalising\n",
    "fragments into four categories, two of them small enough to look like noise —\n",
    "and the two small ones come from a single team in a single area, so they are not\n",
    "a random subset of anything."
   ],
   "id": "cell-003"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(fs[\"sex_head_of_household\"].value_counts())"
   ],
   "id": "cell-004"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid[\"sex_head\"] = (\n",
    "    valid[\"sex_head_of_household\"]\n",
    "    .str.strip()\n",
    "    .str.lower()\n",
    "    .str[0]\n",
    "    .map({\"f\": \"female\", \"m\": \"male\"})\n",
    ")\n",
    "print(valid[\"sex_head\"].value_counts(dropna=False))"
   ],
   "id": "cell-005"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking the first letter after lower-casing handles `f`, `F`, `female` and\n",
    "`Female` in one step. Mapping explicitly afterwards means an unexpected value\n",
    "becomes `NaN` and is counted, rather than silently becoming a fifth category.\n",
    "\n",
    "## The cut that matters: displacement status"
   ],
   "id": "cell-006"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def group_shares(df, by):\n",
    "    counts = pd.crosstab(df[by], df[\"consumption_group\"])\n",
    "    shares = (counts.div(counts.sum(axis=1), axis=0) * 100).round(1)\n",
    "    shares[\"households\"] = counts.sum(axis=1)\n",
    "    return shares.sort_values(\"poor\", ascending=False)\n",
    "\n",
    "group_shares(valid, \"displacement_status\")"
   ],
   "id": "cell-007"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The gradient is the finding: displaced and returnee households are worse off than\n",
    "residents and hosts. But look at the household counts before believing the\n",
    "ordering — the smallest group here has fewer than a hundred households, and a\n",
    "percentage from that base moves by a whole point when three households change\n",
    "category.\n",
    "\n",
    "## How much of that ordering is real?\n",
    "\n",
    "Attach an interval before ranking anything. Without one, a difference of four\n",
    "points between two groups reads as a finding when it may be a coin flip."
   ],
   "id": "cell-008"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import beta\n",
    "\n",
    "def wilson(successes, n, confidence=0.95):\n",
    "    if n == 0:\n",
    "        return (np.nan, np.nan)\n",
    "    alpha = 1 - confidence\n",
    "    lower = beta.ppf(alpha / 2, successes, n - successes + 1) if successes > 0 else 0.0\n",
    "    upper = beta.ppf(1 - alpha / 2, successes + 1, n - successes) if successes < n else 1.0\n",
    "    return (lower, upper)\n",
    "\n",
    "def inadequate_rate(df, by):\n",
    "    inadequate = df[\"consumption_group\"].isin([\"poor\", \"borderline\"])\n",
    "    out = df.assign(inadequate=inadequate).groupby(by).agg(\n",
    "        households=(\"inadequate\", \"size\"),\n",
    "        cases=(\"inadequate\", \"sum\"),\n",
    "    )\n",
    "    out[\"rate\"] = out[\"cases\"] / out[\"households\"]\n",
    "    bounds = out.apply(lambda r: wilson(r[\"cases\"], r[\"households\"]), axis=1)\n",
    "    out[\"low\"] = [b[0] for b in bounds]\n",
    "    out[\"high\"] = [b[1] for b in bounds]\n",
    "    return out.sort_values(\"rate\", ascending=False).round(3)\n",
    "\n",
    "inadequate_rate(valid, \"displacement_status\")"
   ],
   "id": "cell-009"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Poor and borderline are combined here because that is the population a food\n",
    "assistance caseload is drawn from, and because splitting a small group across\n",
    "three categories leaves nothing to estimate.\n",
    "\n",
    "Read the intervals against each other. Where they overlap, the ordering between\n",
    "those two groups is not supported — you can say displaced households are worse\n",
    "off than residents; you may not be able to say returnees are worse off than\n",
    "displaced.\n",
    "\n",
    "## Livelihood: where the sample runs out"
   ],
   "id": "cell-010"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inadequate_rate(valid, \"main_livelihood\")"
   ],
   "id": "cell-011"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The bottom of that table is where a survey stops being able to answer the\n",
    "question. A livelihood group with a couple of dozen households produces a rate\n",
    "with an interval spanning twenty points or more — it is not a finding, and\n",
    "putting it in a ranked table invites someone to act on it."
   ],
   "id": "cell-012"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "by_livelihood = inadequate_rate(valid, \"main_livelihood\")\n",
    "by_livelihood[\"interval width\"] = (\n",
    "    by_livelihood[\"high\"] - by_livelihood[\"low\"]\n",
    ").round(3)\n",
    "by_livelihood[[\"households\", \"rate\", \"interval width\"]]"
   ],
   "id": "cell-013"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A practical rule:** decide a minimum cell size before you look at the results,\n",
    "write it in the analysis plan, and report groups below it as a single \"other\"\n",
    "row with their combined rate. Deciding afterwards is how a threshold gets chosen\n",
    "to make a particular group look bad.\n",
    "\n",
    "## Two-way cuts run out faster"
   ],
   "id": "cell-014"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "two_way = pd.crosstab(\n",
    "    valid[\"displacement_status\"], valid[\"sex_head\"],\n",
    "    values=valid[\"consumption_group\"].isin([\"poor\", \"borderline\"]),\n",
    "    aggfunc=\"mean\",\n",
    ").round(3)\n",
    "\n",
    "counts = pd.crosstab(valid[\"displacement_status\"], valid[\"sex_head\"])\n",
    "\n",
    "pd.concat({\"rate\": two_way, \"households\": counts}, axis=1)"
   ],
   "id": "cell-015"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every additional dimension divides the sample again. A three-way cut of this\n",
    "survey — displacement by sex by livelihood — would put single-digit household\n",
    "counts in most cells, and a table of percentages computed on four households is\n",
    "not evidence, however neatly it prints.\n",
    "\n",
    "## What to report\n",
    "\n",
    "The cuts the sample supports, with intervals, and an explicit statement of the\n",
    "cuts it does not. \"We could not estimate food insecurity separately for\n",
    "fishing households, because barely a hundred were surveyed\" is a useful sentence. Its\n",
    "absence is how a reader assumes every row in your table is equally solid."
   ],
   "id": "cell-016"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "file_extension": ".py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}