{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computing FCS, HHS and rCSI from raw components\n",
    "\n",
    "*Food security survey, 2024*\n",
    "\n",
    "Cassion · data-analysis.cassion.dev\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/thecassion/cassion-learning-platform/blob/main/apps/data-analysis/public/datasets/examples/food-security-survey-2024/composite-indicators.python.en.ipynb)"
   ],
   "id": "cell-000"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What this produces\n",
    "\n",
    "The three composite food security indicators, built from their components rather\n",
    "than read from a precomputed column — because the errors all happen in the\n",
    "building.\n",
    "\n",
    "Reference figures from the dataset's quality notes: on the standard 21/35 FCS\n",
    "thresholds about 1% of households are poor and 23% borderline; on the 28/42 set,\n",
    "7% and 39%. Neither is wrong. Failing to state which you used is.\n",
    "\n",
    "Every dataset on this platform is synthetic. No real household is described.\n",
    "\n",
    "## Setup"
   ],
   "id": "cell-001"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "URL = (\n",
    "    \"https://data-analysis.cassion.dev/datasets/files/\"\n",
    "    \"food-security-survey-2024.v1.csv\"\n",
    ")\n",
    "\n",
    "fs = pd.read_csv(URL, dtype={\"household_id\": \"string\"})\n",
    "print(fs.shape)\n",
    "fs.head()"
   ],
   "id": "cell-002"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Range-check before you score\n",
    "\n",
    "Twenty-three records hold a consumption value above seven days, which is\n",
    "impossible against a seven-day recall. A score computed without range-checking\n",
    "inherits the impossible value and inflates that household — silently, because\n",
    "the result is still a plausible number."
   ],
   "id": "cell-003"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FCS_WEIGHTS = {\n",
    "    \"fcs_cereals_tubers\": 2,\n",
    "    \"fcs_pulses\": 3,\n",
    "    \"fcs_vegetables\": 1,\n",
    "    \"fcs_fruit\": 1,\n",
    "    \"fcs_meat_fish_eggs\": 4,\n",
    "    \"fcs_dairy\": 4,\n",
    "    \"fcs_oils_fats\": 0.5,\n",
    "    \"fcs_sugar\": 0.5,\n",
    "}\n",
    "\n",
    "components = list(FCS_WEIGHTS)\n",
    "impossible = (fs[components] > 7).sum().sum()\n",
    "blank = fs[components].isna().sum().sum()\n",
    "\n",
    "print(f\"values above 7 days : {int(impossible)}\")\n",
    "print(f\"blank cells         : {int(blank)}\")\n",
    "fs[components].agg([\"min\", \"max\"]).T"
   ],
   "id": "cell-004"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Out of range is not a measurement. Set it to missing rather than clipping to\n",
    "# 7 — clipping invents a value the enumerator never recorded.\n",
    "for column in components:\n",
    "    fs[column] = fs[column].where(fs[column].between(0, 7))"
   ],
   "id": "cell-005"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The blank that is not a zero\n",
    "\n",
    "About 127 cells across dairy, fruit and meat are blank. **Treating a blank as\n",
    "zero days scores the household as eating less than it did**, and the components\n",
    "most often blank carry the heaviest weights — dairy and meat are 4 each. This is\n",
    "the most consequential silent failure in this dataset, and it is one line of code\n",
    "either way."
   ],
   "id": "cell-006"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fs[\"fcs_complete\"] = fs[components].notna().all(axis=1)\n",
    "\n",
    "# NaN propagates, so an incomplete household gets no score at all — which is the\n",
    "# correct outcome. The zero-filled version is computed only to measure the harm.\n",
    "fs[\"fcs\"] = sum(fs[c] * w for c, w in FCS_WEIGHTS.items())\n",
    "fs[\"fcs_zero_filled\"] = sum(fs[c].fillna(0) * w for c, w in FCS_WEIGHTS.items())\n",
    "\n",
    "incomplete = ~fs[\"fcs_complete\"]\n",
    "print(f\"households with an incomplete FCS: {int(incomplete.sum())}\")\n",
    "\n",
    "pd.DataFrame({\n",
    "    \"complete households (real score)\": fs.loc[~incomplete, \"fcs\"].describe(),\n",
    "    \"incomplete households, zero-filled\": fs.loc[incomplete, \"fcs_zero_filled\"].describe(),\n",
    "}).round(1)"
   ],
   "id": "cell-007"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The zero-filled households average about six points below the households that\n",
    "answered fully. That gap is not a finding about their diet — it is the blanks\n",
    "being counted as days of not eating."
   ],
   "id": "cell-008"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def share(scores, poor, borderline):\n",
    "    s = scores.dropna()\n",
    "    return {\n",
    "        \"poor %\": round((s <= poor).mean() * 100, 2),\n",
    "        \"borderline %\": round(((s > poor) & (s <= borderline)).mean() * 100, 2),\n",
    "        \"households\": len(s),\n",
    "    }\n",
    "\n",
    "pd.DataFrame({\n",
    "    \"exclude incomplete (correct)\": share(fs[\"fcs\"], 21, 35),\n",
    "    \"zero-fill and keep\": share(fs[\"fcs_zero_filled\"], 21, 35),\n",
    "}).T"
   ],
   "id": "cell-009"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the 21/35 thresholds the distortion is small. At 28/42 it is not:"
   ],
   "id": "cell-010"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "zero_filled_incomplete = fs.loc[incomplete, \"fcs_zero_filled\"]\n",
    "print(f\"of the {len(zero_filled_incomplete)} incomplete households, zero-filling classifies\")\n",
    "print(f\"  {int((zero_filled_incomplete <= 21).sum())} as poor at 21/35\")\n",
    "print(f\"  {int((zero_filled_incomplete <= 28).sum())} as poor at 28/42\")\n",
    "\n",
    "pd.DataFrame({\n",
    "    \"exclude incomplete (correct)\": share(fs[\"fcs\"], 28, 42),\n",
    "    \"zero-fill and keep\": share(fs[\"fcs_zero_filled\"], 28, 42),\n",
    "}).T"
   ],
   "id": "cell-011"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thirty-two households classified as having poor food consumption on scores that\n",
    "are artificially low, because a blank was read as a zero. Every one of them would\n",
    "be counted in a caseload.\n",
    "\n",
    "## Both threshold sets, side by side"
   ],
   "id": "cell-012"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def consumption_group(score, poor, borderline):\n",
    "    if pd.isna(score):\n",
    "        return None\n",
    "    if score <= poor:\n",
    "        return \"poor\"\n",
    "    if score <= borderline:\n",
    "        return \"borderline\"\n",
    "    return \"acceptable\"\n",
    "\n",
    "valid = fs[fs[\"fcs_complete\"]].copy()\n",
    "\n",
    "for label, (poor, borderline) in {\n",
    "    \"21/35\": (21, 35),\n",
    "    \"28/42\": (28, 42),\n",
    "}.items():\n",
    "    valid[f\"group {label}\"] = valid[\"fcs\"].apply(\n",
    "        lambda s: consumption_group(s, poor, borderline)\n",
    "    )\n",
    "\n",
    "pd.DataFrame({\n",
    "    label: valid[f\"group {label}\"].value_counts(normalize=True)\n",
    "    for label in [\"21/35\", \"28/42\"]\n",
    "}).round(3)"
   ],
   "id": "cell-013"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 28/42 set is used where oil and sugar are consumed near-universally, which\n",
    "inflates every score and makes the standard cut-offs too generous. Choosing it is\n",
    "a judgement about the food system, not about the data — and **the choice moves\n",
    "the headline from 1% poor to 7%**. State which set you used, in the same sentence\n",
    "as the number.\n",
    "\n",
    "## The Household Hunger Scale\n",
    "\n",
    "HHS is valid only when all three questions are answered. A partial response must\n",
    "be **excluded, not zero-filled** — zero-filling scores a hungry household as food\n",
    "secure."
   ],
   "id": "cell-014"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HHS_ITEMS = [\n",
    "    \"hhs_no_food_in_house\",\n",
    "    \"hhs_sleep_hungry\",\n",
    "    \"hhs_day_and_night_without_eating\",\n",
    "]\n",
    "\n",
    "complete_hhs = fs[HHS_ITEMS].notna().all(axis=1)\n",
    "partial_hhs = fs[HHS_ITEMS].isna().any(axis=1) & fs[HHS_ITEMS].notna().any(axis=1)\n",
    "\n",
    "print(f\"complete: {int(complete_hhs.sum())}   partial: {int(partial_hhs.sum())}\")\n",
    "\n",
    "fs[\"hhs\"] = fs[HHS_ITEMS].sum(axis=1).where(complete_hhs)\n",
    "fs[\"hhs_category\"] = pd.cut(\n",
    "    fs[\"hhs\"], [-1, 1, 3, 6],\n",
    "    labels=[\"little to none\", \"moderate\", \"severe\"],\n",
    ")\n",
    "\n",
    "(fs[\"hhs_category\"].value_counts(normalize=True).sort_index() * 100).round(1)"
   ],
   "id": "cell-015"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note `.sum(axis=1)` would happily return a score for a partial response — pandas\n",
    "treats missing as zero in a row-wise sum. The `.where(complete_hhs)` is what\n",
    "makes the exclusion real, and leaving it out is exactly the failure the note\n",
    "warns about.\n",
    "\n",
    "## The reduced Coping Strategies Index"
   ],
   "id": "cell-016"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "RCSI_WEIGHTS = {\n",
    "    \"rcsi_less_preferred_food\": 1,\n",
    "    \"rcsi_borrowed_food\": 2,\n",
    "    \"rcsi_limit_portion_size\": 1,\n",
    "    \"rcsi_restrict_adult_consumption\": 3,\n",
    "    \"rcsi_reduce_meal_numbers\": 1,\n",
    "}\n",
    "\n",
    "fs[\"rcsi\"] = sum(fs[c] * w for c, w in RCSI_WEIGHTS.items())\n",
    "fs[\"rcsi\"].describe().round(1)"
   ],
   "id": "cell-017"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weights are severity, not frequency: restricting adult consumption so\n",
    "children can eat scores 3, buying less preferred food scores 1. They are fixed by\n",
    "the standard — do not adjust them to fit a context, because a locally-weighted\n",
    "rCSI is not comparable to anything.\n",
    "\n",
    "## Do not use one as a proxy for the other"
   ],
   "id": "cell-018"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "both = fs[fs[\"fcs_complete\"]]\n",
    "correlation = both[[\"fcs\", \"rcsi\"]].corr().iloc[0, 1]\n",
    "print(f\"correlation between FCS and rCSI: {correlation:.3f}\")"
   ],
   "id": "cell-019"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "About -0.45. They measure related but distinct things: a household can eat\n",
    "monotonously without yet resorting to coping strategies, and another can be\n",
    "coping heavily while still eating a varied diet on borrowed food. Reporting one\n",
    "as a stand-in for the other loses real information, and it is the kind of\n",
    "shortcut that survives until someone asks why the two tables disagree."
   ],
   "id": "cell-020"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(\n",
    "    both[\"fcs\"].apply(lambda s: consumption_group(s, 21, 35)),\n",
    "    pd.cut(both[\"rcsi\"], [-1, 3, 18, 100], labels=[\"low\", \"medium\", \"high\"]),\n",
    "    normalize=\"index\",\n",
    ").round(3)"
   ],
   "id": "cell-021"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What this does and does not produce\n",
    "\n",
    "These are the food consumption evidence used **in** an IPC analysis. They are not\n",
    "an IPC phase. A phase is assigned by a technical working group convening several\n",
    "outcome indicators against contributing factors, and a table that prints \"Phase\n",
    "3\" out of an FCS distribution has skipped the entire analytical process the\n",
    "classification exists to represent."
   ],
   "id": "cell-022"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "file_extension": ".py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}