---
title: "Food insecurity by displacement status and livelihood"
subtitle: "Food security survey, 2024"
author: "Cassion · data-analysis.cassion.dev"
format:
  html:
    toc: true
    code-fold: false
jupyter: python3
---

## What this produces

Food consumption groups cut by displacement status, livelihood and sex of
household head — and, more usefully, an honest account of **which cuts are large
enough to act on**. Disaggregation is where a survey stops being a headline and
starts being a targeting decision, and it is also where a small sample quietly
runs out.

Every dataset on this platform is synthetic. No real household is described.

## Setup and scoring

The scoring is the previous example condensed: range-check, exclude incomplete,
weight.

```{python}
import pandas as pd
import numpy as np

URL = (
    "https://data-analysis.cassion.dev/datasets/files/"
    "food-security-survey-2024.v1.csv"
)

fs = pd.read_csv(URL, dtype={"household_id": "string"})

FCS_WEIGHTS = {
    "fcs_cereals_tubers": 2, "fcs_pulses": 3, "fcs_vegetables": 1,
    "fcs_fruit": 1, "fcs_meat_fish_eggs": 4, "fcs_dairy": 4,
    "fcs_oils_fats": 0.5, "fcs_sugar": 0.5,
}
components = list(FCS_WEIGHTS)

for column in components:
    fs[column] = fs[column].where(fs[column].between(0, 7))

fs["fcs"] = sum(fs[c] * w for c, w in FCS_WEIGHTS.items())
fs["consumption_group"] = pd.cut(
    fs["fcs"], [-np.inf, 21, 35, np.inf],
    labels=["poor", "borderline", "acceptable"],
)

valid = fs[fs["fcs"].notna()].copy()
print(f"{len(valid)} households with a complete FCS, of {len(fs)} surveyed")
```

## Clean the disaggregation variable first

One team in Sud recorded the head of household's sex as `Female` and `Male`
rather than `f` and `m`. A sex-disaggregated table built without normalising
fragments into four categories, two of them small enough to look like noise —
and the two small ones come from a single team in a single area, so they are not
a random subset of anything.

```{python}
print(fs["sex_head_of_household"].value_counts())
```

```{python}
valid["sex_head"] = (
    valid["sex_head_of_household"]
    .str.strip()
    .str.lower()
    .str[0]
    .map({"f": "female", "m": "male"})
)
print(valid["sex_head"].value_counts(dropna=False))
```

Taking the first letter after lower-casing handles `f`, `F`, `female` and
`Female` in one step. Mapping explicitly afterwards means an unexpected value
becomes `NaN` and is counted, rather than silently becoming a fifth category.

## The cut that matters: displacement status

```{python}
def group_shares(df, by):
    counts = pd.crosstab(df[by], df["consumption_group"])
    shares = (counts.div(counts.sum(axis=1), axis=0) * 100).round(1)
    shares["households"] = counts.sum(axis=1)
    return shares.sort_values("poor", ascending=False)

group_shares(valid, "displacement_status")
```

The gradient is the finding: displaced and returnee households are worse off than
residents and hosts. But look at the household counts before believing the
ordering — the smallest group here has fewer than a hundred households, and a
percentage from that base moves by a whole point when three households change
category.

## How much of that ordering is real?

Attach an interval before ranking anything. Without one, a difference of four
points between two groups reads as a finding when it may be a coin flip.

```{python}
from scipy.stats import beta

def wilson(successes, n, confidence=0.95):
    if n == 0:
        return (np.nan, np.nan)
    alpha = 1 - confidence
    lower = beta.ppf(alpha / 2, successes, n - successes + 1) if successes > 0 else 0.0
    upper = beta.ppf(1 - alpha / 2, successes + 1, n - successes) if successes < n else 1.0
    return (lower, upper)

def inadequate_rate(df, by):
    inadequate = df["consumption_group"].isin(["poor", "borderline"])
    out = df.assign(inadequate=inadequate).groupby(by).agg(
        households=("inadequate", "size"),
        cases=("inadequate", "sum"),
    )
    out["rate"] = out["cases"] / out["households"]
    bounds = out.apply(lambda r: wilson(r["cases"], r["households"]), axis=1)
    out["low"] = [b[0] for b in bounds]
    out["high"] = [b[1] for b in bounds]
    return out.sort_values("rate", ascending=False).round(3)

inadequate_rate(valid, "displacement_status")
```

Poor and borderline are combined here because that is the population a food
assistance caseload is drawn from, and because splitting a small group across
three categories leaves nothing to estimate.

Read the intervals against each other. Where they overlap, the ordering between
those two groups is not supported — you can say displaced households are worse
off than residents; you may not be able to say returnees are worse off than
displaced.

## Livelihood: where the sample runs out

```{python}
inadequate_rate(valid, "main_livelihood")
```

The bottom of that table is where a survey stops being able to answer the
question. A livelihood group with a couple of dozen households produces a rate
with an interval spanning twenty points or more — it is not a finding, and
putting it in a ranked table invites someone to act on it.

```{python}
by_livelihood = inadequate_rate(valid, "main_livelihood")
by_livelihood["interval width"] = (
    by_livelihood["high"] - by_livelihood["low"]
).round(3)
by_livelihood[["households", "rate", "interval width"]]
```

**A practical rule:** decide a minimum cell size before you look at the results,
write it in the analysis plan, and report groups below it as a single "other"
row with their combined rate. Deciding afterwards is how a threshold gets chosen
to make a particular group look bad.

## Two-way cuts run out faster

```{python}
two_way = pd.crosstab(
    valid["displacement_status"], valid["sex_head"],
    values=valid["consumption_group"].isin(["poor", "borderline"]),
    aggfunc="mean",
).round(3)

counts = pd.crosstab(valid["displacement_status"], valid["sex_head"])

pd.concat({"rate": two_way, "households": counts}, axis=1)
```

Every additional dimension divides the sample again. A three-way cut of this
survey — displacement by sex by livelihood — would put single-digit household
counts in most cells, and a table of percentages computed on four households is
not evidence, however neatly it prints.

## What to report

The cuts the sample supports, with intervals, and an explicit statement of the
cuts it does not. "We could not estimate food insecurity separately for
fishing households, because barely a hundred were surveyed" is a useful sentence. Its
absence is how a reader assumes every row in your table is equally solid.