---
title: "SMART plausibility report"
subtitle: "SMART nutrition survey, 2024"
author: "Cassion · data-analysis.cassion.dev"
format:
  html:
    toc: true
    code-fold: false
jupyter: python3
---

## What a plausibility report is for

Before a SMART survey's prevalence figure is used, the survey itself is
assessed: were the measurements taken properly, were the children sampled
properly, and is the resulting distribution shaped like a real population? A
survey that fails these checks does not get a caveat — it gets rejected, or
re-run.

This report reproduces the standard checks and ends with a verdict.

Every dataset on this platform is synthetic. No real child is described.

## Setup

```{python}
import pandas as pd
import numpy as np

BASE = "https://data-analysis.cassion.dev/datasets/"

smart = pd.read_csv(BASE + "files/smart-nutrition-survey-2024.v1.csv",
                    dtype={"child_id": "string", "sex": "string"})
reference = pd.read_csv(BASE + "reference/who-2006-weight-for-lenhei.csv")

print(f"{len(smart)} children, {smart['cluster'].nunique()} clusters, "
      f"{smart['team'].nunique()} teams")
```

## Check 1: completeness and impossible values

```{python}
impossible = (
    (smart["weight_kg"] < 2) | (smart["weight_kg"] > 30)
    | (smart["height_cm"] < 45) | (smart["height_cm"] > 130)
)

pd.DataFrame({
    "count": [
        len(smart),
        int(smart["age_months"].isna().sum()),
        int(smart["weight_kg"].isna().sum()),
        int(impossible.sum()),
    ]
}, index=["children", "missing age", "missing weight", "impossible measurement"])
```

Fourteen impossible measurements out of 930 is 1.5% — acceptable for a field
survey, and they are excluded rather than corrected because you cannot know what
the enumerator meant.

## Check 2: z-scores, so the later checks have something to work on

```{python}
plausible = smart[~impossible.fillna(False)].copy()
plausible["standard"] = np.where(plausible["age_months"] < 24, "L", "H")

lying_should_stand = (plausible["standard"] == "H") & plausible["measured_lying"]
stand_should_lie = (plausible["standard"] == "L") & ~plausible["measured_lying"]

plausible["lenhei"] = plausible["height_cm"]
plausible.loc[lying_should_stand, "lenhei"] -= 0.7
plausible.loc[stand_should_lie, "lenhei"] += 0.7
plausible["lenhei_key"] = (plausible["lenhei"] * 10).round() / 10

lms = reference.set_index(["sex", "lorh", "lenhei"])[["l", "m", "s"]]
scored = plausible.join(lms, on=["sex", "standard", "lenhei_key"])

raw_z = ((scored["weight_kg"] / scored["m"]) ** scored["l"] - 1) / (scored["l"] * scored["s"])

def sd_at(row, n):
    return row["m"] * (1 + row["l"] * row["s"] * n) ** (1 / row["l"])

def who_adjust(row, z):
    if pd.isna(z):
        return z
    if z > 3:
        sd3, sd2 = sd_at(row, 3), sd_at(row, 2)
        return 3 + (row["weight_kg"] - sd3) / (sd3 - sd2)
    if z < -3:
        sd3, sd2 = sd_at(row, -3), sd_at(row, -2)
        return -3 + (row["weight_kg"] - sd3) / (sd2 - sd3)
    return z

scored["whz"] = [who_adjust(r, z) for r, z in zip(scored.to_dict("records"), raw_z)]
```

## Check 3: flagged records

```{python}
mean_z, sd_z = scored["whz"].mean(), scored["whz"].std()

flags = pd.DataFrame({
    "flagged": [
        int((scored["whz"].abs() > 5).sum()),
        int(((scored["whz"] - mean_z).abs() > 3 * sd_z).sum()),
    ],
    "share": [
        (scored["whz"].abs() > 5).mean(),
        ((scored["whz"] - mean_z).abs() > 3 * sd_z).mean(),
    ],
}, index=["WHO (fixed -5 to +5)", "SMART (3 SD from survey mean)"]).round(4)
flags
```

SMART treats above 2.5% flagged as a problem and above 5% as grounds for
rejection. Both rules clear that comfortably here.

## Check 4: the standard deviation of the z-score

This is the single most informative number in the report. SMART expects the SD of
weight-for-height z between 0.8 and 1.2. A real population has a spread close to
1; measurement error widens it.

```{python}
analysable = scored[(scored["whz"].abs() <= 5) & scored["whz"].notna()]

print(f"mean weight-for-height z: {analysable['whz'].mean():.3f}")
print(f"SD                      : {analysable['whz'].std():.3f}")
```

At the top of the acceptable band. That is a warning, not a failure, and the next
two checks locate its source.

## Check 5: digit preference

An enumerator reading a height board under pressure rounds. A team whose
measurements pile up on `.0` and `.5` is not measuring to the millimetre they are
recording.

```{python}
scored["last_digit"] = ((scored["height_cm"] * 10).round() % 10).astype("Int64")

digits = pd.crosstab(scored["team"], scored["last_digit"], normalize="index") * 100
digits.round(1)
```

```{python}
rounded = digits[[0, 5]].sum(axis=1).round(1)
rounded.name = "% ending .0 or .5"
rounded.to_frame().assign(expected=20.0)
```

Team 2 records about 69% of its heights on a whole or half centimetre, against
17 to 23% for the other teams. With ten possible last digits, 20% is what an
unbiased team produces. This is the source of the wide standard deviation, and it
is a training issue with a name attached.

## Check 6: age heaping

Ages reported by carers rather than documents pile up on whole years. It matters
because age determines which growth standard applies — the length/height rule
switches at exactly 24 months.

```{python}
ages = scored["age_months"].dropna()
whole_years = ages.isin([12, 24, 36, 48, 60])

print(f"children at an exact whole year: {int(whole_years.sum())} ({whole_years.mean():.1%})")

ages.value_counts().reindex([23, 24, 25, 35, 36, 37, 47, 48, 49]).to_frame("children")
```

Sixty-six children recorded at exactly 24 months against 15 and 19 either side;
eighty at 36 months against 28 and 17. That is not a birth pattern, it is
rounding. Since 24 months is the boundary between the length and height
standards, some of those children are being scored against the wrong reference.

## Check 7: sex ratio

```{python}
counts = scored["sex"].value_counts()
ratio = counts.get("m", 0) / counts.get("f", 1)

chi_square = (counts.get("m", 0) - len(scored) / 2) ** 2 / (len(scored) / 4)

print(f"boys: {counts.get('m', 0)}   girls: {counts.get('f', 0)}")
print(f"ratio: {ratio:.3f}   chi-square against 1:1: {chi_square:.2f}")
```

A ratio near 1.0 and a chi-square well under 3.84 — no evidence that one sex was
preferentially sampled or preferentially skipped.

## Check 8: bias between teams

```{python}
by_team = analysable.groupby("team").agg(
    children=("whz", "size"),
    mean_z=("whz", "mean"),
    sd_z=("whz", "std"),
    mean_height=("height_cm", "mean"),
    mean_weight=("weight_kg", "mean"),
)
by_team["gam"] = analysable.groupby("team").apply(
    lambda g: ((g["whz"] < -2) | g["oedema"]).mean(), include_groups=False
)
by_team.round(3)
```

Team 3's mean z-score is -1.10 against -0.43 to -0.69 for the others, and its GAM
comes out near 22% against 10 to 16%. Clusters were assigned to teams
independently of nutrition status, so a real difference of half a z-score between
teams is not a plausible reading. Team 3 is measuring long, light, or both.

```{python}
spread = by_team["mean_z"].max() - by_team["mean_z"].min()
print(f"spread in mean z-score across teams: {spread:.2f}")
print("SMART treats a between-team spread above ~0.3 z as a supervision problem.")
```

## The verdict

```{python}
verdict = pd.DataFrame([
    ("Impossible measurements", "1.5% excluded", "pass"),
    ("Flagged records", f"{flags.loc['SMART (3 SD from survey mean)', 'share']:.1%} SMART", "pass"),
    ("SD of weight-for-height z", f"{analysable['whz'].std():.2f}", "warning"),
    ("Digit preference", "team 2 at 69% on .0/.5", "fail"),
    ("Age heaping", f"{whole_years.mean():.0%} on whole years", "warning"),
    ("Sex ratio", f"{ratio:.2f}", "pass"),
    ("Between-team bias", f"{spread:.2f} z spread", "fail"),
], columns=["check", "value", "result"])
verdict
```

**Would this survey be accepted?** Not as it stands. The prevalence figure is
computable and the sampling looks sound, but two checks fail on the same root
cause: one team measured differently from the others, and a second rounded its
heights. Both are supervision and training problems, and both inflate the spread
that the prevalence estimate rests on.

The defensible action is not to publish 14.9% with a footnote. It is to re-measure
team 3's clusters if the survey is still in the field, or to publish with the team
comparison in the body of the report rather than an annex — so the reader can see
that a sixth of the sample was measured by someone whose results do not match
anyone else's.

## What to report

Every check with its value and its threshold, the verdict, and the action. A
plausibility annex that reports only the checks that passed is not a plausibility
report.