cassionData Analysis

Back to the lessonLesson 8 of 8Producing and handing over

A script another officer can run

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 25

    What this lesson covers

    • The handover test
    • Arguments, not edits
    • Fail loudly, early
    • Logging, not print
    • The shape of the script
    • Testing the part that computes
    • Making the run reproducible
    • Handing it over
    • Where this course ends
    Speaker notes
    Arguments instead of edits, logging instead of print, failing loudly instead of producing a wrong number — turning the analysis into something that runs next quarter without you in the room.

  2. Slide 2 / 25

    The handover test

    • An analysis is finished when someone else can run it on next quarter's export without editing the code.
    Speaker notes
    An analysis is finished when someone else can run it on next quarter's export without editing the code. Not "without much editing" — without editing. That standard rules out the three habits every notebook accumulates: a path written for one machine, a value changed by hand between runs, and a step that only works if you already know which cell to run first.

  3. Slide 3 / 25

    Arguments, not edits — In Python (cont.)

    # scripts/indicator_table.py
import argparse
from pathlib import Path

def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Compute GAM and SAM rates by commune from a screening register."
    )
    parser.add_argument(
        "--input", type=Path, required=True,
        help="Path to the screening register CSV.",
    )
    parser.add_argument(
        "--output", type=Path, required=True,
        help="Directory to write tables into. Created if missing.",
    )

  4. Slide 4 / 25

    Arguments, not edits — In Python (cont.)

        parser.add_argument(
        "--gam-threshold", type=int, default=125,
        help="MUAC cut-off in mm for global acute malnutrition (default: 125).",
    )
    parser.add_argument(
        "--min-assessment-rate", type=float, default=0.8,
        help="Communes below this assessment rate are flagged, not dropped.",
    )
    return parser.parse_args()

  5. Slide 5 / 25

    Arguments, not edits — Shell

    uv run python scripts/indicator_table.py \
    --input data/raw/muac-screening-artibonite-2024.v1.csv \
    --output outputs/tables

  6. Slide 6 / 25

    Arguments, not edits

    • Put the thresholds in arguments and the defaults in the code — A threshold that is only ever changed by editing a line…
    Speaker notes
    type=Path converts the string for you, and required=True means a forgotten argument produces a clear message rather than a KeyError two hundred lines in. Put the thresholds in arguments and the defaults in the code. A threshold that is only ever changed by editing a line will eventually be changed and not changed back. One that is an argument with a documented default is visible in the command that produced the output, which is what you want when someone asks how last quarter's figure was computed. --help is generated from the same declarations, and it is the documentation people actually read:

  7. Slide 7 / 25

    Arguments, not edits — Shell

    uv run python scripts/indicator_table.py --help

  8. Slide 8 / 25

    Fail loudly, early — In Python (cont.)

    def read_register(path: Path) -> pd.DataFrame:
    if not path.exists():
        raise SystemExit(f"Input file not found: {path}")

    muac = pd.read_csv(
        path,
        dtype={"child_id": "string", "commune": "string"},
        na_values={"muac_mm": ["-99"]},
    )

    expected = {"child_id", "commune", "screening_date", "muac_mm", "oedema"}
    missing = expected - set(muac.columns)
    if missing:
        raise SystemExit(f"Input is missing columns: {sorted(missing)}")

    if muac["child_id"].isna().any():
    Speaker notes
    A script that produces a wrong number silently is worse than one that crashes. Check what you assume, at the point you assume it.

  9. Slide 9 / 25

    Fail loudly, early — In Python (cont.)

            raise SystemExit("Some rows have no child_id; refusing to continue.")

    return muac

  10. Slide 10 / 25

    Fail loudly, early — In Python

    import warnings
import pandas as pd

warnings.simplefilter("error", pd.errors.ChainedAssignmentError)
    Speaker notes
    SystemExit with a message exits with a non-zero status and prints one readable line, rather than a traceback the reader has to interpret. Use it for "this input is not what I was promised"; use assert for "this cannot happen unless the code is wrong". Turn the pandas warning that means a silent failure into an error: Chained assignment never modifies the frame, and by default it only warns. In a script whose output someone will report, stopping is the right response.

  11. Slide 11 / 25

    Logging, not print — In Python

    import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)-8s %(message)s",
    datefmt="%H:%M:%S",
)
log = logging.getLogger(__name__)

log.info("Read %d rows from %s", len(muac), args.input)
log.warning("%d communes below the assessment-rate floor", len(flagged))
log.info("Wrote %s", output_path)
    Speaker notes
    print goes to standard output alongside the results, has no severity, and cannot be turned down.

  12. Slide 12 / 25

    Logging, not print — Example

    09:14:02 INFO     Read 4218 rows from data/raw/muac-...v1.csv
09:14:02 WARNING  2 communes below the assessment-rate floor
09:14:02 INFO     Wrote outputs/tables/gam_by_commune.csv
    Speaker notes
    Three things this buys that print does not: a timestamp, a severity someone can filter on, and the ability to add --verbose without touching every call. Use the %s form rather than an f-string — the formatting is skipped entirely when the level is disabled. Log the numbers that would let someone reconstruct the run: rows read, rows excluded and why, the denominator, the output path. A log that says "done" tells nobody anything.

  13. Slide 13 / 25

    The shape of the script — In Python (cont.)

    # scripts/indicator_table.py
"""Compute GAM and SAM rates by commune from a MUAC screening register."""

import argparse
import logging
from pathlib import Path

import pandas as pd

log = logging.getLogger(__name__)

GAM_DEFAULT, SAM_DEFAULT = 125, 115


def read_register(path: Path) -> pd.DataFrame:
    ...

  14. Slide 14 / 25

    The shape of the script — In Python (cont.)

    

def indicator_table(muac: pd.DataFrame, gam_mm: int, sam_mm: int) -> pd.DataFrame:
    measured = muac["muac_mm"].notna() | muac["oedema"].notna()
    gam = measured & ((muac["muac_mm"] < gam_mm) | (muac["oedema"] == True))
    sam = measured & ((muac["muac_mm"] < sam_mm) | (muac["oedema"] == True))

    table = (
        muac.assign(measured=measured, gam=gam, sam=sam)
        .groupby("commune", dropna=False, observed=True)
        .agg(
            screened=("child_id", "size"),
            assessed=("measured", "sum"),
            gam_cases=("gam", "sum"),
            sam_cases=("sam", "sum"),
        )

  15. Slide 15 / 25

    The shape of the script — In Python (cont.)

        )
    table["assessment_rate"] = (table["assessed"] / table["screened"]).round(3)
    table["gam_rate"] = (table["gam_cases"] / table["assessed"]).round(3)
    table["sam_rate"] = (table["sam_cases"] / table["assessed"]).round(3)
    return table.reset_index()


def main() -> None:
    args = parse_args()
    logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(message)s")

    muac = read_register(args.input)
    log.info("Read %d rows", len(muac))

    table = indicator_table(muac, args.gam_threshold, SAM_DEFAULT)

  16. Slide 16 / 25

    The shape of the script — In Python (cont.)

        thin = table.loc[table["assessment_rate"] < args.min_assessment_rate, "commune"]
    if len(thin):
        log.warning("Assessment rate below floor: %s", ", ".join(thin))

    args.output.mkdir(parents=True, exist_ok=True)
    destination = args.output / "gam_by_commune.csv"
    table.to_csv(destination, index=False)
    log.info("Wrote %s (%d communes)", destination, len(table))


if __name__ == "__main__":
    main()

  17. Slide 17 / 25

    The shape of the script

    • Functions take arguments and return values — indicator_table does not read a file, does not know where output goes,…
    • main() is the only place that does input and output — Reading, writing and logging live there; everything else is…
    • The if __name__ == "__main__" guard — means importing this module from a notebook runs nothing
    • Thin data is flagged, not dropped — A commune below the assessment-rate floor stays in the table with a warning beside…
    Speaker notes
    Four things about that structure are deliberate. Functions take arguments and return values. indicator_table does not read a file, does not know where output goes, and does not touch a global. That is what makes it testable and what lets the notebook import it. main() is the only place that does input and output. Reading, writing and logging live there; everything else is computation. The if __name__ == "__main__" guard means importing this module from a notebook runs nothing. Without it, from indicator_table import indicator_table executes the whole script. Thin data is flagged, not dropped. A commune below the assessment-rate floor stays in the table with a warning beside it. Dropping it would change the district denominator and nothing in the output would say so.

  18. Slide 18 / 25

    Testing the part that computes — In Python (cont.)

    # tests/test_indicator_table.py
import pandas as pd
from indicator_table import indicator_table


def test_denominator_excludes_unmeasured_children():
    muac = pd.DataFrame({
        "child_id": ["a", "b", "c"],
        "commune": ["X", "X", "X"],
        "muac_mm": [110, 130, None],
        "oedema": [False, False, None],
    })

    table = indicator_table(muac, gam_mm=125, sam_mm=115)

    assert table.loc[0, "screened"] == 3
    Speaker notes
    Once the computation is a function taking a frame, testing it is three lines:

  19. Slide 19 / 25

    Testing the part that computes — In Python (cont.)

        assert table.loc[0, "assessed"] == 2      # the unmeasured child is not a denominator
    assert table.loc[0, "gam_rate"] == 0.5
    Speaker notes
    That test encodes the decision from lesson 6 — which children are in the denominator — in a form that fails if someone changes it. It is worth writing for every indicator whose definition you had to argue about.

  20. Slide 20 / 25

    Making the run reproducible — In Python (cont.)

    import json
import subprocess
from datetime import datetime, timezone


def run_metadata(args) -> dict:
    return {
        "generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "input": str(args.input),
        "gam_threshold": args.gam_threshold,
        "pandas": pd.__version__,
        "commit": subprocess.run(
            ["git", "rev-parse", "--short", "HEAD"],
            capture_output=True, text=True,
        ).stdout.strip() or "not a git repository",
    }
    Speaker notes
    Record what produced the output, beside the output:

  21. Slide 21 / 25

    Making the run reproducible — In Python (cont.)

    

(args.output / "run.json").write_text(json.dumps(run_metadata(args), indent=2))

  22. Slide 22 / 25

    Making the run reproducible — Shell

    rm -rf outputs/
uv run python scripts/indicator_table.py --input data/raw/muac.csv --output outputs/tables
git status
    Speaker notes
    Six months later, "which threshold produced this table" has an answer that does not depend on anyone's memory. Then the check from lesson 2, which is the real test: If that does not restore the outputs exactly, something is not reproducible.

  23. Slide 23 / 25

    Handing it over — Example

    ## Run

    uv sync
    uv run python scripts/indicator_table.py \
        --input data/raw/muac-screening-artibonite-2024.v1.csv \
        --output outputs/tables

Thresholds default to MUAC < 125 mm (GAM) and < 115 mm (SAM).
Pass --gam-threshold to change it; the value used is recorded in
outputs/tables/run.json.
    Speaker notes
    The README from lesson 2 needs one more line now — the command with real arguments: Two commands, and the thresholds visible without opening the code. That is the whole handover.

  24. Slide 24 / 25

    Where this course ends

    • You can build an environment that reinstalls offline, read any export without corrupting it, turn its codes into values, aggregate to a numerator and a denominator that came from the same operation, get the date arithmetic right, and hand the result to someone else as a script rather than as a favour.
    Speaker notes
    You can build an environment that reinstalls offline, read any export without corrupting it, turn its codes into values, aggregate to a numerator and a denominator that came from the same operation, get the date arithmetic right, and hand the result to someone else as a script rather than as a favour. What this course deliberately did not teach is what to compute — which indicator, which denominator, which threshold, and how to defend it. That is Data Analysis Foundations, and the sector courses in Sector Analysis on the programme roadmap take it further into the classifications your cluster holds you to. If your team works in R rather than Python, R for Programme Data covers the same ground in that language.

  25. Slide 25 / 25

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson