cassionData Analysis

Back to the lessonLesson 2 of 8An environment that will still run

A project someone else can open

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 19

    What this lesson covers

    • The shape of a project
    • data/raw is read-only
    • Paths that work on someone else's machine
    • Outputs are disposable, and that is the test
    • What goes in version control
    • Notebooks and scripts do different jobs
    • A README that is actually read
    • What comes next
    Speaker notes
    Where raw data, code and outputs live, why a hardcoded path guarantees the script runs on one machine, and the one rule that makes an analysis rerunnable — never write into the folder you read from.

  2. Slide 2 / 19

    The shape of a project — Example

    muac-analysis/
  data/
    raw/            the export exactly as it arrived, never edited
    interim/        intermediate files, safe to delete
  outputs/
    tables/
    figures/
  scripts/
    read_register.py
    indicator_table.py
  notebooks/
    exploration.ipynb
  pyproject.toml
  uv.lock
  README.md
  .gitignore
    Speaker notes
    Almost every analysis in this sector has the same four kinds of file, and giving each a fixed home removes a category of question no one should have to ask. The important line is the first one.

  3. Slide 3 / 19

    data/raw is read-only — Example

    data/raw/muac-screening-artibonite-2024.v1.csv
data/raw/muac-screening-artibonite-2024.v2.csv
    Speaker notes
    The export as it arrived is the only thing in the project you cannot reproduce. Everything else — the cleaned table, the figures, the indicator tables — can be regenerated by running the code again. The raw file cannot: the server it came from will have moved on. So it is never edited, never overwritten, never sorted in Excel "just to look at it". If a correction arrives, it lands beside the original as a new file with a new name rather than replacing it. That naming is worth being deliberate about:

  4. Slide 4 / 19

    data/raw is read-only — Example

    data/raw/muac_final.csv
data/raw/muac_final_v2.csv
data/raw/muac_FINAL_corrected(1).csv
    Speaker notes
    not An analysis that ran on v1 must keep reproducing on v1. If the filename is overwritten, last quarter's number can never be explained again — and explaining last quarter's number is a thing you will be asked to do.

  5. Slide 5 / 19

    Paths that work on someone else's machine — In Python

    # Runs on exactly one computer.
muac = pd.read_csv("C:/Users/marie/Desktop/muac-screening-artibonite-2024.v1.csv")
    Speaker notes
    This is the single most common reason a colleague's script fails on your laptop:

  6. Slide 6 / 19

    Paths that work on someone else's machine — In Python

    from pathlib import Path
import pandas as pd

# __file__ is this script; its parent is scripts/, whose parent is the project.
PROJECT = Path(__file__).resolve().parent.parent
RAW = PROJECT / "data" / "raw"
OUTPUTS = PROJECT / "outputs"

muac = pd.read_csv(RAW / "muac-screening-artibonite-2024.v1.csv")
    Speaker notes
    The fix is to write paths relative to the project, and to let Python work out where the project is.

  7. Slide 7 / 19

    Paths that work on someone else's machine

    • It runs from anywhere. python scripts/indicator_table.py and `python…
    • It works on Windows. Path joins with / in your source and produces \ on Windows. Never build a path by string…
    • It is greppable. Every file the script touches is under RAW or OUTPUTS, so a reader can see the inputs and…
    Speaker notes
    Three things that buys you: In a notebook there is no __file__. Set the project root explicitly in the first cell instead, and keep the rest of the notebook relative to it:

  8. Slide 8 / 19

    Paths that work on someone else's machine — In Python

    from pathlib import Path

PROJECT = Path.cwd().parent      # notebook lives in notebooks/
assert (PROJECT / "data" / "raw").exists(), f"not the project root: {PROJECT}"
    Speaker notes
    The assertion matters more than it looks. A notebook run from the wrong directory otherwise fails several cells later with a confusing error about a missing column.

  9. Slide 9 / 19

    Outputs are disposable, and that is the test — Shell

    rm -rf outputs/
uv run python scripts/indicator_table.py
git status                      # should show nothing unexpected
    Speaker notes
    If deleting outputs/ and rerunning the scripts does not restore it exactly, something in the analysis is not reproducible — a manual step, an edit made in Excel, a cell run out of order.

  10. Slide 10 / 19

    Outputs are disposable, and that is the test — In Python

    OUTPUTS = PROJECT / "outputs" / "tables"
OUTPUTS.mkdir(parents=True, exist_ok=True)

indicator_table.to_csv(OUTPUTS / "gam_by_commune.csv", index=False)
    Speaker notes
    Make this a habit before handing anything over. It is the cheapest possible check and it catches the failure that is hardest to diagnose later. Create output directories from the code rather than expecting them to exist: parents=True creates intermediate directories; exist_ok=True means a second run does not fail. A colleague cloning the project gets the folders without being told to make them.

  11. Slide 11 / 19

    What goes in version control

    CommitDo not commit
    scripts/, notebooks/data/raw/ — see below
    pyproject.toml, uv.lockdata/interim/
    README.mdoutputs/
    .gitignore.venv/, __pycache__/

  12. Slide 12 / 19

    What goes in version control — Example

    .venv/
__pycache__/
*.pyc
data/raw/
data/interim/
outputs/
.ipynb_checkpoints/

  13. Slide 13 / 19

    What goes in version control

    • Never commit raw beneficiary data — identified or pseudonymised
    Speaker notes
    Never commit raw beneficiary data, identified or pseudonymised. Git keeps every version of every file forever; a file deleted in a later commit is still in the history, and a repository that was internal on Monday can be shared on Friday. If the data is synthetic or already public, committing it is a convenience. If it is not, the raw directory stays out and the README says where the file comes from. Credentials follow the same rule and are stricter: a DHIS2 token, a KoboToolbox API key or a database password never appears in a script, a notebook or a commit.

  14. Slide 14 / 19

    What goes in version control — In Python

    import os

# Read from the environment; the value never enters the repository.
token = os.environ["DHIS2_TOKEN"]
    Speaker notes
    Keep the value in a .env file that .gitignore covers, and document the variable's name — not its value — in the README.

  15. Slide 15 / 19

    Notebooks and scripts do different jobs — In Python

    # scripts/read_register.py
from pathlib import Path
import pandas as pd

def read_register(path: Path) -> pd.DataFrame:
    return pd.read_csv(
        path,
        dtype={"child_id": "string", "commune": "string"},
        na_values={"muac_mm": ["-99"]},
    )
    Speaker notes
    Both belong in a project and they are not interchangeable. A notebook is for looking: exploring a new export, checking a distribution, producing a figure you will paste into a report. It runs out of order, holds state you cannot see, and diffs badly, which makes it a poor place for anything that must run identically next quarter. A script is for producing: it runs top to bottom, takes arguments, and either succeeds or fails. When an exploration turns into a number someone will report, move it into a script. The practical workflow is to explore in notebooks/, then move the settled part into scripts/ and import it back:

  16. Slide 16 / 19

    Notebooks and scripts do different jobs — In Python

    # In the notebook
import sys
sys.path.append(str(PROJECT / "scripts"))

from read_register import read_register

muac = read_register(RAW / "muac-screening-artibonite-2024.v1.csv")
    Speaker notes
    Now the notebook and the script read the file the same way, and there is one place to fix when the sentinel changes.

  17. Slide 17 / 19

    A README that is actually read — Example

    # MUAC screening indicator tables

Produces GAM and SAM rates by commune from the Artibonite screening register.

## Run

    uv sync
    uv run python scripts/indicator_table.py \
        --input data/raw/muac-screening-artibonite-2024.v1.csv \
        --output outputs/tables

## Data

data/raw is not committed. Download the register from the programme
share and place it there under its versioned filename.
    Speaker notes
    Two commands and three sentences beat a page nobody finishes:

  18. Slide 18 / 19

    What comes next

    • The project has a shape and the paths survive a change of machine.
    Speaker notes
    The project has a shape and the paths survive a change of machine. The next unit gets the data in: reading a CSV, an Excel workbook and a fixed-width export without losing a leading zero or misreading a date.

  19. Slide 19 / 19

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson