Three kinds of file, and only one of them is editable

Slide 1 / 21

What this lesson covers

• The layout
• The test that decides whether you have it right
• Make the raw data unwriteable, literally
• Naming, which is provenance
• What goes in the repository
• The README that is actually read
• Report it whole
• What comes next

Slide 2 / 21

The layout — Example

project/
  data/
    raw/            never edited, never written to, ideally chmod -w
    reference/      lookup tables, thresholds, admin boundary codes
  R/  or  src/      the only files anyone edits
  outputs/
    derived/        cleaned data, disposable
    figures/        charts and the CSV of values behind each
    reports/        rendered documents
  tests/
  README.md
  environment lock  uv.lock or renv.lock

Slide 3 / 21

The layout

• Three kinds of file and the distinction is not tidiness
• Raw data is read-only — The export from CommCare, the CSV from DHIS2, the spreadsheet the ministry sent
• Derived data is disposable — Anything in outputs/ can be deleted at any moment and reproduced by running the code
• Code is the only thing edited — Every correction, every recode, every exclusion is a line of code, which means it is…

Slide 4 / 21

The test that decides whether you have it right — Shell

rm -rf outputs/ && make all      # or: python run.py, Rscript run.R
git status --short               # should be empty

Slide 5 / 21

The test that decides whether you have it right

• Delete every output, rerun, and the repository should look untouched — That single command is the whole of…
• Three ways it usually fails — and each names something real:
• A file in outputs/ that nothing produces — Someone made it by hand, once, and every run since has depended on it
• A step that only runs interactively — A cell in a notebook, a line pasted into a console, a manual download
• Something that must be run in a particular order and does not say so — The pipeline works on your machine because you…

Slide 6 / 21

Make the raw data unwriteable, literally — Shell

chmod -R a-w data/raw/

Slide 7 / 21

Make the raw data unwriteable, literally — In R

# In R, the equivalent discipline is a function that only ever reads:
read_raw <- function(name) readr::read_csv(file.path("data/raw", name))

Slide 8 / 21

Make the raw data unwriteable, literally

• It costs one command and prevents the failure it names — An accidental to_csv("data/raw/survey.csv") in a notebook is…
• Where the raw file is large or restricted, commit its checksum instead

Slide 9 / 21

Make the raw data unwriteable, literally — In Python

import hashlib, pathlib

def fingerprint(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

print(fingerprint(pathlib.Path("data/raw/survey-2025.csv")))

Slide 10 / 21

Make the raw data unwriteable, literally — In R

tools::md5sum("data/raw/survey-2025.csv")

Slide 11 / 21

Make the raw data unwriteable, literally

• A checksum in the repository turns "is this the same export?" into a question with an answer — Ninety per cent of "the…

Slide 12 / 21

Naming, which is provenance

Slide 13 / 21

Naming, which is provenance

• Version by filename, never by overwriting — This platform's datasets do exactly that: a correction ships as .v2.csv…
• That is why the dataset files can be cached immutably — and why the course PDFs — which are regenerated in place —…

Slide 14 / 21

What goes in the repository

Slide 15 / 21

What goes in the repository

• The fourth row is a judgement rather than a rule — This platform commits its generated PDFs, decks and figures —…

Slide 16 / 21

The README that is actually read — Example

# Nutrition surveillance analysis, Artibonite

## What this produces
A quarterly GAM estimate by commune, and the figures in the cluster report.

## Run it
    uv sync
    uv run python run.py

## Where the data comes from
data/raw/muac-screening-*.csv — monthly export from the screening database,
downloaded by the M&E officer on the 5th. Checksums in data/raw/CHECKSUMS.

## What you need to know
Commune names arrive spelled four ways; normalisation is in src/clean.py and
must run before anything else.

Slide 17 / 21

The README that is actually read

• The last section is the one that saves a successor a week — It is the knowledge that lives in your head, and the…

Slide 18 / 21

Report it whole — Example

Analysis reproducibility

  Raw exports are read-only under data/raw/ with SHA-256 checksums committed.
  All cleaning, exclusion and recoding is in version-controlled code; no raw
  file has been edited.

  Every output in outputs/ is reproducible by `uv run python run.py` from a
  clean checkout. Deleting outputs/ and rerunning leaves the repository
  unchanged.

  Dataset files are versioned by filename. The v1 file referenced by the
  March report has not been modified; the correction ships as v2.

Slide 19 / 21

Report it whole

• The second paragraph is the claim to make and the one to test before making — It is checkable in one command, and an…

Slide 20 / 21

What comes next

• Version control is what makes "the only thing edited is code" into a history you can read.

Slide 21 / 21

Where this goes next

Read the full lesson, with runnable code Back to the lesson