Three kinds of file, and only one of them is editable

The layout

project/
  data/
    raw/            never edited, never written to, ideally chmod -w
    reference/      lookup tables, thresholds, admin boundary codes
  R/  or  src/      the only files anyone edits
  outputs/
    derived/        cleaned data, disposable
    figures/        charts and the CSV of values behind each
    reports/        rendered documents
  tests/
  README.md
  environment lock  uv.lock or renv.lock

Three kinds of file and the distinction is not tidiness.

Raw data is read-only. The export from CommCare, the CSV from DHIS2, the spreadsheet the ministry sent. Once it lands in data/raw/ nothing writes to it and nobody opens it in Excel to fix a typo — a corrected raw file is a file whose provenance you have destroyed.

Derived data is disposable. Anything in outputs/ can be deleted at any moment and reproduced by running the code. If deleting outputs/ loses something, that something was not derived and belongs somewhere else.

Code is the only thing edited. Every correction, every recode, every exclusion is a line of code, which means it is visible, reviewable and rerunnable.

The test that decides whether you have it right

rm -rf outputs/ && make all      # or: python run.py, Rscript run.R
git status --short               # should be empty

Delete every output, rerun, and the repository should look untouched. That single command is the whole of reproducibility as an operational property, and it is uncomfortable the first time.

Three ways it usually fails, and each names something real:

A file in outputs/ that nothing produces. Someone made it by hand, once, and every run since has depended on it. It is raw data wearing the wrong name.

A step that only runs interactively. A cell in a notebook, a line pasted into a console, a manual download. If it is not in a script, it did not happen.

Something that must be run in a particular order and does not say so. The pipeline works on your machine because you know the order.

Make the raw data unwriteable, literally

chmod -R a-w data/raw/

# In R, the equivalent discipline is a function that only ever reads:
read_raw <- function(name) readr::read_csv(file.path("data/raw", name))

It costs one command and prevents the failure it names. An accidental to_csv("data/raw/survey.csv") in a notebook is the most destructive thing an analyst does, it produces no error, and nothing downstream can detect it afterwards.

Where the raw file is large or restricted, commit its checksum instead.

import hashlib, pathlib

def fingerprint(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

print(fingerprint(pathlib.Path("data/raw/survey-2025.csv")))

tools::md5sum("data/raw/survey-2025.csv")

A checksum in the repository turns “is this the same export?” into a question with an answer. Ninety per cent of “the numbers changed and I do not know why” is a source file that changed and nobody noticed.

Naming, which is provenance

Name	What it tells you
survey.csv	Nothing
final_survey_v2_FINAL.xlsx	That there are several and this one won an argument
household-survey-2025.v1.csv	What, when, and which version

Version by filename, never by overwriting. This platform’s datasets do exactly that: a correction ships as .v2.csv and never replaces .v1.csv, because a notebook pinned to v1 has to keep reproducing.

That is why the dataset files can be cached immutably, and why the course PDFs — which are regenerated in place — deliberately cannot be.

What goes in the repository

In	Out
Code, every script	Raw data with personal information
The environment lock file	Anything above about 50 MB
Small reference tables	Credentials, tokens, connection strings
Generated outputs the build cannot rebuild	.Rhistory, __pycache__, .DS_Store
A README	Outputs that regenerate in seconds

The fourth row is a judgement rather than a rule. This platform commits its generated PDFs, decks and figures — because the deploy runs on Cloudflare with no TeX and no Python, so a contributor without the toolchain can still ship a content change. Commit a generated artefact when rebuilding it is not available to everyone who needs it, and not otherwise.

The README that is actually read

Four sections, and it goes stale less often than a long one.

# Nutrition surveillance analysis, Artibonite

## What this produces
A quarterly GAM estimate by commune, and the figures in the cluster report.

## Run it
    uv sync
    uv run python run.py

## Where the data comes from
data/raw/muac-screening-*.csv — monthly export from the screening database,
downloaded by the M&E officer on the 5th. Checksums in data/raw/CHECKSUMS.

## What you need to know
Commune names arrive spelled four ways; normalisation is in src/clean.py and
must run before anything else.

The last section is the one that saves a successor a week. It is the knowledge that lives in your head, and the handover lesson is about getting the rest of it out.

Report it whole

Analysis reproducibility

  Raw exports are read-only under data/raw/ with SHA-256 checksums committed.
  All cleaning, exclusion and recoding is in version-controlled code; no raw
  file has been edited.

  Every output in outputs/ is reproducible by `uv run python run.py` from a
  clean checkout. Deleting outputs/ and rerunning leaves the repository
  unchanged.

  Dataset files are versioned by filename. The v1 file referenced by the
  March report has not been modified; the correction ships as v2.

The second paragraph is the claim to make and the one to test before making. It is checkable in one command, and an analysis that cannot pass it should say so rather than claim otherwise.

What comes next

Version control is what makes “the only thing edited is code” into a history you can read. The next lesson is about using git on analysis work — and about the one thing that must never enter a repository holding programme data, because git is designed to never forget.