Back to the lesson·Lesson 8 of 8·Someone else runs it
The case study is the thing you are reading
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- Why the platform is the example
- Raw data that is generated, not collected
- Committed output, for a reason that names its condition
- Checks in three places, and the constraint that decided where
- The check that a reference graph cannot express
- What the repository refuses to hold
- Reproducibility at the level of the language
- What this repository does not do
- Report it whole
- What comes next
Speaker notes
Twenty datasets from seventeen seeded scripts that regenerate byte-identically, twenty test files and 4,212 assertions, seven checks that fail the build, and a schema that refuses non-synthetic data. Every claim in this course is checkable in this repository.Why the platform is the example
- Every other course here uses a dataset.
Speaker notes
Every other course here uses a dataset. This one uses the repository, for a reason worth stating: a reproducible workflow is a claim about an artefact, and the only convincing demonstration is an artefact you can check. Everything below is verifiable from a clone.Raw data that is generated, not collected — Shell
pnpm datasets:generate git status --short apps/data-analysis/public/datasets/files # (nothing)Raw data that is generated, not collected
- Twenty CSV files, seventeen generator scripts, and regenerating all of them changes nothing — Each generator is seeded…
- That is a stronger property than "the pipeline reruns" — It means rerunning is a verification step: if
git status… - And the defects are deliberate — Every entry in a dataset's
knownIssueslist corresponds to a block in its generator…
Speaker notes
Twenty CSV files, seventeen generator scripts, and regenerating all of them changes nothing. Each generator is seeded —SEED = 20240622, in the file — so the output is a function of the code alone. That is a stronger property than "the pipeline reruns". It means rerunning is a verification step: ifgit statusis not empty after a regeneration, either the generator changed or the committed file was edited by hand, and both are things you want to know. And the defects are deliberate. Every entry in a dataset'sknownIssueslist corresponds to a block in its generator — the four spellings of Nord-Ouest, the eleven unit errors, the two duplicated students. Change one and you change the other, which is the same discipline as the figures being generated from the data.Committed output, for a reason that names its condition
- Because the deploy runs on Cloudflare Pages with no TeX, no Python and no pandoc — The build copies static files, so…
- That is the general rule from lesson 1, applied — Commit a generated artefact when rebuilding it is not available to…
- And it works because the output is deterministic —
SOURCE_DATE_EPOCHpins pdfTeX's timestamp and pandoc's, so a…
Speaker notes
The repository commits things it could rebuild: dataset CSVs, course PDFs, slide decks, figures, notebooks. Because the deploy runs on Cloudflare Pages with no TeX, no Python and no pandoc. The build copies static files, so LaTeX is never on the deploy critical path and a contributor without the toolchain can still ship a content change. That is the general rule from lesson 1, applied. Commit a generated artefact when rebuilding it is not available to everyone who needs it. Not because rebuilding is slow — because rebuilding is unavailable. And it works because the output is deterministic.SOURCE_DATE_EPOCHpins pdfTeX's timestamp and pandoc's, so a.pptx— a zip whose entry times would otherwise churn — rebuilds byte-identically. Without that, committing generated output would produce a diff on every build and nobody would review any of it.Checks in three places, and the constraint that decided where
Where What Why there Zod schema Field types, enums, required fields Runs on one entry, no filesystem astro buildSeven graph-level checks Needs the whole collection pnpm testFilesystem checks The build has no node:fsChecks in three places, and the constraint that decided where
- The Astro build prerenders inside a Cloudflare worker with no
node:fs— so a check that has to look at a file on disk… - Put the check where it can run — That is a real engineering constraint, and it produced a better arrangement than…
Speaker notes
The Astro build prerenders inside a Cloudflare worker with nonode:fs, so a check that has to look at a file on disk cannot live there.dataset-files.test.tsasserts that declared rows and bytes match the files;course-pdfs.test.tsasserts that every published course ships both PDF editions. Put the check where it can run. That is a real engineering constraint, and it produced a better arrangement than tidiness would have: the fast checks are in the schema, the graph checks are in the build, and the filesystem checks are in a test suite that runs first in CI.- The Astro build prerenders inside a Cloudflare worker with no
The check that a reference graph cannot express — Example
REFERENCE_RULES verifies that a declared slug resolves. It cannot verify that an entry is pointed at.The check that a reference graph cannot express
- So a course could ship with no lab and no exercise, and every gate would stay green — That is what…
- Any "every X has at least one Y" rule needs a test of that shape — and recognising which of your constraints are of…
Speaker notes
So a course could ship with no lab and no exercise, and every gate would stay green. That is whatcourse-practice.test.tsexists for — counting backwards from each published course to the practice that must name it. Any "every X has at least one Y" rule needs a test of that shape, and recognising which of your constraints are of that kind is most of designing the check suite.What the repository refuses to hold — Example
.refine((d) => d.dataQuality.synthetic === true, { message: "Only synthetic or fully de-identified datasets may be published.", })What the repository refuses to hold
- A dataset that is not declared synthetic fails the build — This audience models beneficiary, protection, GBV and…
- It pairs with generating the data rather than collecting it — The guarantee is structural at both ends: the generator…
Speaker notes
A dataset that is not declared synthetic fails the build. This audience models beneficiary, protection, GBV and clinical data, and "no record traces to a real person" is made a property of the repository rather than a promise in a note. It pairs with generating the data rather than collecting it. The guarantee is structural at both ends: the generator cannot produce a real person, and the schema will not accept a claim that it did not.Reproducibility at the level of the language — Example
{ "packageManager": "pnpm@11.9.0", "engines": { "node": ">=22" } }Reproducibility at the level of the language
- The package manager is pinned, not just the packages — A different pnpm resolves the lockfile differently, and that is…
- TypeScript is held at 6.x with the reason written down — TypeScript 7 does not yet expose the programmatic API `astro…
Speaker notes
The package manager is pinned, not just the packages. A different pnpm resolves the lockfile differently, and that is the layer below the one most projects pin. TypeScript is held at 6.x with the reason written down: TypeScript 7 does not yet expose the programmatic APIastro checkdepends on, sopnpm typecheckfails outright on 7. The pin names the condition for removing it, which is what separates a considered constraint from an inherited one.What this repository does not do
- No container — A lock file plus a pinned package manager reproduces the JavaScript side; the Python and TeX sides…
- No end-to-end determinism test in CI — The dataset regeneration is deterministic and verified by hand; there is no CI…
- Handover documentation is thin —
CLAUDE.mdanddocs/DEFINITION_OF_DONE.mdcarry the decisions, which is more than… - Naming the gaps is the point — A case study that claims everything is a case study nobody can learn from, and the three…
Speaker notes
Being honest about the gaps is part of the case study. No container. A lock file plus a pinned package manager reproduces the JavaScript side; the Python and TeX sides depend on what the contributor has.pnpm figures:buildneeds only a stock Python, andpnpm course:pdfwarns rather than failing when TeX is absent — a deliberate trade so a contributor without a toolchain can still ship content. No end-to-end determinism test in CI. The dataset regeneration is deterministic and verified by hand; there is no CI job that regenerates everything and fails on a diff. That would be the next thing to add. Handover documentation is thin.CLAUDE.mdanddocs/DEFINITION_OF_DONE.mdcarry the decisions, which is more than most repositories have and less than lesson 7 asks for. Naming the gaps is the point. A case study that claims everything is a case study nobody can learn from, and the three above are the honest state of a repository that is otherwise unusually strict with itself.Report it whole — Example (cont.)
Reproducibility of this platform Data 20 CSV files generated by 17 seeded scripts. Regenerating all of them produces byte-identical output; `pnpm datasets:generate` followed by `git status` is the verification. Checks 7 fail the build (schema, references, locale, translation parity, topic-sector, synthetic-only, programme spine). 20 test files and 4,212 assertions run first in CI. Output Course PDFs, slide decks, figures and notebooks are committed and deterministic. SOURCE_DATE_EPOCH pins pdfTeX and pandoc. Environment pnpm 11.9.0 and Node 22 pinned. TypeScript held at 6.x with the reason and the removal condition documented.Report it whole — Example (cont.)
Refused A dataset not declared synthetic fails the build. Not done No container; no CI job that regenerates everything and fails on a diff; handover documentation thinner than this course asks for.Report it whole
- The last row is the one to copy into your own version — A reproducibility statement without a "not done" section is a…
Speaker notes
The last row is the one to copy into your own version. A reproducibility statement without a "not done" section is a statement nobody has audited.What comes next
- The thread through all twenty is the same — A number in a programme report is a claim, and everything this platform…
- The last of those is the one that makes the others checkable — An analysis nobody can rerun is an analysis whose other…
Speaker notes
That is the course, and it is the last one in the programme's spine. The thread through all twenty is the same. A number in a programme report is a claim, and everything this platform teaches is about what has to be true for the claim to hold — a denominator you can defend, an interval that says how sure you are, a comparison group that is one, a chart that does not say more than the data, and a pipeline that produces the same answer when someone else runs it. The last of those is the one that makes the others checkable. An analysis nobody can rerun is an analysis whose other properties have to be taken on trust, and this audience is judged on defending a number in a review rather than on producing it.