A pipeline that fails is better than one that guesses

The failure that has no error message

The monthly export arrives. A column that used to say true now says Y. Your boolean cast turns every one of them into missing, the denominator shrinks by 30%, and the pipeline runs to completion and produces a report.

Nothing errored. The chart is drawn, the percentage is plausible, the file is dated today, and the number is wrong.

That is the failure mode this lesson exists for, and it is much more common than a crash. A crash gets fixed the same morning.

Seven checks, each with a real failure behind it

Every one of these corresponds to a defect documented in this platform’s own datasets.

One: the columns you expect are present.

REQUIRED = {"household_id", "district", "water_source", "round_trip_minutes"}
missing = REQUIRED - set(survey.columns)
if missing:
    raise ValueError(f"Export is missing columns: {sorted(missing)}")

stopifnot(all(required %in% names(survey)))

Two: the row count is in the range you expect.

if not 2_000 <= len(survey) <= 3_000:
    raise ValueError(f"Expected 2,000-3,000 households, got {len(survey):,}")

Half an export is the commonest silent failure. A truncated download, a filter left on, a date range off by a month — all produce a valid file with too few rows.

Three: the codes are the codes you know.

KNOWN = {"piped-into-dwelling", "piped-into-yard", "public-tap", "borehole",
         "protected-well", "protected-spring", "unprotected-well",
         "unprotected-spring", "surface-water", "tanker-truck", "rainwater"}
unknown = set(survey["water_source"].dropna()) - KNOWN
if unknown:
    raise ValueError(f"Unknown water_source values: {sorted(unknown)}")

setdiff(unique(survey$water_source), known)

A new code is a decision, not a data point. Somebody added an option to the form and the analysis has to decide where it belongs — silently dropping it into “other” is the decision being made by a .fillna().

Four: the identifier is unique where it should be.

duplicates = survey["household_id"].duplicated().sum()
if duplicates:
    raise ValueError(f"{duplicates} duplicate household_id values")

This platform’s school roster has exactly this defect — two students appear twice after a transfer that was never de-registered — and a bare merge fans their rows out. A check would have caught it at the join rather than in a coefficient.

Five: the missingness is where you expect it.

completeness = survey.notna().mean()
if completeness["district"] < 0.99:
    raise ValueError(f"district is {completeness['district']:.1%} complete")

Six: the numbers are in a possible range.

implausible = survey[(survey["litres_per_person_day"] < 0)
                     | (survey["litres_per_person_day"] > 200)]
if len(implausible) > 20:
    raise ValueError(f"{len(implausible)} implausible litres values")

Note the threshold rather than zero tolerance. Eleven households with a unit error is a documented defect this analysis handles; two hundred is a new problem.

Seven: the output is what you declared.

assert summary["n"].sum() == len(survey), "rows lost between input and summary"

A row count that changes across a join is the single most useful assertion in programme analysis, because an inner join that drops a third of the data looks exactly like an inner join that drops nothing.

Fail at the point of failure, not at the end

def load_survey(path: pathlib.Path) -> pd.DataFrame:
    survey = pd.read_csv(path)
    check_columns(survey)
    check_rows(survey)
    check_codes(survey)
    return survey                # nothing downstream runs on a bad file

load_survey <- function(path) {
  survey <- readr::read_csv(path)
  check_columns(survey); check_rows(survey); check_codes(survey)
  survey
}

Check at the boundary — where data enters, and after every join. A check at the end of the pipeline tells you something is wrong; a check at the boundary tells you what.

Raise, do not warn. A warning in a log nobody reads is the same as no check, and the log is not read precisely on the busy days when the export breaks.

What this platform does

Seven checks fail astro build, deliberately, and they are the same shape.

Check	Catches
Zod schema	A field missing or of the wrong type
Cross-collection references	A path pointing at a course that does not exist
Files outside a locale directory	An entry with no language
Translation parity	A published entry in one language only
Topic-sector consistency	A topic tagged outside its sector
Synthetic-only	A dataset that is not declared synthetic
Programme spine	A published course missing from the curriculum map

Four more run in pnpm test rather than in the build, because they need node:fs and the build prerenders inside a Cloudflare worker that has none. That is a real constraint that shaped where checks live, and it is worth naming: put the check where it can run, not where it feels tidiest.

One of them checks a relation the reference graph structurally cannot. The reference rules verify that a declared slug resolves; they cannot verify that an entry is pointed at. So a course could ship with no practice attached and every gate would stay green — which is what course-practice.test.ts exists for, counting backwards from each published course to the lab and exercise that must name it.

Any “every X has at least one Y” rule needs a test of that shape. A reference graph is the wrong tool for it.

The rule worth taking

Any field naming a shipped file needs a ready flag and a filesystem test beside it. This platform learned it three times: twenty worked examples declared and never written, thirty-five project deliverables declared and never written, and a set of figure paths that pointed at nothing.

A path in frontmatter is a string, and nothing in a build can tell whether it points at anything. So the schema defaults ready to false and a test checks the file exists.

Report it whole

Pipeline checks

  The loader validates every raw export before anything downstream runs:
  required columns present, 2,000-3,000 rows, water_source values within the
  known set, household_id unique, district at least 99% complete.

  Implausible litres-per-person values are tolerated up to 20 rows, which is
  the documented unit-entry defect; above that the run stops.

  Row counts are asserted across every join. A join that changes the row
  count stops the pipeline rather than producing a summary.

  All checks raise rather than warn. The March run stopped on an unknown
  water_source value ("piped-shared"), which turned out to be a new form
  option added upstream; it is now mapped explicitly in src/clean.py.

The last sentence is what makes the section credible. A checks section that has never caught anything is a checks section nobody has tested.

What comes next

Everything so far assumes you are still here. The next lesson is about the document that has to work when you are not — written for someone with your job and none of your context.