An extract you can point at — Routine Data and DHIS2 — Data Analysis

The question this lesson answers

Six months after you publish a district coverage figure, somebody reruns the query and gets something else. That will happen, and there are four legitimate reasons for it: late data entry, a revised denominator, a reassigned facility, and analytics tables that had not run.

You cannot prevent any of them. You can make it possible to say which one it was, and that is entirely a matter of what you kept.

The shape

extracts/
  2026-07-28/
    manifest.json          what was requested, by whom, when
    raw/
      analytics.json       the response, untouched
      dataValueSets.json
      dataElements.json
      organisationUnits.json
      dataSets.json
      indicators.json
    tidy/
      coverage_by_facility_period.csv

Three rules give that folder its value.

One dated folder per extraction, never overwritten. Disk is cheaper than the meeting.

The raw response is kept untouched. If your parsing turns out to be wrong, the data is still there. A tidy CSV is a derived artefact and a derived artefact cannot be un-derived.

The manifest is the point. Everything else is recoverable from it.

The manifest

import json
import datetime as dt
from pathlib import Path

def write_manifest(folder, request, system_info):
    manifest = {
        "extracted_at": dt.datetime.now(dt.timezone.utc).isoformat(),
        "extracted_by": "me-officer@example.org",
        "instance": request["base_url"],
        "dhis2_version": system_info.get("version"),
        "analytics_last_run": system_info.get("lastAnalyticsTableSuccess"),
        "request": {
            "endpoint": request["endpoint"],
            "dx": request["dx"],
            "ou": request["ou"],
            "pe": request["pe"],
        },
        "rows_returned": request["rows"],
        "script_version": "extract.py 1.3",
    }
    Path(folder, "manifest.json").write_text(json.dumps(manifest, indent=2))
    return manifest

manifest <- list(
  extracted_at        = format(Sys.time(), "%Y-%m-%dT%H:%M:%SZ", tz = "UTC"),
  instance            = base_url,
  analytics_last_run  = system_info$lastAnalyticsTableSuccess,
  request             = list(endpoint = "analytics", dx = dx, ou = ou, pe = pe),
  rows_returned       = nrow(tidy),
  script_version      = "extract.R 1.3"
)

jsonlite::write_json(manifest, file.path(folder, "manifest.json"), auto_unbox = TRUE)

Two fields are the ones that earn their place. analytics_last_run tells you whether late entry could have been included. pe records the exact periods requested, which is what makes a relative window like “last twelve months” reconstructible rather than a description of the day the script ran.

The rest is provenance the Joining and Reshaping course already asked for, arriving here with a system to pull it from.

The script

def extract(base_url, dx, ou, pe, token, out_root="extracts"):
    session = requests.Session()
    session.headers["Authorization"] = f"ApiToken {token}"

    folder = Path(out_root, dt.date.today().isoformat())
    (folder / "raw").mkdir(parents=True, exist_ok=True)
    (folder / "tidy").mkdir(exist_ok=True)

    info = session.get(f"{base_url}/system/info", timeout=60).json()

    payload = analytics(dx, ou, pe, session)
    (folder / "raw" / "analytics.json").write_text(json.dumps(payload))

    for name, path in METADATA_ENDPOINTS.items():
        meta = session.get(f"{base_url}/{path}", timeout=120).json()
        (folder / "raw" / f"{name}.json").write_text(json.dumps(meta))

    tidy = to_frame(payload)
    tidy.to_csv(folder / "tidy" / "coverage_by_facility_period.csv", index=False)

    write_manifest(folder, {"base_url": base_url, "endpoint": "analytics",
                            "dx": dx, "ou": ou, "pe": pe, "rows": len(tidy)}, info)
    return folder

extract <- function(base_url, dx, ou, pe, token) {
  folder <- file.path("extracts", Sys.Date())
  dir.create(file.path(folder, "raw"), recursive = TRUE, showWarnings = FALSE)
  dir.create(file.path(folder, "tidy"), showWarnings = FALSE)
  # ... same shape: system info, analytics, metadata, tidy, manifest
  folder
}

The window is computed, then passed in. The caller decides which twelve months; the script records them. That one choice is what separates an extract you can point at from one you can only rerun.

def last_full_months(n, today=None):
    today = today or dt.date.today()
    first_of_this = today.replace(day=1)
    months = []
    for i in range(n, 0, -1):
        m = first_of_this - dt.timedelta(days=1)
        for _ in range(i - 1):
            m = m.replace(day=1) - dt.timedelta(days=1)
        months.append(m.strftime("%Y%m"))
    return months

last_full_months <- function(n) {
  ends <- seq(Sys.Date(), by = "-1 month", length.out = n + 1)[-1]
  rev(format(ends, "%Y%m"))
}

Note last_full_months, not last_months. The current month is always incomplete, and including it makes every trend appear to fall — the partial period trap from the joining course, arriving with a system that will happily serve it to you.

Validate at the boundary

The extract is a read from someone else’s system, which makes it exactly the seam the cleaning course put a contract on.

def check(tidy, expected_org_units, expected_periods):
    problems = []
    if tidy["ou"].nunique() != expected_org_units:
        problems.append(f"{tidy['ou'].nunique()} org units, expected {expected_org_units}")
    if tidy["pe"].nunique() != expected_periods:
        problems.append(f"{tidy['pe'].nunique()} periods, expected {expected_periods}")
    if tidy["value"].isna().any():
        problems.append("null values in the response")
    return problems

check <- function(tidy, expected_ou, expected_pe) {
  c(if (n_distinct(tidy$ou) != expected_ou) "unexpected org unit count",
    if (n_distinct(tidy$pe) != expected_pe) "unexpected period count")
}

The most valuable of those is the org unit count. A silently smaller extract is the commonest API failure in practice — a permissions change, a reorganisation, a unit closed — and it produces a district total that is quietly lower with nothing in the file to say why.

Diff against the last extract

Two dated folders make a comparison possible, and the comparison is where the four explanations get separated.

def compare(old_folder, new_folder, keys=("ou", "pe", "dx")):
    old = pd.read_csv(Path(old_folder, "tidy", "coverage_by_facility_period.csv"))
    new = pd.read_csv(Path(new_folder, "tidy", "coverage_by_facility_period.csv"))

    merged = old.merge(new, on=list(keys), how="outer",
                       suffixes=("_old", "_new"), indicator=True)
    changed = merged[merged["value_old"] != merged["value_new"]]
    return {
        "in_both": int((merged["_merge"] == "both").sum()),
        "new_only": int((merged["_merge"] == "right_only").sum()),
        "gone": int((merged["_merge"] == "left_only").sum()),
        "values_changed": len(changed),
    }

compare <- function(old, new) {
  full_join(old, new, by = c("ou", "pe", "dx"), suffix = c("_old", "_new")) |>
    summarise(changed = sum(value_old != value_new, na.rm = TRUE),
              gone = sum(is.na(value_new)), added = sum(is.na(value_old)))
}

Four numbers, and each points at a different cause. Values changed with the same keys is late entry or a revision. Rows gone is a reassignment or a permissions change. Rows added is late reporting arriving. And a large change with analytics_last_run moving in between is the analytics tables, not the data.

Run this every month and keep the output. It is the DQA course’s finding series, built from a source you now control.

What this replaces

Before:  a spreadsheet in an email, monthly, method unknown
After:   extracts/2026-07-28/, one command, manifest, raw responses,
         a diff against last month, and a figure you can defend in September

The script is about eighty lines. It is the single highest-return piece of code in this course, and it is the one most teams never write because the manual export works fine on the day.

What comes next

You have a routine figure you can point at. The last unit puts it beside a survey estimate of the same thing — three numbers for acute malnutrition, from 8.7% to 14.9% — and decomposes the gap into the two things that actually cause it.