Asking the system instead of exporting by hand — Routine Data and DHIS2 — Data Analysis

The monthly export is the problem

Somebody opens the interface, picks the org units, picks the periods, picks the data elements, clicks export, and emails a spreadsheet. Next month somebody does it again, slightly differently, and nobody can say how.

That is the process this lesson replaces. Not because the interface is bad — it is how most people should use DHIS2 — but because an analysis that begins with a hand-built file cannot be reproduced, and reproducing it is the whole of what module 2 was about.

Two endpoints, two questions

The single most useful thing to know about the API is that there are two ways to get data out and they answer different questions.

Endpoint	Returns	Use it for
`/api/dataValueSets`	Raw stored values, exactly as entered	Reproducing the register; anything where you must see what was typed
`/api/analytics`	Aggregated, calculated values	Indicators, totals up the hierarchy, anything the system computes

The difference matters more than it sounds.

dataValueSets gives you the truth as stored — one row per data element, org unit, period and category option combination. No aggregation, no indicator arithmetic, nothing filled in. This is what you want when the analysis has to be defensible down to the data entry.

analytics gives you the answer the system would show on a dashboard — aggregated up the hierarchy, indicators evaluated, aggregation operators applied. This is what you want when you need the same number the ministry is looking at.

Pull both when the two are supposed to agree. Where they do not, the gap is an aggregation rule you have not accounted for, and finding it is worth an afternoon.

Addressing the cube

analytics requests are dimensional. You are asking for a slice of a cube, and each dimension is named.

GET /api/analytics.json
  ?dimension=dx:PENTA3_UID;BCG_UID
  &dimension=ou:LEVEL-4;DISTRICT_UID
  &dimension=pe:202401;202402;202403
  &displayProperty=NAME

dx — data dimension: data elements, indicators, data element operands.
ou — org units. LEVEL-4 means every unit at level 4; putting a district UID beside it scopes to that subtree.
pe — periods, either explicit (202401) or relative (LAST_12_MONTHS).

Prefer explicit periods over relative ones in a script. LAST_12_MONTHS returns a different window every month, which is convenient for a dashboard and fatal for reproducibility. Compute the window in your code, then ask for it by name, so the request itself records what was asked.

import requests

BASE = "https://play.dhis2.org/40/api"

def analytics(dx, ou, pe, session):
    response = session.get(
        f"{BASE}/analytics.json",
        params={
            "dimension": [f"dx:{';'.join(dx)}", f"ou:{ou}", f"pe:{';'.join(pe)}"],
            "displayProperty": "NAME",
            "skipMeta": "false",
        },
        timeout=120,
    )
    response.raise_for_status()
    return response.json()

library(httr2)

analytics <- function(dx, ou, pe) {
  request("https://play.dhis2.org/40/api/analytics.json") |>
    req_url_query(
      dimension = paste0("dx:", paste(dx, collapse = ";")),
      dimension = paste0("ou:", ou),
      dimension = paste0("pe:", paste(pe, collapse = ";")),
      displayProperty = "NAME",
      .multi = "explode"
    ) |>
    req_perform() |>
    resp_body_json()
}

skipMeta=false is deliberate. The metadata block carries the names behind every UID in the response, and without it you have a table of identifiers nobody can read.

The response shape

analytics returns a headers block, a rows array of arrays, and a metadata dictionary. Turn it into a frame immediately and join the names on:

def to_frame(payload):
    import pandas as pd

    columns = [h["name"] for h in payload["headers"]]
    frame = pd.DataFrame(payload["rows"], columns=columns)

    names = {uid: item["name"]
             for uid, item in payload["metaData"]["items"].items()}
    for column in ("dx", "ou", "pe"):
        if column in frame:
            frame[f"{column}_name"] = frame[column].map(names)
    return frame

to_frame <- function(payload) {
  cols <- vapply(payload$headers, \(h) h$name, character(1))
  rows <- do.call(rbind, lapply(payload$rows, \(r) unlist(r)))
  as.data.frame(rows, stringsAsFactors = FALSE) |> setNames(cols)
}

Keep the UIDs as well as the names. Names change; UIDs do not. A script that joins on a name breaks the first time somebody corrects a spelling, and it breaks silently by returning fewer rows.

The four things to get right first

Authentication. Use a personal access token where the instance supports it, basic auth otherwise, and never put either in the script. An environment variable or a credentials file outside the repository, always.

import os

session = requests.Session()
session.headers["Authorization"] = f"ApiToken {os.environ['DHIS2_TOKEN']}"

req_headers(request(url), Authorization = paste("ApiToken", Sys.getenv("DHIS2_TOKEN")))

An analysis repository holding a working credential for a national health information system is a serious incident, and it has happened often enough that this is the first thing a reviewer should look for.

Paging. Metadata endpoints page by default at 50. Data endpoints will happily return millions of rows and time out first.

def paged(url, session, page_size=1000):
    page = 1
    while True:
        payload = session.get(url, params={"page": page, "pageSize": page_size},
                              timeout=120).json()
        yield payload
        pager = payload.get("pager", {})
        if page >= pager.get("pageCount", 1):
            return
        page += 1

# Same idea: loop until page == pageCount, and never assume one request is all of it.

Rate and size. Ask for one district and one year at a time, not the country and five years. A request that returns in eight seconds beats one that times out at ninety, and the loop is three lines. Be a good citizen of a server that also has data clerks on it.

Analytics tables are stale. This is the one that surprises people. analytics reads from pre-aggregated tables that are rebuilt on a schedule, typically nightly. Data entered today is not in analytics until the tables run. dataValueSets sees it immediately.

So the two endpoints can legitimately disagree, and the correct response is not to file a bug. Check when analytics last ran:

info = session.get(f"{BASE}/system/info", timeout=60).json()
print(info.get("lastAnalyticsTableSuccess"))

resp <- request(paste0(base, "/system/info")) |> req_perform() |> resp_body_json()
resp$lastAnalyticsTableSuccess

Record that timestamp with every extract. It is the difference between “the numbers changed” and “the numbers changed because the analytics tables had not run when I pulled”.

Pull the metadata too

Everything lesson 1 asked for is an endpoint.

/api/dataElements.json?fields=id,name,aggregationType,categoryCombo[id,name]
/api/organisationUnits.json?fields=id,name,level,parent[id]&paging=false
/api/dataSets.json?fields=id,name,periodType,organisationUnits~size
/api/indicators.json?fields=id,name,numerator,denominator,indicatorType[name]

Four requests. They give you the aggregation operator, the pinned hierarchy, the expected-reports denominator and the indicator definitions — which is every question the earlier lessons said you would need to ask a colleague.

Save them beside the data, and the next lesson makes that automatic.

What comes next

You can ask the system a question. The next lesson makes the asking reproducible: one script, a stated window, the metadata pulled alongside, and a manifest that records exactly what was requested and when.