cassionData Analysis

Back to the lessonLesson 5 of 8Getting the data out

Asking the system instead of exporting by hand

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 24

    What this lesson covers

    • The monthly export is the problem
    • Two endpoints, two questions
    • Addressing the cube
    • The response shape
    • The four things to get right first
    • Pull the metadata too
    • What comes next
    Speaker notes
    Two endpoints that answer different questions, the dimension syntax that addresses a cube, and the four things to get right before any of it — authentication, paging, rate limits and the fact that analytics tables are stale until someone runs them.

  2. Slide 2 / 24

    The monthly export is the problem

    • Somebody opens the interface, picks the org units, picks the periods, picks the data elements, clicks export, and emails a spreadsheet.
    Speaker notes
    Somebody opens the interface, picks the org units, picks the periods, picks the data elements, clicks export, and emails a spreadsheet. Next month somebody does it again, slightly differently, and nobody can say how. That is the process this lesson replaces. Not because the interface is bad — it is how most people should use DHIS2 — but because an analysis that begins with a hand-built file cannot be reproduced, and reproducing it is the whole of what module 2 was about.

  3. Slide 3 / 24

    Two endpoints, two questions

    EndpointReturnsUse it for
    /api/dataValueSetsRaw stored values, exactly as enteredReproducing the register; anything where you must see what was typed
    /api/analyticsAggregated, calculated valuesIndicators, totals up the hierarchy, anything the system computes
    Speaker notes
    The single most useful thing to know about the API is that there are two ways to get data out and they answer different questions.

  4. Slide 4 / 24

    Two endpoints, two questions

    • dataValueSets gives you the truth as stored — one row per data element, org unit, period and category option…
    • analytics gives you the answer the system would show on a dashboard — aggregated up the hierarchy, indicators…
    • Pull both when the two are supposed to agree — Where they do not, the gap is an aggregation rule you have not accounted…
    Speaker notes
    The difference matters more than it sounds. dataValueSets gives you the truth as stored — one row per data element, org unit, period and category option combination. No aggregation, no indicator arithmetic, nothing filled in. This is what you want when the analysis has to be defensible down to the data entry. analytics gives you the answer the system would show on a dashboard — aggregated up the hierarchy, indicators evaluated, aggregation operators applied. This is what you want when you need the same number the ministry is looking at. Pull both when the two are supposed to agree. Where they do not, the gap is an aggregation rule you have not accounted for, and finding it is worth an afternoon.

  5. Slide 5 / 24

    Addressing the cube — Example

    GET /api/analytics.json
  ?dimension=dx:PENTA3_UID;BCG_UID
  &dimension=ou:LEVEL-4;DISTRICT_UID
  &dimension=pe:202401;202402;202403
  &displayProperty=NAME
    Speaker notes
    analytics requests are dimensional. You are asking for a slice of a cube, and each dimension is named.

  6. Slide 6 / 24

    Addressing the cube

    • dx — data dimension: data elements, indicators, data element operands.
    • ou — org units. LEVEL-4 means every unit at level 4; putting a district UID beside it scopes to that subtree.
    • pe — periods, either explicit (202401) or relative (LAST_12_MONTHS).
    • Prefer explicit periods over relative ones in a script — LAST_12_MONTHS returns a different window every month, which…
    Speaker notes
    Prefer explicit periods over relative ones in a script. LAST_12_MONTHS returns a different window every month, which is convenient for a dashboard and fatal for reproducibility. Compute the window in your code, then ask for it by name, so the request itself records what was asked.

  7. Slide 7 / 24

    Addressing the cube — In Python

    import requests

BASE = "https://play.dhis2.org/40/api"

def analytics(dx, ou, pe, session):
    response = session.get(
        f"{BASE}/analytics.json",
        params={
            "dimension": [f"dx:{';'.join(dx)}", f"ou:{ou}", f"pe:{';'.join(pe)}"],
            "displayProperty": "NAME",
            "skipMeta": "false",
        },
        timeout=120,
    )
    response.raise_for_status()
    return response.json()

  8. Slide 8 / 24

    Addressing the cube — In R

    library(httr2)

analytics <- function(dx, ou, pe) {
  request("https://play.dhis2.org/40/api/analytics.json") |>
    req_url_query(
      dimension = paste0("dx:", paste(dx, collapse = ";")),
      dimension = paste0("ou:", ou),
      dimension = paste0("pe:", paste(pe, collapse = ";")),
      displayProperty = "NAME",
      .multi = "explode"
    ) |>
    req_perform() |>
    resp_body_json()
}
    Speaker notes
    skipMeta=false is deliberate. The metadata block carries the names behind every UID in the response, and without it you have a table of identifiers nobody can read.

  9. Slide 9 / 24

    The response shape — In Python

    def to_frame(payload):
    import pandas as pd

    columns = [h["name"] for h in payload["headers"]]
    frame = pd.DataFrame(payload["rows"], columns=columns)

    names = {uid: item["name"]
             for uid, item in payload["metaData"]["items"].items()}
    for column in ("dx", "ou", "pe"):
        if column in frame:
            frame[f"{column}_name"] = frame[column].map(names)
    return frame
    Speaker notes
    analytics returns a headers block, a rows array of arrays, and a metadata dictionary. Turn it into a frame immediately and join the names on:

  10. Slide 10 / 24

    The response shape — In R

    to_frame <- function(payload) {
  cols <- vapply(payload$headers, \(h) h$name, character(1))
  rows <- do.call(rbind, lapply(payload$rows, \(r) unlist(r)))
  as.data.frame(rows, stringsAsFactors = FALSE) |> setNames(cols)
}

  11. Slide 11 / 24

    The response shape

    • Keep the UIDs as well as the names — Names change; UIDs do not
    Speaker notes
    Keep the UIDs as well as the names. Names change; UIDs do not. A script that joins on a name breaks the first time somebody corrects a spelling, and it breaks silently by returning fewer rows.

  12. Slide 12 / 24

    The four things to get right first

    • Authentication — Use a personal access token where the instance supports it, basic auth otherwise, and never put either…
    Speaker notes
    Authentication. Use a personal access token where the instance supports it, basic auth otherwise, and never put either in the script. An environment variable or a credentials file outside the repository, always.

  13. Slide 13 / 24

    The four things to get right first — In Python

    import os

session = requests.Session()
session.headers["Authorization"] = f"ApiToken {os.environ['DHIS2_TOKEN']}"

  14. Slide 14 / 24

    The four things to get right first — In R

    req_headers(request(url), Authorization = paste("ApiToken", Sys.getenv("DHIS2_TOKEN")))

  15. Slide 15 / 24

    The four things to get right first

    • Paging — Metadata endpoints page by default at 50
    Speaker notes
    An analysis repository holding a working credential for a national health information system is a serious incident, and it has happened often enough that this is the first thing a reviewer should look for. Paging. Metadata endpoints page by default at 50. Data endpoints will happily return millions of rows and time out first.

  16. Slide 16 / 24

    The four things to get right first — In Python

    def paged(url, session, page_size=1000):
    page = 1
    while True:
        payload = session.get(url, params={"page": page, "pageSize": page_size},
                              timeout=120).json()
        yield payload
        pager = payload.get("pager", {})
        if page >= pager.get("pageCount", 1):
            return
        page += 1

  17. Slide 17 / 24

    The four things to get right first — In R

    # Same idea: loop until page == pageCount, and never assume one request is all of it.

  18. Slide 18 / 24

    The four things to get right first

    • Rate and size — Ask for one district and one year at a time, not the country and five years
    • Analytics tables are stale — This is the one that surprises people
    Speaker notes
    Rate and size. Ask for one district and one year at a time, not the country and five years. A request that returns in eight seconds beats one that times out at ninety, and the loop is three lines. Be a good citizen of a server that also has data clerks on it. Analytics tables are stale. This is the one that surprises people. analytics reads from pre-aggregated tables that are rebuilt on a schedule, typically nightly. Data entered today is not in analytics until the tables run. dataValueSets sees it immediately. So the two endpoints can legitimately disagree, and the correct response is not to file a bug. Check when analytics last ran:

  19. Slide 19 / 24

    The four things to get right first — In Python

    info = session.get(f"{BASE}/system/info", timeout=60).json()
print(info.get("lastAnalyticsTableSuccess"))

  20. Slide 20 / 24

    The four things to get right first — In R

    resp <- request(paste0(base, "/system/info")) |> req_perform() |> resp_body_json()
resp$lastAnalyticsTableSuccess

  21. Slide 21 / 24

    The four things to get right first

    • Record that timestamp with every extract — It is the difference between "the numbers changed" and "the numbers changed…
    Speaker notes
    Record that timestamp with every extract. It is the difference between "the numbers changed" and "the numbers changed because the analytics tables had not run when I pulled".

  22. Slide 22 / 24

    Pull the metadata too — Example

    /api/dataElements.json?fields=id,name,aggregationType,categoryCombo[id,name]
/api/organisationUnits.json?fields=id,name,level,parent[id]&paging=false
/api/dataSets.json?fields=id,name,periodType,organisationUnits~size
/api/indicators.json?fields=id,name,numerator,denominator,indicatorType[name]
    Speaker notes
    Everything lesson 1 asked for is an endpoint. Four requests. They give you the aggregation operator, the pinned hierarchy, the expected-reports denominator and the indicator definitions — which is every question the earlier lessons said you would need to ask a colleague. Save them beside the data, and the next lesson makes that automatic.

  23. Slide 23 / 24

    What comes next

    • You can ask the system a question.
    Speaker notes
    You can ask the system a question. The next lesson makes the asking reproducible: one script, a stated window, the metadata pulled alongside, and a manifest that records exactly what was requested and when.

  24. Slide 24 / 24

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson