Where most indicator bugs are born

An indicator is two numbers

Almost every figure this sector reports is a numerator over a denominator, and almost every argument about a figure is an argument about the denominator. The practical consequence: compute both in the same call.

library(dplyr)

by_commune <- muac |>
  mutate(
    assessed = !is.na(muac_mm) | !is.na(oedema),
    gam = assessed & (muac_mm < 125 | oedema == "true")
  ) |>
  group_by(commune) |>
  summarise(
    screened  = n(),
    assessed  = sum(assessed, na.rm = TRUE),
    gam_cases = sum(gam, na.rm = TRUE),
    .groups = "drop"
  ) |>
  mutate(gam_rate = round(gam_cases / assessed, 3))

Three columns and every one is load-bearing. screened is how many children came; assessed is how many produced a usable measurement and is the denominator; gam_cases is the numerator. Two separate calculations drift — one gets a filter the other does not — and the ratio becomes a number nobody can reconstruct.

n() and the two ways to count

summarise(muac, rows = n(), measured = sum(!is.na(muac_mm)))

n() counts rows in the group, including those with missing values. Summing a logical counts the TRUEs. That distinction is the difference between the two denominators above, and it is worth saying out loud every time you use one.

sum() on a logical with NA in it returns NA, which is R being loud again:

sum(c(TRUE, NA))
#> [1] NA

sum(c(TRUE, NA), na.rm = TRUE)
#> [1] 1

Reach for na.rm = TRUE deliberately, and know that it removes the row from the count rather than counting it as FALSE. Those are different claims.

The grouping survives, and it is the commonest bug

This is the one to internalise. summarise() removes the last grouping variable and leaves the rest:

out <- muac |>
  mutate(month = format(screening_date, "%Y-%m")) |>
  group_by(commune, month) |>
  summarise(n = n())

`summarise()` has regrouped the output.
ℹ Summaries were computed grouped by commune and month.
ℹ Output is grouped by commune.
ℹ Use `summarise(.groups = "drop_last")` to silence this message.

class(out)[1]
#> [1] "grouped_df"

dplyr::group_vars(out)
#> [1] "commune"

The result is still grouped by commune. Every verb after this silently operates per commune:

out |> mutate(share = n / sum(n))     # share within the commune, not the district
out |> arrange(desc(n))               # sorts within each commune
out |> slice_max(n, n = 5)            # top 5 per commune, not overall

Each of those produces a plausible number that answers a different question from the one you asked. Nothing errors.

Say what you want, every time:

summarise(..., .groups = "drop")        # ungrouped result — the usual answer
summarise(..., .groups = "drop_last")   # keep all but the last (the default)
summarise(..., .groups = "keep")        # keep all groupings

Or avoid the state entirely with per-operation grouping, which dplyr gained in 1.1:

muac |>
  summarise(n = n(), .by = c(commune, month))

.by groups for that call only and returns an ungrouped result. Prefer it in new code: there is no leftover state, so there is no bug of this shape.

A missing key becomes its own group

Here R and pandas take opposite bets.

d <- tibble::tibble(g = c("a", NA, "a"), v = c(1, 2, 3))

d |> group_by(g) |> summarise(s = sum(v), .groups = "drop")

# A tibble: 2 × 2
  g         s
  <chr> <dbl>
1 a         4
2 <NA>      2

The NA group is kept. pandas drops it by default, so the same operation in the two languages gives you a different set of rows and a different total.

R’s default is the safer one, and in this sector the missing key is usually the interesting one: a facility with no district recorded, a household with no site code, a child with no commune. But it means a table can arrive at a report with an <NA> row nobody meant to publish.

d |> filter(!is.na(g)) |> group_by(g) |> summarise(s = sum(v), .groups = "drop")

Excluding it is fine. Excluding it without noticing is how the parts stop summing to the whole. Assert instead, where the key should be complete:

stopifnot(!any(is.na(muac$commune)))

Unused factor levels: count() drops, table() keeps

The second reversal, and R is not consistent with itself here — which is why the lesson on factors covers it and this one repeats it.

d <- tibble::tibble(k = factor(c("a", "b", "a"), levels = c("a", "b", "c")))

nrow(count(d, k))                #> 2   — the empty level is gone
nrow(count(d, k, .drop = FALSE)) #> 3   — kept
length(table(d$k))               #> 3   — kept

Both are right for different questions, and it is the same question the Python course raises about observed=. Reporting on facilities that submitted data wants the empty level gone; reporting coverage against a list of facilities that should have submitted wants it there, because a facility with zero rows is the finding.

Pass .drop explicitly in anything that produces a reported table.

Grouped mutate(): a group statistic on every row

summarise() collapses; mutate() on a grouped frame returns a value per original row. This is how you compare a row against its own group without a join:

muac |>
  group_by(commune) |>
  mutate(
    commune_mean = mean(muac_mm, na.rm = TRUE),
    vs_commune   = muac_mm - commune_mean
  ) |>
  ungroup()

That is the check the SMART survey analysis project uses to find a team measuring half a kilogram light.

Note the ungroup(). A grouped frame that escapes into the rest of a script is the same bug as the one above, arriving from a different direction.

count() for when that is all you want

muac |> count(commune, outcome, sort = TRUE)
muac |> count(commune, wt = muac_mm)          # sum a column instead of counting rows

count() groups, counts and ungroups in one call, so it cannot leave state behind. Where a count is all you need, it is the better verb.

Two keys, and the table a report wants

by_commune_month <- muac |>
  mutate(month = format(screening_date, "%Y-%m")) |>
  summarise(cases = sum(gam, na.rm = TRUE), .by = c(commune, month))

wide <- by_commune_month |>
  tidyr::pivot_wider(names_from = month, values_from = cases, values_fill = 0)

values_fill = 0 is safe here because a commune-month with no cases genuinely had zero. It would be wrong for a rate, where the absence means “not computed”, not “zero percent” — a distinction worth checking every time you reach for it.

Writing the result out

readr::write_csv(by_commune, here::here("outputs", "tables", "gam_by_commune.csv"))

For a table someone will read rather than compute on, write the definition with it:

definitions <- tibble::tibble(
  indicator   = "GAM",
  numerator   = "MUAC < 125 mm or bilateral pitting oedema",
  denominator = "children with a MUAC measurement or a recorded oedema assessment"
)

readr::write_csv(definitions, here::here("outputs", "tables", "definitions.csv"))

A number and its definition travel together, or the monthly argument about the denominator starts again.

The three reversals, in one place

	pandas	R / dplyr
Missing value in mean()	skipped silently	returns NA until you say na.rm
Missing group key	dropped	kept as its own group
Unused category	dropped by observed=True	kept by table(), dropped by count()

None is better. Each is a different bet about what silence should mean, and knowing which bet you are inside is most of the job.

What comes next

The table aggregates correctly and both halves of every rate come from one call. The next unit reshapes: pivot_longer() on the flattened repeat group a CommCare or Kobo export arrives as, and then the function that makes the whole cleaning run again next quarter.