Back to the lesson·Lesson 6 of 8·The verbs
Where most indicator bugs are born
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- An indicator is two numbers
n()and the two ways to count- The grouping survives, and it is the commonest bug
- A missing key becomes its own group
- Unused factor levels:
count()drops,table()keeps - Grouped
mutate(): a group statistic on every row count()for when that is all you want- Two keys, and the table a report wants
- Writing the result out
- The three reversals, in one place
- What comes next
Speaker notes
group_by() and summarise() — the grouping that survives the call, the missing key that becomes its own group, and the three defaults that are the exact opposite of pandas'.An indicator is two numbers — In R
library(dplyr) by_commune <- muac |> mutate( assessed = !is.na(muac_mm) | !is.na(oedema), gam = assessed & (muac_mm < 125 | oedema == "true") ) |> group_by(commune) |> summarise( screened = n(), assessed = sum(assessed, na.rm = TRUE), gam_cases = sum(gam, na.rm = TRUE), .groups = "drop" ) |> mutate(gam_rate = round(gam_cases / assessed, 3))Speaker notes
Almost every figure this sector reports is a numerator over a denominator, and almost every argument about a figure is an argument about the denominator. The practical consequence: compute both in the same call. Three columns and every one is load-bearing.screenedis how many children came;assessedis how many produced a usable measurement and is the denominator;gam_casesis the numerator. Two separate calculations drift — one gets a filter the other does not — and the ratio becomes a number nobody can reconstruct.n()and the two ways to count — In Rsummarise(muac, rows = n(), measured = sum(!is.na(muac_mm)))n()and the two ways to count — In Rsum(c(TRUE, NA)) #> [1] NA sum(c(TRUE, NA), na.rm = TRUE) #> [1] 1Speaker notes
n()counts rows in the group, including those with missing values. Summing a logical counts theTRUEs. That distinction is the difference between the two denominators above, and it is worth saying out loud every time you use one.sum()on a logical withNAin it returnsNA, which is R being loud again: Reach forna.rm = TRUEdeliberately, and know that it removes the row from the count rather than counting it asFALSE. Those are different claims.The grouping survives, and it is the commonest bug — In R
out <- muac |> mutate(month = format(screening_date, "%Y-%m")) |> group_by(commune, month) |> summarise(n = n())Speaker notes
This is the one to internalise.summarise()removes the last grouping variable and leaves the rest:The grouping survives, and it is the commonest bug — Example
`summarise()` has regrouped the output. ℹ Summaries were computed grouped by commune and month. ℹ Output is grouped by commune. ℹ Use `summarise(.groups = "drop_last")` to silence this message.The grouping survives, and it is the commonest bug — In R
class(out)[1] #> [1] "grouped_df" dplyr::group_vars(out) #> [1] "commune"The grouping survives, and it is the commonest bug — In R
out |> mutate(share = n / sum(n)) # share within the commune, not the district out |> arrange(desc(n)) # sorts within each commune out |> slice_max(n, n = 5) # top 5 per commune, not overallSpeaker notes
The result is still grouped by commune. Every verb after this silently operates per commune:The grouping survives, and it is the commonest bug
- Say what you want, every time
Speaker notes
Each of those produces a plausible number that answers a different question from the one you asked. Nothing errors. Say what you want, every time:The grouping survives, and it is the commonest bug — In R
summarise(..., .groups = "drop") # ungrouped result — the usual answer summarise(..., .groups = "drop_last") # keep all but the last (the default) summarise(..., .groups = "keep") # keep all groupingsThe grouping survives, and it is the commonest bug — In R
muac |> summarise(n = n(), .by = c(commune, month))Speaker notes
Or avoid the state entirely with per-operation grouping, which dplyr gained in 1.1:.bygroups for that call only and returns an ungrouped result. Prefer it in new code: there is no leftover state, so there is no bug of this shape.A missing key becomes its own group — In R
d <- tibble::tibble(g = c("a", NA, "a"), v = c(1, 2, 3)) d |> group_by(g) |> summarise(s = sum(v), .groups = "drop")Speaker notes
Here R and pandas take opposite bets.A missing key becomes its own group — Example
# A tibble: 2 × 2 g s <chr> <dbl> 1 a 4 2 <NA> 2A missing key becomes its own group — In R
d |> filter(!is.na(g)) |> group_by(g) |> summarise(s = sum(v), .groups = "drop")Speaker notes
TheNAgroup is kept. pandas drops it by default, so the same operation in the two languages gives you a different set of rows and a different total. R's default is the safer one, and in this sector the missing key is usually the interesting one: a facility with no district recorded, a household with no site code, a child with no commune. But it means a table can arrive at a report with an<NA>row nobody meant to publish.A missing key becomes its own group — In R
stopifnot(!any(is.na(muac$commune)))Speaker notes
Excluding it is fine. Excluding it without noticing is how the parts stop summing to the whole. Assert instead, where the key should be complete:Unused factor levels:
count()drops,table()keeps — In Rd <- tibble::tibble(k = factor(c("a", "b", "a"), levels = c("a", "b", "c"))) nrow(count(d, k)) #> 2 — the empty level is gone nrow(count(d, k, .drop = FALSE)) #> 3 — kept length(table(d$k)) #> 3 — keptSpeaker notes
The second reversal, and R is not consistent with itself here — which is why the lesson on factors covers it and this one repeats it.Unused factor levels:
count()drops,table()keeps- Pass
.dropexplicitly — in anything that produces a reported table
Speaker notes
Both are right for different questions, and it is the same question the Python course raises aboutobserved=. Reporting on facilities that submitted data wants the empty level gone; reporting coverage against a list of facilities that should have submitted wants it there, because a facility with zero rows is the finding. Pass.dropexplicitly in anything that produces a reported table.- Pass
Grouped
mutate(): a group statistic on every row — In Rmuac |> group_by(commune) |> mutate( commune_mean = mean(muac_mm, na.rm = TRUE), vs_commune = muac_mm - commune_mean ) |> ungroup()Speaker notes
summarise()collapses;mutate()on a grouped frame returns a value per original row. This is how you compare a row against its own group without a join:Grouped
mutate(): a group statistic on every row- Note the
ungroup()— A grouped frame that escapes into the rest of a script is the same bug as the one above,…
Speaker notes
That is the check the SMART survey analysis project uses to find a team measuring half a kilogram light. Note theungroup(). A grouped frame that escapes into the rest of a script is the same bug as the one above, arriving from a different direction.- Note the
count()for when that is all you want — In Rmuac |> count(commune, outcome, sort = TRUE) muac |> count(commune, wt = muac_mm) # sum a column instead of counting rowsSpeaker notes
count()groups, counts and ungroups in one call, so it cannot leave state behind. Where a count is all you need, it is the better verb.Two keys, and the table a report wants — In R
by_commune_month <- muac |> mutate(month = format(screening_date, "%Y-%m")) |> summarise(cases = sum(gam, na.rm = TRUE), .by = c(commune, month)) wide <- by_commune_month |> tidyr::pivot_wider(names_from = month, values_from = cases, values_fill = 0)Speaker notes
values_fill = 0is safe here because a commune-month with no cases genuinely had zero. It would be wrong for a rate, where the absence means "not computed", not "zero percent" — a distinction worth checking every time you reach for it.Writing the result out — In R
readr::write_csv(by_commune, here::here("outputs", "tables", "gam_by_commune.csv"))Writing the result out — In R
definitions <- tibble::tibble( indicator = "GAM", numerator = "MUAC < 125 mm or bilateral pitting oedema", denominator = "children with a MUAC measurement or a recorded oedema assessment" ) readr::write_csv(definitions, here::here("outputs", "tables", "definitions.csv"))Speaker notes
For a table someone will read rather than compute on, write the definition with it: A number and its definition travel together, or the monthly argument about the denominator starts again.The three reversals, in one place
pandas R / dplyr Missing value in mean()skipped silently returns NAuntil you sayna.rmMissing group key dropped kept as its own group Unused category dropped by observed=Truekept by table(), dropped bycount()Speaker notes
None is better. Each is a different bet about what silence should mean, and knowing which bet you are inside is most of the job.What comes next
- The table aggregates correctly and both halves of every rate come from one call.
Speaker notes
The table aggregates correctly and both halves of every rate come from one call. The next unit reshapes:pivot_longer()on the flattened repeat group a CommCare or Kobo export arrives as, and then the function that makes the whole cleaning run again next quarter.