Back to the lesson·Lesson 4 of 8·Data that keeps its meaning
Factors, and the order a report needs
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- A factor is an integer with a lookup table
- The
as.numeric()trap - Setting the order you want
- Ordered factors
- Unused levels, and the two defaults that disagree
- Collapsing a long tail
- Factors and missing values
- Where factors bite in a join
- What comes next
Speaker notes
What a factor really is, the as.numeric() trap that silently returns level positions, and the forcats verbs that put a JMP ladder in ladder order instead of alphabetical order.A factor is an integer with a lookup table — In R
service <- factor(c("basic", "limited", "unimproved", "basic")) levels(service) #> [1] "basic" "limited" "unimproved" as.integer(service) #> [1] 1 2 3 1Speaker notes
Underneath, R stores1 2 3 1and a character vector of levels. Everything that follows — the ordering problem, theas.numeric()trap, unused levels — comes from that one fact. Notice the levels are alphabetical. Nothing aboutbasic,limited,unimprovedsays that is the order, and it is exactly backwards for a JMP service ladder, where unimproved is worst and basic is better than limited. Left alone, every table and every chart built from that column reads in an order that means nothing.The
as.numeric()trap — In Rage <- factor(c("10", "9", "8")) as.numeric(age) #> [1] 1 3 2Speaker notes
This is the single most expensive factor mistake, and it does not warn.The
as.numeric()trap — In Ras.numeric(as.character(age)) #> [1] 10 9 8Speaker notes
Those are level positions, not values. The levels are alphabetical —"10","8","9"— so"10"is position 1 and"9"is position 3.as.character()first, always. It arises whenever a numeric column was read as text — an Excel column with one stray"n/a"in it, a CSV read withcol_character()— and converted to a factor along the way. Watch for a mean that is suspiciously close to the number of categories.Setting the order you want — In R
library(forcats) ladder <- fct_relevel(service, "unimproved", "limited", "basic") levels(ladder) #> [1] "unimproved" "limited" "basic"Setting the order you want — In R
fct_infreq(source) # most common level first fct_reorder(name, value) # order one factor by another columnSpeaker notes
fct_relevel()names the order explicitly. Declare it once, near where the column is created, and every table, chart and model downstream inherits it. Two forcats verbs earn their place immediately:Setting the order you want — In R
library(dplyr) by_commune |> mutate(commune = fct_reorder(commune, gam_rate)) |> ggplot2::ggplot(ggplot2::aes(gam_rate, commune)) + ggplot2::geom_col()Speaker notes
fct_reorder()is what turns an unreadable bar chart into a readable one — the communes sorted by their GAM rate rather than by the alphabet — and it does it without a manuallevels =you have to update when the data changes.Ordered factors — In R
service[1] >= service[2] #> Warning: '>=' not meaningful for factors #> [1] NASpeaker notes
fct_relevel()sets the order of the levels. It does not make the factor comparable:Ordered factors — In R
ladder <- factor( c("basic", "limited"), levels = c("unimproved", "limited", "basic"), ordered = TRUE ) ladder[1] >= ladder[2] #> [1] TRUESpeaker notes
For a ladder, where "at least basic" is a real question, declare it ordered:Ordered factors
- Use ordered factors sparingly — They change how models treat the variable (polynomial contrasts rather than dummies),…
Speaker notes
Now>= "basic"computes the JMP coverage indicator directly. Withoutordered = TRUEit returnsNAwith a warning, which — if the warning is not read — becomes a coverage figure ofNAor, worse, a filter that selects nothing. Use ordered factors sparingly. They change how models treat the variable (polynomial contrasts rather than dummies), which is rarely what you want in a regression. For a service ladder, a severity band or an IPC phase, the ordering is worth it. For a commune, it is not.Unused levels, and the two defaults that disagree — In R
d <- tibble::tibble(k = factor(c("a", "b", "a"), levels = c("a", "b", "c"))) nrow(dplyr::count(d, k)) #> [1] 2 length(table(d$k)) #> [1] 3Speaker notes
A level with no rows is still a level. Whether it appears in your output depends on which function you ask, and R is not internally consistent about it.Unused levels, and the two defaults that disagree — In R
dplyr::count(d, k, .drop = FALSE) # keep the empty level droplevels(d$k) # remove levels with no rows forcats::fct_drop(d$k) # the forcats spelling of the same thingSpeaker notes
count()drops the empty level;table()keeps it. Both are right for different questions, and the question is the same one the Python for Programme Data course raises aboutobserved=: reporting on facilities that submitted data wants the empty one gone, and reporting coverage against a list of facilities that should have submitted wants it there, because a facility with zero rows is the finding. Say which you mean:Collapsing a long tail — In R
source <- factor(wash$water_source) length(levels(source)) #> [1] 10 levels(fct_lump_n(source, 4)) #> [1] "borehole" "piped-into-dwelling" "piped-into-yard" "public-tap" "Other"Speaker notes
Ten water sources is too many rows for a report table and too many colours for a chart.Collapsing a long tail
- Do not lump before computing an indicator —
Otherhere mixes protected springs, which are improved, with surface…
Speaker notes
fct_lump_n()keeps the four most common and folds the rest intoOther. Do not lump before computing an indicator.Otherhere mixes protected springs, which are improved, with surface water, which is not — so a coverage rate computed after lumping is wrong. Lump for presentation, on a copy, after the numbers are settled.- Do not lump before computing an indicator —
Collapsing a long tail — In R
fct_recode(source, improved = "borehole", improved = "protected-well", unimproved = "surface-water" )Speaker notes
fct_recode()is the explicit alternative and is what you want for a classification: it names every mapping, so a source that appears in the next export and is not in the list stays itself rather than silently joining a group.Factors and missing values — In R
table(muac$outcome, useNA = "ifany")Speaker notes
NAis not a level, so it survives every reordering and appears intable()only if you ask:Factors and missing values — In R
muac <- muac |> mutate(outcome = fct_na_value_to_level(outcome, level = "not recorded"))Speaker notes
To make missingness an explicit category — which is often right in a report, where "not recorded" is a finding rather than a gap: Once it is a level it will be counted, plotted and summed like any other. That is the point, and it is also the risk: a rate computed over a factor that includes "not recorded" has a different denominator from one that does not.Where factors bite in a join — In R
left_join(muac, sites, by = "commune")Speaker notes
Ifcommuneis a factor in one frame and a character in the other, dplyr coerces and warns. If it is a factor in both with different levels, the join still works — dplyr compares the labels, not the integers — but the result's levels are the union, which can quietly reintroduce empty categories. The simplest habit: join on character columns, convert to factor afterwards. Factors are a presentation and modelling device, not a key type.What comes next
- Columns now hold their meaning and their order.
Speaker notes
Columns now hold their meaning and their order. The next unit works them: the dplyr verbs, and then the grouping call where most indicator bugs are born.