Lab · Intermediate
The defect that was not the problem
Run a five-dimension assessment on a food security survey, quantify what every defect costs the headline, and discover that the largest single driver of the published figure is a threshold choice nobody wrote down.
A data quality assessment is supposed to tell a programme manager whether the published figure can stand. This lab does that on a lean-season food security survey, and the answer is not the one the exercise is designed to make you expect.
Five defects are in the file. Together they move the headline by less than a point. A single undocumented methodological choice moves it by a factor of eight. A DQA that reports the five and not the one has done the work and missed the finding.
The file
food-security-survey-2024.v1.csv — 2,112 synthetic household interviews across
four districts, carrying the raw components of the Food Consumption Score, the
Household Hunger Scale and the reduced Coping Strategies Index. The composite
indicators are not shipped; computing them is part of the work.
Read the dataset’s data quality notes before you start. They state the FCS weights and both threshold sets, and you will need all of it.
Set up first
An RStudio project with the file read-only and an outputs/ directory. The script
must run top to bottom from a clean session — you will run it twice, and the
second run is the one that counts.
Part one: the scorecard
Produce a five-dimension scorecard for this survey. Four of the five are computable from the file; the fifth is not, and saying so precisely is a finding in its own right.
- Completeness. Missing rate per field, per district. Note that blanks concentrate in three of the eight consumption groups.
- Consistency. Values above seven on a seven-day recall; partially administered HHS modules; households whose FCS and rCSI point in incompatible directions.
- Integrity. Digit preference across the consumption columns, and whether any district looks different from the others. Calibrate against chance before you write anything.
- Timeliness. State what you would need and whether the file carries it.
- Accuracy. State what a verification visit would recount and why you cannot do it from here.
Part two: what each defect costs
This is the part that makes it a DQA rather than a cleaning exercise. For each defect, compute the headline with and without the correction, and record the difference.
Compute the headline as the share of households with a poor and a borderline Food Consumption Score, and use the standard 21/35 thresholds for this part.
| Defect | Households affected | Effect on the headline |
|---|---|---|
| Consumption values above 7 days | ? | ? |
| Blank consumption cells treated as zero | ? | ? |
| Partial Household Hunger Scale modules | ? | ? |
| Duplicate enumerations | ? | ? |
| Head-of-household sex recorded two ways | ? | ? |
The last row will have no effect on the headline at all, and it is still a finding — say which analysis it does break.
Part three: the choice that dwarfs them
Now recompute the headline on the 28/42 threshold set, which is used where oil and sugar are consumed near-universally.
Put the two side by side with the defect table from part two. Then write three sentences for the programme manager on which of the two — the data defects or the threshold choice — determines the number in the report, and what has to be documented as a result.
What to hand in
An R script, sourced from a clean session, producing:
- the five-dimension scorecard, with the unmeasurable dimensions shown as findings
- the defect-cost table from part two, with every cell filled and a row count for each
- the headline computed on both threshold sets, with the analysed denominator stated
- a findings table in the five-column form the course uses — finding, root cause, corrective action, owner, deadline — with at least one root cause in each of three different categories
- everything written to
outputs/
Check your numbers
The dataset notes state what a correct analysis finds.
| Expected | |
|---|---|
| Poor food consumption, 21/35 thresholds | about 1% |
| Borderline, 21/35 thresholds | about 23% |
| Poor, 28/42 thresholds | about 7% |
| Borderline, 28/42 thresholds | about 39% |
| Correlation between FCS and rCSI | about -0.45 |
If your poor share on 21/35 is far from 1%, check the FCS weights before anything else. If your denominator is still 2,112 after handling the blanks, you have treated a missing consumption day as zero.
The questions to answer in prose
Three sentences each.
1. The defect table shows the data problems moving the headline by less than a point. Explain why that does not mean the defects are unimportant, and name one use of this survey where a defect of that size would matter.
2. The threshold choice moves the poor share from about 1% to about 7%. Neither set is wrong. Say what must therefore appear beside the figure in the report, and what a reader comparing this survey to last year’s is entitled to assume.
3. All 62 records using Male and Female come from one district. Say what category of root cause that pattern points to, what it rules out, and what the corrective action is.
How to know you are done
Delete outputs/, restart R, source() once, and every table is regenerated
identically. Every finding in your findings table has a number in it, a named
role as owner, and a date. None of them contains the word “training” unless you
can say what the person would be trained to do differently.
What this lab is not
It is not food security analysis. What the FCS measures, why the two threshold sets exist and how an IPC working group converges evidence belong with the food security content. This lab is the assessment: five dimensions with measures, a defect priced against the headline it moves, and a report that puts the largest driver of the number first even when the largest driver is not a defect at all.