Course · Beginner · Data Preparation
Data Cleaning and Validation
Missing values, duplicates, impossible measurements and inconsistent coding — how to find each one, decide what to do about it, and leave a record a data auditor can follow.
What you will be able to do
- Profile a fresh export before touching it, and keep the profile as the record of what the file looked like on arrival
- Show whether missingness clusters in one site or one week, and state what dropping incomplete rows would do to the ranking you publish
- Prove a table has the key you think it has, and find the duplicate records that share no key at all
- Write plausibility rules from the sector's own thresholds, and separate a recoverable unit error from an unrecoverable one
- Reconcile free-text site and village names against an administrative list without inventing a match
- Ship a cleaning log that states every rule, how many rows it touched, what it changed and who approved it
Standards and methodologies
Data Analysis Foundations showed you that routine data arrives with defects and gave you five checks for finding them. This course is the next question: what do you actually do about each one, and how do you prove it afterwards?
That is a bigger question than it sounds. Almost every cleaning decision moves a published number, most of them move it in a direction that is convenient, and almost none of them are recorded. A district ranking that changes when you drop incomplete rows, a caseload that falls by 2% when you deduplicate, a coverage figure that rises because eleven villages were spelled four ways and three of the spellings were dropped as unmatched — each of those is a defensible decision and an indefensible silence.
So the course is built around one discipline: every change is a rule, every rule is counted, and the count ships with the number. By the last unit you will have a validation script that runs on next quarter’s export without being edited and a cleaning log an auditor can read without asking you a question.
Every technique is shown in Python and in R. The examples run against platform datasets that carry the defects genuinely — the MUAC screening register from twelve communes in Artibonite, the WASH household survey with a district name written four ways, and the DHIS2-shaped vaccination extract where a facility that did not report and a facility that vaccinated nobody look identical.
You should have done Data Analysis Foundations, or be comfortable reading an export into a table and grouping it. You do not need statistics; the one course that needs it says so.
Practice
Reading the lessons is not the same as having done the work. Each of these applies the course to a dataset it did not teach on.
- Lab · 120 minThe finding the coding hidOne area recorded disability as Yes and No instead of true and false, splitting the equity finding into four categories and leaving a cell of three cases at 0%. Clean the caseload, defend the denominator, and write a log you could attach to a protection report.
- Exercise · 45 minSix changes and no logYou inherit a cleaning script with six undocumented steps and a coverage figure that has already been reported. Reconstruct the log, measure what each step cost, and identify the two steps that are wrong.
Progress
Take it offline
The whole course as a typeset PDF — every lesson, every code example, the data dictionary and the indicator definitions. Generated from the same source as this page.
The LaTeX source ships alongside each PDF, so an organisation can rebrand the handout or fold a lesson into its own training pack.