Back to the lesson·Lesson 8 of 8·Judgement
What an evaluator does to your numbers
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- The review is the test your indicators were built for
- The six criteria, and the question each asks of your data
- Effectiveness: where the reference sheet pays for itself
- Impact: say what you cannot claim
- Efficiency needs a denominator you probably do not have
- The four gaps you can only close beforehand
- Build the evidence pack as you go
- The one-page summary an evaluator actually wants
- Where this course leaves you
Speaker notes
Six OECD DAC criteria, the question each one asks of your indicator set, and the gaps you can only close before the evaluation rather than during it.The review is the test your indicators were built for
- At some point somebody external — a mid-term review, an end-of-project evaluation, a donor's monitoring adviser — takes your LogFrame, your reference sheets and your reported figures, and asks whether they support what the programme claims.
Speaker notes
At some point somebody external — a mid-term review, an end-of-project evaluation, a donor's monitoring adviser — takes your LogFrame, your reference sheets and your reported figures, and asks whether they support what the programme claims. Most of what determines the answer was decided eighteen months earlier, in choices this course has already covered. This lesson is the checklist read backwards: what will be asked, and what you need to have already done.The six criteria, and the question each asks of your data
Criterion The question What it needs from you Relevance Was this the right thing to do? Needs assessment data, and indicators that measure the need, not only the activity Coherence Did it fit with everything else? Comparable definitions — the standards lesson, cashed in Effectiveness Did it achieve its objectives? Baseline, target, and an outcome indicator that is not an output Efficiency Was it a reasonable use of resources? Reach or volume with a denominator, and cost data joined to it Impact What difference did it make overall? A counterfactual, or an honest statement that you do not have one Sustainability Will the benefits last? Something measured after handover, which almost nobody has Speaker notes
The OECD DAC criteria are the vocabulary almost every evaluation in this sector uses. Read down the right-hand column and you can predict which criteria your programme will score badly on, before anyone visits.Effectiveness: where the reference sheet pays for itself
- What was the target, and where did it come from? The four methods from the previous lesson. "Negotiated" is an…
- What was the baseline, measured how, and when? Same indicator, same method, before.
- Is the reported value computed the same way as the baseline? Version the sheet or you cannot answer this.
- What else could explain the change? The alternatives you listed when answering question two of lesson 2.
Speaker notes
This is the criterion evaluations spend the most time on, and it is answered almost entirely from the artefacts of unit 2. An evaluator will ask, in this order:Effectiveness: where the reference sheet pays for itself — In Python
evidence = pd.DataFrame({ "indicator": ["penta3_coverage_percent_monthly"], "baseline": [61.4], "baseline_period": ["2023 mean"], "target": [85.0], "target_basis": ["negotiated, assumed full outreach budget"], "latest": [72.8], "sheet_version": ["2.1"], "same_method": [True], "confounders": ["reporting completeness rose from 71% to 77% over the period"], })Effectiveness: where the reference sheet pays for itself — In R
evidence <- tibble::tribble( ~indicator, ~baseline, ~target, ~latest, ~sheet_version, ~same_method, "penta3_coverage_percent_monthly", 61.4, 85.0, 72.8, "2.1", TRUE )Speaker notes
Theconfoundersfield is the one that turns a defensive conversation into a professional one. Reporting completeness rising from 71% to 77% raises measured coverage without a single additional child being vaccinated. Saying that yourself, first, is worth more than any number in the row.Impact: say what you cannot claim
- A before-and-after change, labelled as such, with the alternative explanations named.
- A comparison against a non-programme area, with an explicit statement of why the two are or are not comparable.
- Consistency with the theory of change — the intermediate steps moved in the order the theory predicted, which is…
Speaker notes
The impact criterion asks what difference the programme made, which is a counterfactual question. Routine monitoring data cannot answer it, and pretending otherwise is the fastest way to lose an evaluator's confidence in everything else you wrote. What you can honestly offer: What you cannot offer without a design that supports it is attribution. Impact Evaluation Methods, later in the programme, is entirely about the designs that do — randomisation, difference-in-differences, matching, regression discontinuity — and the honest position until then is the one stated plainly.Efficiency needs a denominator you probably do not have
- Efficiency questions are usually cost per unit of result, and they fail on the join rather than on the arithmetic: financial data is by budget line and month, programme data is by activity and facility, and nothing connects them.
Speaker notes
Efficiency questions are usually cost per unit of result, and they fail on the join rather than on the arithmetic: financial data is by budget line and month, programme data is by activity and facility, and nothing connects them. If efficiency is going to be assessed, decide the unit at design time and make the two systems share a key. Retrofitting it at evaluation produces a number nobody believes, including the person who computed it.The four gaps you can only close beforehand
- No baseline on the same definition. Fixed by versioning the sheet at the start.
- No data on the assumptions. Fixed by giving the two or three critical assumptions their own indicators.
- No outcome indicator, only outputs. Fixed at LogFrame design, and lesson 1 is about spotting it.
- No post-handover measurement. Fixed by budgeting one follow-up round, which almost no proposal does and every…
Speaker notes
By the time the evaluator arrives, these are fixed. Every one of them is cheap eighteen months earlier and impossible on the day.Build the evidence pack as you go — Example
evidence/ indicators/ reference sheets, versioned baseline/ values, method notes, extract date targets/ values, basis, revision history series/ every period, every indicator, one row each dqa/ assessment rounds and follow-ups assumptions/ stock-outs, access days, vacancy rates limitations.md what the data cannot supportSpeaker notes
The artefact that makes a review straightforward is a folder, maintained continuously, not assembled in the fortnight before the visit. Six of those seven come out of work this module already required. The reference sheets are unit 2, the DQA rounds are the previous course, the series is the analysis table from the joining course. The evidence pack is mostly a naming convention over things you already have.The one-page summary an evaluator actually wants — In Python
summary = ( evidence[["indicator", "baseline", "target", "latest", "same_method"]] .assign(progress=lambda d: (d["latest"] - d["baseline"]) / (d["target"] - d["baseline"])) ) print(summary.round(2))The one-page summary an evaluator actually wants — In R
evidence |> mutate(progress = (latest - baseline) / (target - baseline)) |> select(indicator, baseline, target, latest, progress, same_method)The one-page summary an evaluator actually wants
An evaluation is not an examination of your programme. It is an examination of whether your evidence supports what your programme said. Those come apart more often than anyone expects, and the reference sheet is where they are held together.
Speaker notes
Progress against trajectory, per indicator, with a flag saying whether the method was constant. Anything more elaborate gets rebuilt by the evaluator anyway; anything less gets rebuilt by the evaluator from your raw files, with their assumptions rather than yours.Where this course leaves you
- You can trace an indicator from the change it is meant to evidence down to the arithmetic, write a reference sheet a stranger can compute from, defend a denominator, distinguish reach from coverage, adopt a standard definition and document your departure from it, set a baseline and a target that survive scrutiny, and assemble the evidence a review will ask for.
Speaker notes
You can trace an indicator from the change it is meant to evidence down to the arithmetic, write a reference sheet a stranger can compute from, defend a denominator, distinguish reach from coverage, adopt a standard definition and document your departure from it, set a baseline and a target that survive scrutiny, and assemble the evidence a review will ask for. The rest of module 3 takes two specific measurement problems seriously. Survey Analysis, Sampling and Weighting is next and is the heaviest course in the programme so far — weights, strata, design effect and confidence intervals, which is what turns an estimate into an estimate with a margin. After it, Routine Data and DHIS2 goes back to the aggregate systems this course kept borrowing from, and answers the question that always follows a coverage figure: what exactly was counted?