What an evaluator does to your numbers

The review is the test your indicators were built for

At some point somebody external — a mid-term review, an end-of-project evaluation, a donor’s monitoring adviser — takes your LogFrame, your reference sheets and your reported figures, and asks whether they support what the programme claims.

Most of what determines the answer was decided eighteen months earlier, in choices this course has already covered. This lesson is the checklist read backwards: what will be asked, and what you need to have already done.

The six criteria, and the question each asks of your data

The OECD DAC criteria are the vocabulary almost every evaluation in this sector uses.

Criterion	The question	What it needs from you
Relevance	Was this the right thing to do?	Needs assessment data, and indicators that measure the need, not only the activity
Coherence	Did it fit with everything else?	Comparable definitions — the standards lesson, cashed in
Effectiveness	Did it achieve its objectives?	Baseline, target, and an outcome indicator that is not an output
Efficiency	Was it a reasonable use of resources?	Reach or volume with a denominator, and cost data joined to it
Impact	What difference did it make overall?	A counterfactual, or an honest statement that you do not have one
Sustainability	Will the benefits last?	Something measured after handover, which almost nobody has

Read down the right-hand column and you can predict which criteria your programme will score badly on, before anyone visits.

Effectiveness: where the reference sheet pays for itself

This is the criterion evaluations spend the most time on, and it is answered almost entirely from the artefacts of unit 2.

An evaluator will ask, in this order:

1. What was the target, and where did it come from? The four methods from the previous lesson. “Negotiated” is an acceptable answer; a blank is not.
2. What was the baseline, measured how, and when? Same indicator, same method, before.
3. Is the reported value computed the same way as the baseline? Version the sheet or you cannot answer this.
4. What else could explain the change? The alternatives you listed when answering question two of lesson 2.

evidence = pd.DataFrame({
    "indicator": ["penta3_coverage_percent_monthly"],
    "baseline": [61.4],
    "baseline_period": ["2023 mean"],
    "target": [85.0],
    "target_basis": ["negotiated, assumed full outreach budget"],
    "latest": [72.8],
    "sheet_version": ["2.1"],
    "same_method": [True],
    "confounders": ["reporting completeness rose from 71% to 77% over the period"],
})

evidence <- tibble::tribble(
  ~indicator,                        ~baseline, ~target, ~latest, ~sheet_version, ~same_method,
  "penta3_coverage_percent_monthly",      61.4,    85.0,    72.8,          "2.1",         TRUE
)

The confounders field is the one that turns a defensive conversation into a professional one. Reporting completeness rising from 71% to 77% raises measured coverage without a single additional child being vaccinated. Saying that yourself, first, is worth more than any number in the row.

Impact: say what you cannot claim

The impact criterion asks what difference the programme made, which is a counterfactual question. Routine monitoring data cannot answer it, and pretending otherwise is the fastest way to lose an evaluator’s confidence in everything else you wrote.

What you can honestly offer:

• A before-and-after change, labelled as such, with the alternative explanations named.
• A comparison against a non-programme area, with an explicit statement of why the two are or are not comparable.
• Consistency with the theory of change — the intermediate steps moved in the order the theory predicted, which is evidence and not proof.

What you cannot offer without a design that supports it is attribution. Impact Evaluation Methods, later in the programme, is entirely about the designs that do — randomisation, difference-in-differences, matching, regression discontinuity — and the honest position until then is the one stated plainly.

Efficiency needs a denominator you probably do not have

Efficiency questions are usually cost per unit of result, and they fail on the join rather than on the arithmetic: financial data is by budget line and month, programme data is by activity and facility, and nothing connects them.

If efficiency is going to be assessed, decide the unit at design time and make the two systems share a key. Retrofitting it at evaluation produces a number nobody believes, including the person who computed it.

The four gaps you can only close beforehand

By the time the evaluator arrives, these are fixed. Every one of them is cheap eighteen months earlier and impossible on the day.

• No baseline on the same definition. Fixed by versioning the sheet at the start.
• No data on the assumptions. Fixed by giving the two or three critical assumptions their own indicators.
• No outcome indicator, only outputs. Fixed at LogFrame design, and lesson 1 is about spotting it.
• No post-handover measurement. Fixed by budgeting one follow-up round, which almost no proposal does and every sustainability section needs.

Build the evidence pack as you go

The artefact that makes a review straightforward is a folder, maintained continuously, not assembled in the fortnight before the visit.

evidence/
  indicators/                reference sheets, versioned
  baseline/                  values, method notes, extract date
  targets/                   values, basis, revision history
  series/                    every period, every indicator, one row each
  dqa/                       assessment rounds and follow-ups
  assumptions/               stock-outs, access days, vacancy rates
  limitations.md             what the data cannot support

Six of those seven come out of work this module already required. The reference sheets are unit 2, the DQA rounds are the previous course, the series is the analysis table from the joining course. The evidence pack is mostly a naming convention over things you already have.

The one-page summary an evaluator actually wants

summary = (
    evidence[["indicator", "baseline", "target", "latest", "same_method"]]
    .assign(progress=lambda d: (d["latest"] - d["baseline"])
                               / (d["target"] - d["baseline"]))
)
print(summary.round(2))

evidence |>
  mutate(progress = (latest - baseline) / (target - baseline)) |>
  select(indicator, baseline, target, latest, progress, same_method)

Progress against trajectory, per indicator, with a flag saying whether the method was constant. Anything more elaborate gets rebuilt by the evaluator anyway; anything less gets rebuilt by the evaluator from your raw files, with their assumptions rather than yours.

An evaluation is not an examination of your programme. It is an examination of whether your evidence supports what your programme said. Those come apart more often than anyone expects, and the reference sheet is where they are held together.

Where this course leaves you

You can trace an indicator from the change it is meant to evidence down to the arithmetic, write a reference sheet a stranger can compute from, defend a denominator, distinguish reach from coverage, adopt a standard definition and document your departure from it, set a baseline and a target that survive scrutiny, and assemble the evidence a review will ask for.

The rest of module 3 takes two specific measurement problems seriously. Survey Analysis, Sampling and Weighting is next and is the heaviest course in the programme so far — weights, strata, design effect and confidence intervals, which is what turns an estimate into an estimate with a margin. After it, Routine Data and DHIS2 goes back to the aggregate systems this course kept borrowing from, and answers the question that always follows a coverage figure: what exactly was counted?