Verification Checklist for AI-Assisted M&E

EvalCommunity Academy · AI Verification Series · Tutorial 1

How to Build a Verification Checklist for AI-Assisted M&E

Learn how to define clear pass/fail criteria for AI-generated outputs, including reports, indicators and data analysis, so every check is consistent, repeatable and defensible.

Level: Intermediate · Reading time: about 10 minutes · Updated: November 2026

Learning objectives

By the end of this tutorial you will be able to:

Explain why vague “check it” instructions fail and pass/fail criteria work.
Write verification criteria that are specific, observable and binary.
Assign severity levels and decision rules to approve, revise or reject an output.
Adapt ready-made checklists for reports, indicators and data analysis.
Pilot and refine a checklist for your own team or assignment.

1. Why you need pass/fail criteria

Most teams that use AI already tell staff to “check the output”. The problem is that “check” means something different to everyone. One person skims for typos, another reruns every calculation, a third trusts anything that looks professional. The result is inconsistent quality and no record of what was actually verified.

A verification checklist with pass/fail criteria fixes this. Each criterion states exactly what must be true for the output to pass, how to check it, and what happens if it fails. Two reviewers using the same checklist should reach the same verdict.

Vague instruction

“Check the figures are correct”
“Make sure quotes are accurate”
Result depends on who checks

Pass/fail criterion

“All headline figures reproduced from raw data within ±0.5 points”
“Every quote found verbatim in a named transcript”
Same verdict whoever checks

2. Anatomy of a good criterion

Every criterion in your checklist should have five parts. If any part is missing, reviewers will fill the gap with their own judgement, and consistency is lost.

1. Statement

What must be true, in one sentence. “Every quote is verbatim.”

2. Pass condition

The observable test that decides pass or fail. “Found word for word in the named transcript.”

3. Method and scope

How to check, and how much. “Search the transcript for each quote; check all quotes.”

4. Severity

How serious a failure is: critical, major or minor (see Section 4).

5. Owner

Who checks it. “Team member who did not run the AI task.”

The SOBER test for every criterion

Specific: names exactly what is checked.
Observable: can be confirmed by looking at evidence, not opinion.
Binary: the answer is pass or fail, not “mostly”.
Evidence-linked: points to the source used to check it.
Realistic: can be done in the time available.

3. Build your checklist in six steps

List your AI-assisted outputs and their stakes

Write down every output where AI plays a role: desk review summaries, indicator updates, coded interview data, report drafts. Mark each as low, medium or high stakes. High-stakes outputs (donor reports, evaluation findings, published figures) need the fullest checklist.

Identify how each output can fail

For each output, list the failure modes: fabricated quotes, wrong denominators, missing districts, overstated causal claims. Ask past reviewers what errors they have caught. You can also ask AI to brainstorm failure modes, then judge the list yourself.

I use AI to draft quarterly indicator summaries from our Indicator Tracking Table. List the ways this output could be wrong or misleading, grouped by source, calculation, completeness and interpretation.

Turn each failure mode into a pass/fail criterion

Flip each failure into a positive statement with a test. “Figures might be wrong” becomes “Headline figures reproduced from raw data within ±0.5 percentage points”. Apply the SOBER test to every line.

Failure mode: totals don’t match subgroups

Criterion: sum of disaggregated values equals the reported total for every indicator

Define method, scope and tolerance

For each criterion, say how to check it and how much to check. Decide whether you check all items or a sample (for example, 10 quotes or 20% of figures, whichever is larger). Where numbers are involved, set a tolerance for rounding so reviewers don’t argue over 0.1.

Assign severity and decision rules

Label each criterion critical, major or minor, and agree in advance what each result means for the output. Section 4 gives a ready-made set of rules.

Assign owners, pilot and refine

Name who checks each criterion, ideally someone other than the person who ran the AI task. Pilot the checklist on two or three real outputs. Where two reviewers disagree, rewrite the criterion until they don’t. Review the checklist at the end of each assignment or quarter.

4. Severity levels and decision rules

Severity turns a list of checks into a decision. Agree these rules before you start reviewing, not after you find a problem.

CRITICAL

The error could mislead a decision, misrepresent people or breach ethics. Examples: fabricated quote, wrong headline figure, identifiable data exposed.

MAJOR

The error weakens accuracy or completeness but does not reverse a conclusion. Examples: a missing caveat, one district omitted from a summary.

MINOR

Presentation or clarity issues. Examples: inconsistent rounding, unclear chart label, formatting.

Reject

Any critical criterion fails. The output is not used until corrected and fully re-checked.

Revise

No critical failures, but one or more major failures. Fix them and re-check those criteria only.

Approve

All critical and major criteria pass. Minor issues are fixed before release, without another full review.

5. Ready-made checklists to adapt

Use these as starting points. Adjust tolerances, sample sizes and owners to your context, and remove criteria that don’t apply.

A. Evaluation reports and report sections

R1. Every finding traces to evidence CRITICAL

Pass if each finding cites at least one source in the evaluation matrix or evidence log. Check: all findings.

R2. Quotes are verbatim and traceable CRITICAL

Pass if every quote is found word for word in a named transcript and used in context. Check: all quotes.

R3. References exist and say what is claimed CRITICAL

Pass if each cited document is located and supports the statement. Check: all references.

R4. Claims match the evaluation design MAJOR

Pass if no causal or attribution language (“caused”, “led to”) appears without a design that supports it. Check: search the text for causal terms.

R5. Negative findings and limitations included MAJOR

Pass if every negative or mixed result in the evidence log appears in the text. Check: compare with the evidence log.

R6. Conclusions follow from findings MAJOR

Pass if each conclusion links to named findings and each recommendation to a conclusion. Check: all conclusions.

B. Indicators and indicator reporting

I1. Values reproduced from source CRITICAL

Pass if each reported value matches an independent recalculation within ±0.5 points (or ±1% for counts). Check: all outcome indicators; 20% sample of output indicators.

I2. Definition matches the results framework CRITICAL

Pass if numerator, denominator and unit match the indicator reference sheet or logframe. Check: all indicators reported.

I3. Disaggregations reconcile MAJOR

Pass if subgroup values (sex, age, location) sum to the total, or the difference is explained. Check: all disaggregated indicators.

I4. Correct comparison with baseline and target MAJOR

Pass if the right baseline, target and period are used, and percentage points are not described as percent change. Check: all comparisons in the text.

I5. Anomalies explained MINOR

Pass if values over 150% of target or changes over 30% since the last period carry a short explanation. Check: all flagged values.

C. Data analysis (quantitative and qualitative)

D1. No identifiable data shared without permission CRITICAL

Pass if the data given to the AI tool contained no names, IDs, phone numbers or GPS points, or sharing was explicitly permitted. Check: the input file.

D2. Analysis is reproducible CRITICAL

Pass if a second person reproduces the key results from the same data using the documented steps. Check: headline results.

D3. Data cleaning decisions documented MAJOR

Pass if removed duplicates, outliers and missing values are listed with reasons. Check: the cleaning log.

D4. AI coding agrees with human coding MAJOR

Pass if a human coder independently codes a sample of transcripts and agrees with the AI’s codes in at least 80% of segments. Check: 10% of transcripts, minimum three.

D5. Charts represent the data fairly MINOR

Pass if axes start at zero for bar charts, scales are labelled, and sample sizes are shown. Check: all charts.

The thresholds above (±0.5 points, 80% agreement, 150% of target) are suggested starting points, not standards. Set your own with your team or client and record them in the checklist.

6. Worked example: an AI-drafted quarterly indicator summary

The output: a MEAL officer uses AI to draft a two-page Q3 summary from the Indicator Tracking Table. A colleague reviews it with checklist B.

I1 passes. All five outcome indicators match the recalculation within tolerance.

I2 passes. Definitions match the indicator reference sheets.

I3 fails (major). Female and male participants sum to 1,240, but the total shows 1,310. The AI included 70 records with sex not recorded without saying so.

I4 fails (major). The text says literacy “increased by 15%”, but it rose from 40% to 55%: 15 percentage points, a 37.5% relative increase.

I5 passes. The one value above 150% of target is explained.

Decision: Revise. No critical failures, two major ones. The officer adds a “sex not recorded” row, corrects the wording to “15 percentage points”, and the reviewer re-checks I3 and I4 only. Both pass, and the summary is approved.

7. Copy-ready checklist template

Paste this into a document or spreadsheet and fill one block per criterion. Keep the completed checklist with your verification log.

AI VERIFICATION CHECKLIST
Output: [name and version]
AI tool and task: [tool / what it did]
Stakes: [low / medium / high]
Reviewer: [name, not the AI operator] · Date: [date]
ID: [e.g. I1]
Criterion: [what must be true]
Pass if: [observable test and tolerance]
Method and scope: [how / all or sample size]
Severity: [critical / major / minor]
Result: [pass / fail] · Notes: [what was found]
DECISION: [approve / revise / reject]
Re-check completed: [date / reviewer]

8. Common mistakes to avoid

Too many criteria

A 60-item checklist won’t be used. Aim for 5 to 10 criteria per output type, focused on the errors that matter most.

Criteria that need opinion

“The analysis is robust” can’t be passed or failed. Rewrite it as something observable.

Checking your own AI output

The person who ran the task tends to see what they expect. Use a second reviewer for high-stakes outputs.

Deciding severity after the fact

If severity is set once an error is found, there is pressure to downgrade it. Fix severity in advance.

Using AI as the only verifier

AI can help pre-screen, but a criterion only passes when a person checks it against the original evidence.

9. Your action plan

Tick off each step as you build your first checklist.

Choose one high-stakes AI-assisted output to start with.
List at least five ways it could fail.
Write a pass/fail criterion for each, and apply the SOBER test.
Set method, scope, tolerance and severity for each criterion.
Agree the approve, revise and reject rules with your team.
Pilot it on two real outputs with two different reviewers.
Rewrite any criterion where the reviewers disagreed.

10. Knowledge check

Tap each question to reveal the answer.

1. Is “the analysis is robust” a good criterion?

No. It is not observable or binary. A better version: “a second analyst reproduces the headline results from the same data using the documented steps.”

2. What are the five parts of a criterion?

Statement, pass condition, method and scope, severity, and owner.

3. One critical criterion fails and everything else passes. What is the decision?

Reject. Any critical failure means the output is not used until it is corrected and fully re-checked.

4. Why set severity before reviewing?

So that nobody is tempted to downgrade an error once it has been found, especially close to a deadline.

5. Literacy rose from 40% to 55%. The AI wrote “increased by 15%”. Pass or fail?

Fail. It is a 15 percentage-point increase, which is a 37.5% relative increase. The text must say which it means.

6. Two reviewers reach different verdicts on the same criterion. What should you do?

Rewrite the criterion. Disagreement shows it is not specific or observable enough. A good criterion gives the same verdict whoever checks.

11. Key takeaways

“Check the output” is not a method. Pass/fail criteria make verification consistent and defensible.
Every criterion needs a statement, pass condition, method and scope, severity and owner.
Set severity and decision rules in advance: any critical failure means reject.
Keep checklists short, focused on the errors that matter, and checked by someone other than the AI operator.
Pilot, compare reviewers’ verdicts, and refine until the checklist gives the same answer whoever uses it.

Continue the AI Verification Series

Explore more EvalCommunity Academy tutorials on using AI responsibly in monitoring, evaluation, accountability and learning.

Explore EvalCommunity Academy

Thresholds and examples in this tutorial are illustrative starting points, not official standards. Adapt them to your organisation’s policies, your client’s requirements and the evaluation standards that apply to your work.