Mission goal: understand what the AI-enhanced version of your role is and why it is emerging.
Step 1 of 5Where are you starting from?
Choose your starting point+10 XP
Where are you today with AI in your evaluation work?
Show answer
Every starting point is valid. Follow the mission suggested for your situation.
Step 2 of 5Meet your new role
Today's role
Evaluation Specialist
AI-enhanced role
AI-Assisted Evaluation Specialist
Level
Specialist · 3–6 years
How the role uses AI
Uses AI: for evaluation design, evidence synthesis, data analysis, report drafting and interpretation support, and checks every output
"AI-Assisted Evaluation Specialist" is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy.
Step 3 of 5What changes most in your job?
Your guess+10 XP
AI speeds up document review, transcription, first-pass coding and drafting. What becomes more important in your work?
Show answer
In the profile, AI speeds up the slowest parts of evaluation work, so the specialist can cover more evidence in the same time. The specialist's judgement on design, triangulation and the strength of evidence becomes more important, because AI can produce fluent conclusions that the data does not support.
Step 4 of 5AI-Enabled, not AI-Dependent
The goal is not to hand your work to AI. It is to become a stronger evaluator who uses AI and stays in charge of it. Here are the five questions, asked about your own work:
1QualityDoes AI make your findings stronger, or only faster?Every AI-drafted finding is checked against the evidence table before it goes into the report.
2AccountabilityWhen a finding in your report is wrong, can you show where it came from?Every finding links to its sources, and your name, not the AI's, is on the report.
3IndependenceDo you still weigh the evidence yourself?You judge the strength of evidence, and remove conclusions that rest on a single interview.
4EthicsAre respondents protected when their words go into AI tools?Consent covers AI processing, transcripts are anonymised, and AI use is disclosed in the methodology.
5Professional developmentAre you getting better at evaluation, or only at prompting?You code a sample yourself and read key studies in full, so your own judgement keeps growing.
The five questions as the series states them
Quality: Does AI improve the quality and credibility of evaluation findings?
Accountability: Who is responsible when AI produces an incorrect interpretation?
Independence: Are evaluators maintaining their ability to think critically and make independent judgments?
Ethics: Are confidentiality, human rights and the interests of affected populations protected?
Professional development: Are we developing stronger evaluators who understand AI, or creating professionals who increasingly depend on it?
Step 5 of 5Your goal for this tutorial
By the end, you will know how to use AI at each stage of an evaluation and check every output against the evidence, which skills to build, and you will have a 12-week plan.
Next: see which of your tasks change, and how.
End of Mission 01
MISSION02
Traditional role vs AI-enhanced role
Before and after · Badge: Role Shifter
8 challenges
Mission goal: see which responsibilities become AI-assisted, which need new skills and which stay human.
Step 1 of 4Meet your case study
Case study · Learning Grants Programme evaluation Illustrative
A fictional evaluation you will follow through missions 2, 3, 4, 8 and 9.
The evaluation
Final evaluation of a programme of cash grants to households with school-age children (2022–2024): 40 days, a team of three.
The data
60 programme documents, an endline survey of 1,200 households, school records, 30 key informant interviews and 8 focus groups.
The tools
Claude, ChatGPT, Whisper, NVivo, Elicit, Zotero, Stata, KoboToolbox and Excel, with the OECD-DAC criteria.
Your objective
Answer the evaluation questions with findings the evidence supports, on time, with every AI step checked and disclosed.
Step 2 of 4Flip your tasks: today and with AI
Less reading, typing and transcribing, more weighing and checking. Tap each card to see how it changes.
Step 3 of 4Mini-game: sort the work
For each task, choose: AI-assisted, a new skill for you, or stays human.
Sort it+10 XP
Summarising programme documents during inception
Show answer
AI-assisted. With every point checked against its document.
Sort it+10 XP
Transcribing interviews
Show answer
AI-assisted. Whisper transcribes; you check a sample against the audio.
Sort it+10 XP
Finding comparable evaluations and studies
Show answer
AI-assisted. AI-supported literature search is one of the profile's AI tasks.
Sort it+10 XP
Reviewing AI coding of interviews against the codebook
Show answer
New skill. AI-assisted qualitative analysis is the profile's first learning priority.
Sort it+10 XP
Structuring conclusions with contribution analysis
Show answer
New skill. Theory-based evaluation is a learning priority in the profile.
Sort it+10 XP
Choosing a design that fits the questions, budget and context
Show answer
Human. Listed in the profile as human judgement.
Sort it+10 XP
Weighing conflicting evidence and judging its strength
Show answer
Human. Listed in the profile as human judgement.
Sort it+10 XP
Protecting respondents and keeping the evaluation independent
Show answer
Human. Listed in the profile as human judgement.
Step 4 of 4Your turn: two minutes
Think about your last evaluation. Which tasks would go in each pile?
Think about it for a moment, then reveal.
A quick way to do it: three columns on a sheet of paper, AI-assisted, new skill, stays human. Keep the list: you will use it in mission 4 (to pick a task to try) and mission 6 (to rate your gaps).
End of Mission 02
MISSION03
What AI actually changes in your daily work
Checks, risks and red flags · Badge: Red Flag Spotter
2 challenges
Mission goal: learn the three checks every AI output in an evaluation needs, and practise spotting problems in an AI draft.
Step 1 of 4What can AI really do for you?
Before you look: which of your tasks could AI help with, and which must stay with you?
Think about it for a moment, then reveal.
AI can assist with
Summarising programme documents
Drafting matrices and data collection tools
First-pass coding of interviews
Drafting report sections
Can be partly automated
Transcription and translation
Literature search
Reference management
Formatting evidence tables
New responsibilities
Checking transcripts against the audio
Reviewing AI coding and recording agreement
Reading cited studies in full
Disclosing AI use in the methodology
Stays with you
The evaluation design
Weighing conflicting evidence
Fair conclusions and recommendations
Protecting respondents and independence
Step 2 of 4The three checks you must always build in
AI drafts sound convincing even when the evidence is thin. Here are the checks, one at a time.
Check against the source. Every summary point, code and citation is checked against the document, transcript or paper it comes from.
Weigh the evidence yourself. You judge the strength of each finding and triangulate. No conclusion rests on a single interview.
Protect respondents and disclose. Consent covers AI processing, transcripts are anonymised, and the methodology says how AI was used.
Step 3 of 4Mini-game: spot the red flags
Case study · Learning Grants Programme evaluation Illustrative
An AI assistant summarised the focus group notes for the findings chapter. Here is the source and the draft.
Click every sentence you would not put in the report.
Focus group notes, 8 groups
Groups saying attendance improved
6 of 8
Groups mentioning late grant payments
5 of 8
Women-only groups
4
Group 3
Girls still miss school during the harvest
Red flags found: 0 of 3+30 XP
Show answer
Three sentences should not go in: "all communities agree the grant transformed education" (six of eight groups, and an overstatement), the claim that late payments made families take children out of school (not in the notes), and the comparison of women's and men's groups (not in the notes).
Step 4 of 4Mission check
Your call+20 XP
An AI assistant coded 30 interviews overnight. What happens before you use the codes in your findings?
Show answer
Review every passage for the key questions and a sample of the rest, and record how often you agreed.
End of Mission 03
MISSION04
Your daily tasks, step by step with AI
Six practical guides · Badge: Workflow Builder
6 challenges
Mission goal: learn how to use AI at each stage of an evaluation, with a human check at every step.
Case study · Learning Grants Programme evaluation Illustrative
Every worked example below uses this evaluation's data, so you can follow one evaluation from inception to the report.
Six real tasks from an evaluation. Pick one and go through it in six short steps: a task prompt, then a QA prompt that checks the AI's work.
YouAIYour teamAutomated shows who does each step. Examples are illustrative.
Choose a task
Guide 1 of 6 From the career profile
Review programme documents for the inception report
From the profile: the specialist gives Claude 60 programme documents and asks for a summary of activities, changes to targets and reported results by year; each point is checked against the cited document before it goes into the inception report.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
You read 60 documents over two weeks.
Changes to targets get missed.
Notes don't record which document said what.
How it works with AI
Claude summarises activities, target changes and results by year, citing each document.
You check every point against the page it cites.
Sources sit in Zotero, so anyone can trace them.
Step 2 of 6Step by step
YouCollect and label the documentsZotero
Store them in Zotero with a short ID each (D01 to D60). Remove personal data first.
AIAsk Claude for a summary by yearClaude
Activities, target changes and reported results, with document and page for each point (prompt below).
YouCheck every point against its page
Open the document. Is the point there, and does it say that?
YouList conflicts between documents
For example, two different targets for the same year.
Your teamAsk the programme team about gaps
What is missing or contradictory?
YouWrite the inception section
Only checked points, each with its document ID and page.
Step 3 of 6Worked example
What you give the AI Illustrative
ID
Document
Year
D04
Annual report
2022
D11
Revised logframe
2023
D17
Annual report
2023
D29
Donor review
2024
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
From the programme documents attached, summarise by year: activities carried out, changes to targets, and reported results.
For every point, give the document ID and page.
If two documents disagree, list both and say so. Do not resolve the conflict.
Do not add any figure that is not in a document.
[attach documents with IDs]
What comes back
Year
Point
Source
2022
Grants paid to 4,000 households
D04, p. 6
2023
Target raised from 5,000 to 6,500 households
D11, p. 2
2023
6,100 households reached
D17, p. 9
2024
Reviewers note a gap in girls' attendance data
D29, p. 14
Four points, each traceable to a page. You open every one before it goes into the inception report.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Year
Point
Source
2023
Target of 6,500 households; 6,100 reached (94%)
D17
2023
Attendance rose by 12%
D17, p. 11
2024
The programme is on track to meet all targets
D29
The QA prompt (run it on every draft)
Act as an independent reviewer of an inception document summary.
Compare each point with the document and page it cites.
List: every point the cited page does not support, every figure that appears in no document, every point that contradicts what a document says, and every point with no page reference.
Do not rewrite the summary. List the problems only.
Documents: [attach]
Summary: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
The 6,500 target is in D11, not D17: the citation is wrong.
D17, p. 11 gives no 12% rise in attendance, and no other document does.
D29 notes a gap in girls' attendance data; it does not say the programme is on track.
Two points have no page reference.
Corrected output
Year
Point
Source
2023
Target raised from 5,000 to 6,500 households
D11, p. 2
2023
6,100 households reached
D17, p. 9
2024
Reviewers note a gap in girls' attendance data
D29, p. 14
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Accountability: every point in the inception summary is checked against the document and page it cites.
Step 6 of 6Quick check
Quick check+25 XP
Claude's summary says attendance rose by 12%, citing D17. D17 doesn't contain that figure. What do you do?
Show answer
Remove it, and re-check everything else cited to that document.
Try it this week
Take five documents from your current evaluation. Ask an AI assistant for a summary with document and page for each point, then check every point.
Draft the evaluation matrix from the terms of reference
From the profile: using an AI assistant to draft an evaluation matrix from the terms of reference and then revising its sub-questions; developing evaluation questions, matrices and designs with the OECD-DAC criteria.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
The matrix is built from a blank page.
Sub-questions copy the ToR without making them answerable.
Data sources are listed, but not linked to questions.
How it works with AI
AI drafts a matrix from the ToR, by OECD-DAC criterion.
You revise the sub-questions so the evidence can answer them.
You decide the design, the sources and the methods.
Step 2 of 6Step by step
YouRead the ToR and decide the design
It must fit the questions, the budget and the context. That stays with you.
AIAsk AI to draft the matrixClaude · ChatGPT
Criteria, questions, sub-questions, judgement criteria, sources and methods (prompt below).
YouRevise every sub-question
Can your data answer it? Is it fair to stakeholders?
YouCheck sources and methods
Available, feasible within the budget, and ethical?
Your teamReview it with the evaluation manager
Agree it before you design the tools.
YouNote how AI was used
One line in the inception report's methodology.
Step 3 of 6Worked example
What you give the AI Illustrative
Terms of reference (excerpt): Final evaluation of the Learning Grants Programme: cash grants to households with school-age children, 2022–2024. Questions: To what extent did the grants improve school attendance, especially for girls? How sustainable are the results after the grants end? Budget: 40 days. Data: endline survey, monitoring data, school records, KIIs and FGDs.
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Draft an evaluation matrix from the terms of reference below.
Use the OECD-DAC criteria the ToR names. For each evaluation question, give 2–3 sub-questions, judgement criteria, data sources and methods.
Use only the data sources the ToR lists. Mark any sub-question the listed data may not answer with [CHECK].
Do not choose the evaluation design.
[paste ToR]
What comes back
Criterion
Sub-question
Data sources
Effectiveness
Did attendance change for girls and boys in grant households?
Endline survey; school records
Effectiveness
Which factors helped or hindered attendance?
KIIs; FGDs
Sustainability
Do households expect to keep children in school after the grants end? [CHECK]
FGDs; endline survey
The [CHECK] is yours to resolve: expectations are not results.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Criterion
Sub-question
Data sources
Effectiveness
Did the grants cause a 20% rise in attendance?
Endline survey
Impact
What was the effect on household income?
Randomised comparison
Sustainability
Are the results sustainable?
None given
The QA prompt (run it on every draft)
Act as an independent reviewer of a draft evaluation matrix.
Compare the matrix with the terms of reference.
List: every sub-question that assumes its answer, every data source or method the ToR does not provide or the budget cannot cover, every sub-question too vague to answer, and every ToR question that is not covered.
Do not rewrite the matrix. List the problems only.
ToR: [paste]
Matrix: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"Did the grants cause a 20% rise" assumes the answer, and a figure nobody has.
A randomised comparison is not in the ToR's data or the 40-day budget.
"Are the results sustainable?" is too vague and has no data source.
Girls' attendance, named in the ToR, is not asked about separately.
Corrected output
Criterion
Sub-question
Data sources
Effectiveness
How did attendance change for girls and for boys in grant households, compared with the baseline?
Endline survey; school records
Sustainability
Which households expect to keep children in school after the grants end, and why?
FGDs; endline survey
Sustainability
Which school practices continue without the grants?
KIIs; school records
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Independence: you choose the design and revise every sub-question. The AI drafts; it does not decide what the evaluation asks.
Step 6 of 6Quick check
Quick check+25 XP
The AI's matrix proposes a randomised comparison. The ToR has no comparison group and 40 days. What do you do?
Show answer
Drop it. The design is your call, and it must fit the questions, budget and context.
Try it this week
Take the ToR of your next evaluation. Ask an AI assistant for a draft matrix, then rewrite three sub-questions so your data can answer them.
Transcribe and code interviews, then review the coding
From the profile: Whisper transcribes 30 KIIs, an AI assistant applies the evaluation codebook, and the specialist reviews all coded passages for the effectiveness questions and a sample of the rest; reviewing AI first-pass coding of 25 KII transcripts against the codebook is a daily task.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Transcription takes weeks, or is outsourced.
Coding 30 interviews by hand delays the analysis.
Nobody records how coding decisions were made.
How it works with AI
Whisper transcribes; you check a sample against the audio.
AI applies your codebook as a first pass.
You review every passage for the key questions and a sample of the rest.
Step 2 of 6Step by step
YouCheck that consent covers AI processing
Respondents agreed to recording and to how it is processed. If not, don't upload.
AutomatedTranscribe with WhisperWhisper
Run it on a secure computer, not a consumer upload.
YouCheck a sample against the audio
For example, five minutes from each of five interviews. Fix names and numbers.
YouAnonymise the transcripts
Replace names, places and roles that identify people before any AI coding.
AIApply the codebook as a first passChatGPT · NVivo
With the passage and a reason for each code (prompt below).
YouReview and recodeNVivo
Every passage for the effectiveness questions; a sample of the rest. Record how often you agree.
Step 3 of 6Worked example
What you give the AI Illustrative
Code
Definition
EFF-ATT
Change in attendance linked to the grant
EFF-BAR
Barriers to attendance that remain
SUS-EXP
Expectations about after the grant ends
HH-USE
How households used the grant
KII 07, passage 12: "Since the grant, my daughter goes every day. Before, she stayed home when we had no money for the uniform."
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Apply the codebook below to the interview passages.
For each passage, give the code or codes, and a one-line reason quoting the words that justify it.
If no code fits, write NONE. If you are unsure, write UNSURE.
Do not create new codes or summarise the interview.
Codebook: [paste]
Passages: [paste, anonymised]
What comes back
Passage
Codes
Reason
KII 07, 12
EFF-ATT; EFF-BAR
"goes every day"; "no money for the uniform"
KII 07, 13
HH-USE
"we bought the uniform and shoes"
KII 07, 14
UNSURE
Mentions the school feeding programme
The UNSURE passage is yours to code. So is EFF-BAR in passage 12: the barrier was before the grant, so it does not remain.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Passage
AI code
KII 03, 5: "The grant came late, so we borrowed."
EFF-ATT
KII 11, 2: "Teachers are often absent."
EFF-TEACH (new code)
KII 07, 12 (the passage above)
EFF-ATT; EFF-BAR
AI note: 28 of 30 interviews show strong positive effects on attendance.
The QA prompt (run it on every draft)
Act as an independent reviewer of AI coding of interview passages.
Compare each code with the codebook definition and the passage.
List: every code whose definition the passage does not meet, every code that is not in the codebook, every barrier coded as remaining when the passage places it in the past, and every summary claim the coding does not show.
Do not recode. List the problems only.
Codebook: [paste]
Coded passages: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
KII 03, 5 is about late payment and borrowing, not attendance: HH-USE, not EFF-ATT.
EFF-TEACH is not in the codebook. Whether the codebook needs a new code is your decision.
In KII 07, 12 the uniform barrier was before the grant, so EFF-BAR (barriers that remain) does not apply.
"28 of 30 show strong positive effects" is unchecked: coding counts passages, not the strength of an effect.
Corrected output
Passage
Reviewed code
KII 03, 5
HH-USE (late payment and borrowing noted)
KII 11, 2
EFF-BAR: teacher absence is a remaining barrier
KII 07, 12
EFF-ATT
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Ethics: consent covers AI processing, and transcripts are anonymised before any AI coding.
Step 6 of 6Quick check
Quick check+25 XP
The AI applies a code that isn't in your codebook. What do you do?
Show answer
Remove it, and decide yourself whether the codebook needs a new code.
Try it this week
Code five passages yourself, then ask an AI assistant to code the same five with your codebook. Where do you disagree, and who is right?
Run a rapid evidence review with AI-assisted search
From the profile: Elicit finds studies on cash transfer effects on school attendance; the specialist screens the 40 returned abstracts and reads the eight most relevant papers before citing them.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Literature searches take longer than the evaluation allows.
Reviews cite what is easy to find, not what is relevant.
Studies are cited from their abstracts.
How it works with AI
Elicit finds candidate studies and summarises their abstracts.
You screen every abstract against your criteria.
You read the most relevant papers in full before citing them.
Step 2 of 6Step by step
YouWrite the question and the criteria
Population, intervention, outcome, context and years. Decide before you search.
AISearch with ElicitElicit
Using your question (prompt below).
YouScreen every abstractZotero
Include or exclude against your criteria, with a reason.
YouRead the most relevant in full
Check the AI's summary against the paper: setting, design, results.
YouJudge the strength of each study
Design, sample, and how similar the context is.
YouCite only what you readZotero
With the page or table for each claim.
Step 3 of 6Worked example
What you give the AI Illustrative
Question: What effects have cash transfers had on school attendance, especially for girls, in low- and middle-income countries? Criteria: evaluations with a comparison group; attendance or enrolment outcomes; published 2010 or later.
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Search question for Elicit:
What are the effects of cash transfers on school attendance, for girls and for boys, in low- and middle-income countries?
For each result, extract: country, design, comparison group (yes or no), outcome measured, and the main result in the authors' words.
What comes back
Step
Studies
Abstracts returned
40
Excluded at screening (no comparison group, other outcome, before 2010)
28
Full text checked
12
Read in full and judged most relevant
8
Only the eight papers you read go into the report.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Evidence summary (AI): Cash transfers consistently increase school attendance by 10–15%, with larger effects for girls (Studies A, B and C). These findings apply directly to the Learning Grants Programme.
The QA prompt (run it on every draft)
Act as an independent reviewer of an evidence summary.
Compare the summary with the papers it cites (full text, not abstracts).
List: every claim a cited paper does not make, every range or average no paper reports, every context difference the summary ignores, and every word such as "consistently" that the evidence does not support.
Do not rewrite the summary. List the problems only.
Papers: [attach]
Summary: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
No paper reports a 10–15% range.
Study C found no significant effect for girls: "larger effects for girls" misreports it.
"Consistently" overstates three studies with different results.
"Apply directly" ignores context: the studies looked at conditional transfers; this programme's grants are unconditional.
Corrected output
Of eight studies read in full, five found increases in attendance (Studies A, B, D, F and G), two found no significant effect for girls (Studies C and E), and one found mixed results. Most studied conditional transfers, unlike this programme, so the evidence is indicative, not directly comparable.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Quality: you read every cited study in full. An AI summary of an abstract is not evidence.
Step 6 of 6Quick check
Quick check+25 XP
Elicit's summary of a study fits your argument well. You haven't read the paper. Can you cite it?
Show answer
Read it in full first, and check the AI's summary against the paper.
Try it this week
Pick one question from your current evaluation. Run it in Elicit, screen the first 20 abstracts, and read the two most relevant papers in full.
Interpret an unexpected result with AI, and test the explanation
From the profile: running a regression on endline survey data and asking AI to explain an unexpected coefficient, then checking it.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
An odd result is reported without explanation, or quietly dropped.
Explanations come from intuition.
Nobody checks whether the model itself is right.
How it works with AI
AI suggests possible explanations, each with a test.
You check the model first, then run the tests.
You report what the data supports, including what stays uncertain.
Step 2 of 6Step by step
YouRun the model you plannedStata
As set out in the evaluation matrix.
AIAsk AI for possible explanationsClaude
Several, each with a check in the data (prompt below).
YouCheck the model firstStata
Variable coding, sample, missing data. Many surprises are errors.
YouRun the checksStata
Keep only the explanations the data supports.
Your teamAsk people who know the context
Field staff, or participants in the validation workshop.
YouReport it honestly
What you found, what explains it and what remains uncertain.
Step 3 of 6Worked example
What you give the AI Illustrative
Endline survey, 1,200 households. Outcome: days attended in the last month. The grant coefficient is positive overall, but negative for girls aged 14–16 (−1.8 days, p = 0.04).
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Here is a regression result from an evaluation, with the model and the variable definitions.
The coefficient for girls aged 14–16 is negative, which we did not expect.
List possible explanations, including data or model errors. For each, say what check in the data would support it or rule it out.
Do not choose an explanation and do not write a finding.
[paste model, variable definitions and output]
What comes back
Coding error: are the age and sex variables coded as intended? Check: tabulate both against the raw data.
Selection: did older girls who had dropped out re-enrol with the grant, and attend less? Check: attendance by previous enrolment.
Survey timing: was the endline during exams or the harvest? Check: survey dates by district.
A real effect, for example more work at home for older girls. Check: ask in FGDs and the validation workshop.
Four explanations to test. None of them is a finding yet.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
AI explanation: The negative coefficient shows that cash grants push older girls into early marriage, which reduces attendance. This is a well-documented effect.
The QA prompt (run it on every draft)
Act as an independent reviewer of an explanation of a statistical result.
Compare the explanation with the data, the model and any evidence cited.
List: every claim the data in this evaluation cannot show, every source claimed but not given, and every simpler explanation (including errors) that was not ruled out first.
Do not rewrite the explanation. List the problems only.
Model and output: [paste]
Explanation: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
The survey has no data on marriage, so this evaluation cannot show it.
"Well-documented" cites no source.
Coding errors, selection and survey timing were not ruled out first.
A sensitive claim about girls, made without evidence, could harm the people the evaluation is meant to serve.
Corrected output
Checks run: the age and sex variables were coded correctly. Girls aged 14–16 who re-enrolled because of the grant attended fewer days than girls already enrolled. Controlling for previous enrolment, the coefficient is −0.4 days (p = 0.51), not significant.
Finding: no clear evidence that the grant reduced attendance for older girls; the re-enrolment of girls who had dropped out explains most of the difference. To be discussed in the validation workshop.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Independence: you test the AI's explanation against the data. A fluent explanation is not a correct one.
Step 6 of 6Quick check
Quick check+25 XP
The AI explains the result with a cause your survey didn't measure. What do you do?
Show answer
Treat it as a hypothesis, test what you can, and report only what the data supports.
Try it this week
Take one surprising result from a recent analysis. Ask an AI assistant for explanations with checks, then run the first two checks.
From the profile: the specialist drafts the sustainability section with an AI assistant using the evidence table as input, then removes two conclusions that rest on a single interview; writing findings that link each finding to its evidence sources; disclosing AI use in the methodology section.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Findings are written from memory of the fieldwork.
Strong claims rest on one quote.
Readers can't see what evidence supports each finding.
How it works with AI
AI drafts from your evidence table, finding by finding.
You check every finding against its sources.
You remove conclusions that rest on a single source, and disclose AI use.
Step 2 of 6Step by step
YouBuild the evidence tableExcel
For each finding: the sources (survey, KIIs, FGDs, documents) and how many.
AIAsk AI to draft the sectionClaude
Only from the evidence table, citing sources (prompt below).
YouCheck every finding against the table
Is it there, and is its strength described correctly?
YouRemove what rests on a single source
Or present it as a single view, clearly labelled.
Your teamValidate with stakeholders
Present the findings in a validation workshop.
YouDisclose AI use in the methodology
What AI did, and how it was checked.
Step 3 of 6Worked example
What you give the AI Illustrative
Finding (sustainability)
Sources
Most households expect to keep children in school after the grants end
Endline survey (68%); 6 of 8 FGDs
Schools kept the attendance tracking system
School records (4 of 5 schools); 3 KIIs
Savings groups will replace the grant
1 KII (district officer)
Local government will fund grants from 2026
1 KII (district officer)
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Draft the sustainability section of an evaluation report using only the evidence table below.
For each finding, state it, cite its sources and describe the strength of the evidence (how many sources, and what kinds).
Where a finding rests on a single source, say so; do not present it as a conclusion.
Do not add evidence, causes or recommendations that are not in the table.
[paste evidence table]
What comes back
Most households expect to keep their children in school after the grants end (endline survey, 68%; six of eight FGDs).
Four of five schools still use the attendance tracking system (school records; three KIIs).
One district officer expects savings groups to replace the grant and local government to fund grants from 2026. These are single views, not findings.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
The programme is highly sustainable: savings groups will replace the grant, and local government has committed to funding grants from 2026. 68% of households will keep their children in school.
The QA prompt (run it on every draft)
Act as an independent reviewer of a draft evaluation report section.
Compare every sentence with the evidence table.
List: every conclusion that rests on a single source, every expectation presented as a fact, every summary word (such as "highly") the evidence does not support, and every finding without a cited source.
Do not rewrite the section. List the problems only.
Evidence table: [paste]
Draft: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
Savings groups replacing the grant rests on one KII.
"Local government has committed" turns one officer's expectation into a commitment.
"68% will keep their children in school" turns an expectation into a fact.
"Highly sustainable" is not supported by the table, and no sources are cited.
Corrected output
Most households expect to keep their children in school after the grants end (endline survey, 68%; six of eight FGDs). Four of five schools still use the attendance tracking system (school records; three KIIs).
One district officer expects savings groups to replace the grant and local government to fund grants from 2026; these are single views, not findings. Methodology note: an AI assistant drafted this section from the evidence table; the evaluator checked each finding against its sources.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Accountability: every finding links to its evidence, and conclusions that rest on a single interview come out.
Step 6 of 6Quick check
Quick check+25 XP
A strong conclusion in the AI draft rests on one interview with a district officer. What do you do?
Show answer
Remove it as a conclusion, or report it as one person's view.
Try it this week
Take one findings section you've written. Build a quick evidence table for it, and mark any finding that rests on a single source.
Pick a recurring task that is not covered above, such as designing a focus group guide or preparing a validation workshop, and use this prompt to draft your own step-by-step guide.
I am an evaluation specialist. Here is a task I do in most evaluations: [describe the task, when, who is involved, which tools and data].
Break it into numbered steps. For each step, say who should do it (me, the evaluation team, the commissioner, an automated tool or an AI assistant), which tool, and what could go wrong.
Mark every step where a person must check the result before it is used, and say what they should check.
Do not suggest putting personal data or identifiable interview content into an AI tool.
End of Mission 04
MISSION05
The skills you need to develop
Skill inventory · Badge: Skill Builder
1 challenge
Mission goal: find out which of the 14 profile skills you already have and which need new learning.
Step 1 of 5Guess first
Your guess+20 XP
Which skill is the first learning priority for this role in the career profile?
Show answer
AI-assisted qualitative analysis, followed by theory-based evaluation, Elicit, research ethics and critical appraisal.
Step 2 of 5Skills you bring with you
Tick the ones you already have. Open a skill to see how to build it.
Your skills match
0 of 14
A. Existing evaluation competencies
Transfers from your role
Questions, matrices and designs that fit the context. AI can draft; you decide.
Build it: Revise one AI-drafted matrix (guide 2).
Transfers from your role
Interview and focus group design, and coding. The base you need to review AI coding.
Build it: Write a codebook with definitions for one evaluation question.
Transfers from your role
Triangulating survey, interview and document evidence.
Build it: Build an evidence table that combines three kinds of source.
Transfers from your role
Tools that measure what the questions ask.
Build it: Check one AI-drafted survey module against your matrix.
Transfers from your role
Models you can check yourself, so AI explanations can be tested.
Build it: Re-run one model with a variable recoded and compare.
Transfers from your role
Screening, appraising and summarising studies.
Build it: Screen 20 abstracts against written criteria.
Transfers from your role
Findings, conclusions and recommendations linked to evidence.
Build it: Link every finding in one chapter to its sources.
Transfers from your role
Validation workshops where stakeholders test your findings.
Build it: Plan how you will present one contested finding.
Step 3 of 5New skills for the AI era
These are the ones most evaluation specialists still need to learn, or to apply in a new way.
Your skills match
0 of 14
B. New AI and evaluation competencies
Needs new learning
Where AI saves evaluators the most time and where errors are hardest to spot. The profile's first learning priority.
Build it: Review AI coding of five interviews and record agreement (guide 3).
Needs new learning
Contribution analysis gives structure to conclusions when evidence is easier to gather but not to interpret (a learning priority).
Build it: Write the contribution story for one outcome.
Needs new learning
Checking summaries, codes and citations against their sources.
Build it: Check every point in one AI document summary (guide 1).
Needs new learning
Prompts that ask for sources, mark uncertainty and forbid invented figures.
Build it: Rewrite one prompt to require a source for every point.
Needs new learning
AI drafts can sound convincing while overstating what the evidence shows (a learning priority).
Build it: Mark every overstatement in one AI draft.
Needs new learning
Consent and confidentiality when interview data meets AI tools (a learning priority).
Build it: Add AI processing to your consent form, with your ethics focal point.
Step 4 of 5Your five learning priorities
In this order, from the career profile:
AI-assisted qualitative analysis. It is where AI saves evaluators the most time, and where errors are hardest to spot.
Theory-based evaluation. Contribution analysis gives structure to conclusions when AI makes evidence easier to gather but not easier to interpret.
Elicit. AI-assisted search makes rapid evidence reviews feasible within evaluation timelines.
Research and evaluation ethics. Using AI with interview data raises consent and confidentiality questions.
Critical appraisal. AI drafts can sound convincing while overstating what the evidence shows.
The tools change; the method doesn't. Whichever tool you use, ask for sources, check a sample against the original, and keep the judgement on design and strength of evidence with you.
End of Mission 05
MISSION06
Interactive skills gap assessment
Know your gaps · Badge: Self-Aware Analyst
1 challenge
Mission goal: measure your readiness in four areas and find your top three gaps.
Step 1 of 3How it works
10 quick statements, about 3 minutes. Rate yourself honestly: this is a learning tool, not a test, and your answers stay in your browser.
Beginner: little or no experience
Developing: I understand it but need practice
Competent: I can apply it independently
Step 2 of 3Questions 1 to 5
Step 3 of 3Questions 6 to 10
Prefer to score it yourself?
Beginner = 0, Developing = 1, Competent = 2. Add them up (maximum 20): 0–7 Getting started · 8–14 Developing · 15–20 Ready to run AI-assisted evaluations. For every Beginner answer, start with the matching action:
Use the routine in mission 4, guide 1.
Use the routine in mission 4, guide 2.
Check five minutes from each of five interviews.
Use the routine in mission 4, guide 3.
Check your consent form with your ethics or data protection focal point.
Mission goal: commit to a realistic 12-week plan and track it week by week.
Step 1 of 5Weeks 1–2: Understand AI
About 3 hours a week (a planning suggestion). Tick steps as you do them: +5 XP each, saved in this browser. The mission completes when every step in all four stages is ticked.
Stage 1 · Weeks 1–20 / 5 done
Understand AI
Find the slow steps in your evaluations and learn what AI gets wrong.
Loot: a list of slow steps, an error log and checked consent forms.
Step 2 of 5Weeks 3–4: Apply AI
Stage 2 · Weeks 3–40 / 4 done
Apply AI
Try AI on four evaluation tasks, with checks.
Loot: a checked summary, a revised matrix and a coding review.
Step 3 of 5Weeks 5–8: Integrate AI
Stage 3 · Weeks 5–80 / 5 done
Integrate AI
Use AI across a live evaluation, with checks at each stage.
Loot: an evaluation with AI used and checked at each stage.
Step 4 of 5Weeks 9–12: Demonstrate competence
Stage 4 · Weeks 9–120 / 5 done
Demonstrate competence
Show the approach is reliable and others can follow it.
Loot: a portfolio project, evidence of time saved and a development plan.
Step 5 of 5Check
Mission check+20 XP
In which stage do you first review AI coding of interviews?
Show answer
Stage 2 (weeks 3–4).
End of Mission 07
MISSION08
Your practical learning project
Boss level · Badge: Evidence Checker
1 challenge + 8 deliverables
Mission goal: produce a portfolio piece that proves you can use AI in evaluation responsibly.
Step 1 of 6Your boss-level project
Boss level project
Run one evaluation with AI at each stage, and every output checked
Objective: show you can use AI from inception to report, check every AI output against its source, and keep the judgement on design and evidence with you.
Case study · Learning Grants Programme evaluation Illustrative
No live evaluation to use? Work through this evaluation, using the examples from mission 4.
Context: one evaluation, or one evaluation question within it. Use anonymised data, and only where consent covers AI processing.
Resources: the terms of reference, programme documents, interview transcripts or audio, your codebook, survey data, Elicit and Zotero, and an approved AI assistant.
Step 2 of 6Deliverables, one by one
Deliverables checklist0 / 8 done
8 steps to a portfolio piece
The mission and the certificate unlock only when all 8 deliverables are ticked.
Step 3 of 6Templates to start from
Four ready-made files for your deliverables. Download, adapt and keep them.
Before you share it: what should a person check in your evaluation?
Think about it for a moment, then reveal.
Every summary point and citation, against its source.
A sample of transcripts, against the audio.
Every coded passage for the key questions.
Every finding, against the evidence table.
No identifiable respondent in any AI step or shared file.
Step 5 of 6How it's judged
Criterion
What good looks like
Traceable
Every point, code and citation traces to its source.
Checked coding
AI coding reviewed, with agreement recorded.
Read, not skimmed
Every cited study read in full.
Strength of evidence
No conclusion rests on a single interview.
Protects respondents
Consent covers AI; transcripts anonymised.
Transparent
The methodology says how AI was used and checked.
Step 6 of 6Check
Mission check+20 XP
Which deliverable shows that your qualitative findings don't depend on unchecked AI codes?
Show answer
The coding review.
End of Mission 08
MISSION09
Responsible AI: what you must never delegate blindly
Guardrails · Badge: Responsible AI Guardian
10 challenges
Mission goal: know exactly which judgements stay with you, and the control for each AI risk.
Step 1 of 6Would you approve this report?
Click every sentence you would not send. There are five problems: an invented figure, an unsupported causal claim, an incorrect percentage, a missing indicator and a misleading interpretation.
Case study · Learning Grants Programme evaluation Illustrative
An AI assistant drafted the effectiveness and sustainability findings from the evidence table. The draft report is due to the evaluation manager today.
Source (evidence table)
Result
Attendance, girls aged 10–13
+2.1 days a month compared with baseline (endline survey)
Attendance, girls aged 14–16
No significant change (p = 0.51)
FGDs saying attendance improved
6 of 8
Households expecting to keep children in school
68% (endline survey)
Schools keeping attendance tracking
4 of 5 (school records; 3 KIIs)
Savings groups replacing the grant
1 KII
Learning outcomes
Not measured
Problems found: 0 of 5+40 XP
Show answer
Five problems: the claim about all girls (no significant change for girls aged 14–16), the 86% (it is 68%, and an expectation), the invented test-score figure, the unsupported claim about savings groups (one interview), and the claim that every finding is reported (school practices are missing).
Step 2 of 6Your decision
Your decision+20 XP
You found the problems. Can the draft go to the evaluation manager?
Show answer
Correct it first. Fix each problem, recheck every finding, then send it.
Step 3 of 6Five questions before you send it
The findings are corrected. Before you send them, answer the five questions. Tick each one you can honestly answer yes to.
Step 4 of 6Mini-game: delegate or never?
AI can transcribe, code, search and draft. It cannot be accountable. For each task: can AI help (with checks), or must you never delegate it?
Delegate?+10 XP
Transcribing interviews
Show answer
AI can help. With a sample checked against the audio.
Delegate?+10 XP
Choosing the evaluation design
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
First-pass coding with your codebook
Show answer
AI can help. With every key passage reviewed and agreement recorded.
Delegate?+10 XP
Weighing conflicting evidence and judging its strength
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Searching for comparable studies
Show answer
AI can help. With every cited study read in full.
Delegate?+10 XP
Drawing conclusions and recommendations that are fair to stakeholders
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Drafting report sections from the evidence table
Show answer
AI can help. With every finding checked against its sources.
Delegate?+10 XP
Protecting respondents and keeping the evaluation independent
Show answer
Never delegate. Listed in the profile as human judgement.
Step 5 of 6Six risks, six controls
What can go wrong when AI enters your evaluation? Name three risks before you look.
Think about it for a moment, then reveal.
Risk
What it looks like
Your control
Fluent overclaiming
"Highly sustainable" from thin evidence
An evidence table; no conclusion on a single source
Wrong citations
A figure cited to a page that doesn't contain it
Check every point against its page
Miscoded passages
A past barrier coded as one that remains
Review key passages; record agreement
Transcription errors
Wrong names or numbers in a quote
Check a sample against the audio
Exposed respondents
Identifiable transcripts in a consumer tool
Consent covers AI; anonymise before any AI step
Hidden AI use
Readers can't judge how findings were produced
An AI-use note in the methodology
Step 6 of 6Tell readers how AI was used
Transparency is a control too. Add a short note to the methodology section, and adapt it to what was actually done.
AI tools supported this evaluation. Whisper transcribed the interviews, and a sample was checked against the audio. An AI assistant applied the codebook as a first pass; all passages for the effectiveness questions and a sample of the others were reviewed by the evaluator, with [x]% agreement. Elicit supported the literature search, and every cited study was read in full. An AI assistant drafted sections from the evidence table, and every finding was checked against its sources by [name]. Transcripts were anonymised before AI processing, as agreed in the consent process.
The AI-Enabled Senior M&E Specialist is covered in Tutorial 03 and the AI-Enabled Evaluation Manager in Tutorial 09.
Step 2 of 5What you need to show
Strengths you build on
Evaluation design with the OECD-DAC criteria
Qualitative and quantitative fieldwork
Triangulating across sources
Findings, conclusions and recommendations
What you need to show
AI coding reviewed, with agreement recorded
Cited studies read in full
Findings linked to an evidence table
AI explanations tested against the data
AI use disclosed in the methodology
Step 3 of 5Tool: build your LinkedIn headline
Pick the pieces that are true for you. Only claim what you have actually done.
Your headline
Step 4 of 5CV bullets and qualifications
CV bullet templates
Summarised [60] programme documents with Claude during inception, checking every point against its source before use.
Reviewed AI first-pass coding of [30] KIIs against the codebook, with [x]% agreement recorded.
Ran a rapid evidence review with Elicit: screened [40] abstracts and read [8] papers in full before citing them.
Drafted findings from an evidence table with AI support, removed conclusions that rested on a single interview, and disclosed AI use in the methodology.
Replace the brackets with what you have actually done.
Qualifications in the profile
Master's degree in evaluation, development studies, social sciences, economics or a related field.
3–6 years conducting or supporting programme evaluations, including qualitative and quantitative fieldwork.
Useful certification: IPDET evaluation training programme.
Step 5 of 5Check
Mission check+20 XP
What's the best way to use "AI-Assisted Evaluation Specialist" on your CV?
Show answer
Use it to describe your capabilities, backed by evidence, unless your employer actually uses the title.
End of Mission 10
MISSION11
Your next steps
Commit and celebrate · Badge: Transition Ready
1 challenge
Mission goal: turn what you learned into a two-week plan and unlock your certificate.
Step 1 of 2Commit to your next two weeks
Pick the actions you commit to for the next two weeks. Your plan builds itself below.
Start learning
Do this
Show this
My plan
Recommended first project: the evaluation from mission 8, starting with the inception document review.
Step 2 of 2Your completion certificate
Complete all 11 missions to unlock it. Add your name, take a screenshot and share your progress.
AI & M&E Career Transition Series · Tutorial 08
Your name
has completed the learning path
From Evaluation Specialist to AI-Assisted Evaluation Specialist
This is a learning-progress certificate. It records what you completed in this free tool, in your own browser; it is not independently verified and not a formal qualification. Your mission 8 project is the evidence to show an employer.
End of Mission 11
Questions and answers
What does an AI-Assisted Evaluation Specialist do?
Designs and conducts evaluations of programmes, projects and strategies, answering questions about relevance, effectiveness, efficiency, impact and sustainability. Uses AI tools for evaluation design, evidence synthesis, data analysis, report drafting and interpretation support, and checks every AI output against the evidence.
How is AI changing the Evaluation Specialist role?
AI speeds up the slowest parts of evaluation work, such as document review, transcription, first-pass coding and drafting, so the specialist can cover more evidence in the same time. The specialist's judgement on design, triangulation and the strength of evidence becomes more important, because AI can produce fluent conclusions that the data does not support.
Which judgements should an AI-Assisted Evaluation Specialist never hand over to AI?
The career profile lists these as human judgement:
Choosing an evaluation design that fits the questions, budget and context.
Weighing conflicting evidence and judging its strength.
Drawing conclusions and recommendations that are fair to stakeholders.
Protecting respondents and keeping the evaluation independent.
Which skills should I learn first?
The learning priorities in the career profile, in order:
AI-assisted qualitative analysis. Qualitative analysis is where AI saves evaluators the most time and where errors are hardest to spot.
Theory-based evaluation. Contribution analysis gives structure to conclusions when AI makes evidence easier to gather but not easier to interpret.
Elicit. AI-assisted search makes rapid evidence reviews feasible within evaluation timelines.
Research and evaluation ethics. Using AI with interview data raises consent and confidentiality questions.
Critical appraisal. AI drafts can sound convincing while overstating what the evidence shows.
The profile lists 14 skills in all: Evaluation design, Qualitative methods, Mixed-methods integration, Theory-based evaluation, Survey and questionnaire design, Statistical analysis, Evidence synthesis, AI-assisted qualitative analysis, Validating AI outputs, Prompt design, Critical appraisal, Research and evaluation ethics, Analytical report writing and Facilitation.
Which tools does this role use?
The career profile names: OECD-DAC evaluation criteria, Claude, ChatGPT, Whisper (speech-to-text), NVivo, Elicit, Zotero, Stata, KoboToolbox and Excel. The tutorial shows how to check every AI output, whichever tool you use.
What qualifications and experience does the profile list?
Master's degree in evaluation, development studies, social sciences, economics or a related field.
3–6 years conducting or supporting programme evaluations, including qualitative and quantitative fieldwork.
Useful certification: IPDET evaluation training programme.
How long does the transition take?
This tutorial suggests a 12-week plan of about 3 hours a week, in four stages: understand AI (weeks 1–2), apply AI (weeks 3–4), integrate AI (weeks 5–8) and demonstrate competence (weeks 9–12). It is a planning suggestion, not a requirement.
Is "AI-Assisted Evaluation Specialist" an official job title?
No. It is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy. Use it to describe your capabilities, backed by evidence, unless your employer actually uses the title.
