Mission goal: understand what the AI-enhanced version of your role is and why it is emerging.
Step 1 of 5Where are you starting from?
Choose your starting point+10 XP
Where are you today with AI in your M&E work?
Show answer
Every starting point is valid. Follow the mission suggested for your situation.
Step 2 of 5Meet your new role
Today's role
Senior M&E Specialist
AI-enhanced role
AI-Enabled Senior M&E Specialist
Level
Senior / Advisor · 6–10 years
How the role uses AI
Integrates AI: leads AI integration across M&E functions and validates AI-supported findings
"AI-Enabled Senior M&E Specialist" is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy.
Step 3 of 5What changes most in your job?
Your guess+10 XP
When AI takes on more of the analysis, where does your time go?
Show answer
In the profile, the senior specialist moves from doing most analysis personally to designing how AI-assisted analysis is done and checked across the portfolio: setting review standards, testing AI outputs on real programme data, and explaining to management what results can and cannot support.
Step 4 of 5AI-Enabled, not AI-Dependent
The goal is not to hand your work to AI. It is to become a stronger senior M&E specialist who uses AI and stays in charge of it. Here are the five questions, asked about your own work:
1QualityDo the AI tools you approve make findings across the portfolio more credible, and have you tested them on your own data?A pilot on a past dataset with known results, before any rollout.
2AccountabilityWhen an AI-supported finding reaches a donor, is it clear who validated it, and against which rule?Written validation rules and a named reviewer for every AI-supported evaluation.
3IndependenceDo you still set the standard of proof for causal claims, whatever the AI suggests?You weigh evidence quality against pressure for positive results.
4EthicsDo evaluation teams protect participants' data and disclose AI use for transcription, translation and coding?A guidance note on what may go into AI tools, and disclosure in every report.
5Professional developmentAre the specialists you mentor learning to judge AI output, or learning to accept it?Peer review that asks how every AI result was checked.
The five questions as the series states them
Quality: Does AI improve the quality and credibility of evaluation findings?
Accountability: Who is responsible when AI produces an incorrect interpretation?
Independence: Are evaluators maintaining their ability to think critically and make independent judgments?
Ethics: Are confidentiality, human rights and the interests of affected populations protected?
Professional development: Are we developing stronger evaluators who understand AI, or creating professionals who increasingly depend on it?
Step 5 of 5Your goal for this tutorial
By the end, you will know how to test, supervise and explain AI-supported analysis step by step, which skills to build, and you will have a 12-week plan.
Next: see which of your tasks change, and how.
End of Mission 01
MISSION02
Traditional role vs AI-enhanced role
Before and after · Badge: Role Shifter
8 challenges
Mission goal: see which responsibilities become AI-assisted, which need new skills and which stay human.
Step 1 of 4Meet your case study
Case study · Multi-Donor Resilience Portfolio Illustrative
A fictional portfolio you will follow through missions 2, 3, 4, 8 and 9.
The portfolio
Three programmes (food security, education and health) in Countries A, B and C, two donors, and about four external evaluations a year.
The data
Twelve evaluation reports from four years, KII and FGD transcripts in several languages, 6,000 feedback messages a year, and quarterly IPTTs.
The tools
Claude, ChatGPT, NVivo, MAXQDA, Whisper, Python (pandas), Power BI and SharePoint.
Your objective
Set how AI is used and checked across the portfolio, and advise management on what AI-supported results can support.
Step 2 of 4Flip your tasks: today and with AI
You do less of the analysis yourself and more of the checking and standard-setting. Tap each card to see how it changes.
Step 3 of 4Mini-game: sort the work
For each task, choose: AI-assisted, a new skill for you, or stays human.
Sort it+10 XP
First-pass coding of 60 KII transcripts
Show answer
AI-assisted. AI can code at scale, once it has been tested on your own data.
Sort it+10 XP
Listing recurring findings across twelve evaluation reports
Show answer
AI-assisted. AI reads and lists; you check every finding against its source report.
Sort it+10 XP
Transcribing and translating FGD recordings
Show answer
AI-assisted. Useful with checks, such as a back-translated sample (the profile's example).
Sort it+10 XP
Testing an AI tool's accuracy on your organisation's own data
Show answer
New skill. Evaluating AI systems is the first learning priority in the profile.
Sort it+10 XP
Writing validation rules such as double-coding samples and source tracing
Show answer
New skill. Defining validation rules for AI-supported analysis is new work for the role.
Sort it+10 XP
Deciding which M&E tasks AI may support
Show answer
Human. Listed in the profile as human judgement.
Sort it+10 XP
Setting the standard of proof for causal claims
Show answer
Human. Listed in the profile as human judgement, whatever the AI suggests.
Sort it+10 XP
Weighing evidence quality against pressure for positive results
Show answer
Human. Listed in the profile as a judgement that stays human.
Step 4 of 4Your turn: two minutes
Think about the evaluations and reviews you oversaw last quarter. Which tasks would go in each pile?
Think about it for a moment, then reveal.
A quick way to do it: three columns on a sheet of paper, AI-assisted, new skill, stays human. Keep the list: you will use it in mission 4 (to pick a task to try) and mission 6 (to rate your gaps).
End of Mission 02
MISSION03
What AI actually changes in your daily work
Checks, risks and red flags · Badge: Red Flag Spotter
2 challenges
Mission goal: learn the checks that make AI-supported analysis credible across a portfolio, and practise spotting problems in AI drafts.
Step 1 of 4What can AI really do for you?
Before you look: which of your tasks could AI help with, and which must stay with you?
Think about it for a moment, then reveal.
AI can assist with
First-pass coding of transcripts
Transcription and translation
Comparing findings across evaluations
First drafts of summaries and narratives
Can be partly automated
Agreement checks on coded samples
Breaking indicators down by site and source
Scores from large volumes of feedback
New responsibilities
Testing AI tools on your own data
Validation rules for AI-supported analysis
Requiring disclosure of AI use
Prompt templates for the wider team
Stays with you
Which tasks AI may support
Whether findings are credible enough
The standard of proof for causal claims
Evidence quality over pressure for good news
Step 2 of 4The three checks you must always set
Across a portfolio, AI output is only usable when everyone checks it the same way. Here are the checks, one at a time.
Test on your own data. Pilot any AI tool on a past dataset with known results before wider rollout (80 hand-coded transcripts in the profile's example).
Validation rules. Double-coding samples, source tracing and back-translation, written down so every team applies them.
Disclosure. Evaluation reports say which AI tools were used, for what, and how they were checked.
Step 3 of 4Mini-game: spot the red flags
Case study · Multi-Donor Resilience Portfolio Illustrative
An AI assistant summarised the three country reports for the regional review. Here is the source data and the draft.
Click every sentence you would not send to management.
What the three IPTTs show for the year
Country A: households with improved food consumption
62% (baseline 48%)
Country B: children attending school
84% (last year 81%)
Country C: clinic visits
14,200 against a target of 18,000 (79%)
Reports received
All three countries
Red flags found: 0 of 3+30 XP
Show answer
Three sentences should not be sent: the "dramatic rise thanks to the programme" (3 points, cause unknown), "Country C exceeded its target" (it is at 79%) and the promise about next year.
Step 4 of 4Mission check
Your call+20 XP
A vendor shows an AI tool that summarises evaluation reports perfectly on its own examples. What do you do before approving it for your teams?
Show answer
Test it first on your own past reports, where you know the right answer, before anyone relies on it.
End of Mission 03
MISSION04
Your daily tasks, step by step with AI
Six practical guides · Badge: Workflow Builder
6 challenges
Mission goal: learn how to test, supervise and explain AI-supported analysis in six of your real tasks, step by step.
Case study · Multi-Donor Resilience Portfolio Illustrative
Every worked example below uses this portfolio's data, so you can follow one system from evaluation to management brief.
Six real tasks from your working week. Pick one and go through it in six short steps: a task prompt, then a QA prompt that checks the AI's work.
YouAIYour teamAutomated shows who does each step. Examples are illustrative.
Choose a task
Guide 1 of 6 From the career profile
Test an AI coding tool before your teams use it
From the profile: before approving AI coding for a mid-term evaluation, the senior specialist runs Claude on 80 transcripts already coded by hand, compares agreement code by code, and drops two codes the model applies unreliably.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
A new AI tool is adopted because a colleague found it useful.
Nobody knows how accurate it is on your own transcripts.
Problems surface in the evaluation report, after the donor reads it.
How it works with AI
You run the tool on transcripts already coded by hand.
You compare agreement code by code, not just overall.
You keep only the codes the tool applies reliably, and write the decision down.
Step 2 of 6Step by step
YouPick a test set with known answersMAXQDA
Use transcripts your team already coded by hand and trusts (80 in the profile's example). Remove names and identifying details first.
YouSet the bar in advanceExcel
Decide the agreement each code must reach before you run the test, for example 80%, and write it down.
AIRun the tool on the test setClaude
Give Claude the codebook and the anonymised transcripts in batches. Ask for one code per segment and a short reason.
AutomatedCompare code by codeExcel
Agreement per code = matching segments ÷ segments checked. A good overall score can hide one weak code.
YouDrop or fix weak codes
Codes below the bar stay with human coders, or are redefined and tested again. In the profile's example, two codes were dropped.
YouWrite the decision downSharePoint
Record what was tested, the result per code and what the tool may be used for. This is what you show management.
Step 3 of 6Worked example
What you give the AI Illustrative
Code
Segments checked
AI matches the hand code
Partner capacity
120
108
Community ownership
95
88
Gender norms
60
33
Funding gaps
70
64
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Below are agreement results from a test of AI coding on transcripts we had already coded by hand.
For each code, calculate agreement (matches ÷ segments checked) as a percentage.
List the codes below 80%.
Do not suggest why a code fails and do not recommend a tool. Report the numbers only.
[paste the table]
What comes back
Code
Agreement
Below 80%?
Partner capacity
90%
No
Community ownership
93%
No
Gender norms
55%
Yes
Funding gaps
91%
No
Overall agreement is 85% (293 of 345), which looks fine. Code by code, Gender norms fails, so it stays with human coders.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Overall agreement is 85%, so the tool is reliable for all codes. Gender norms reached 75%. We recommend rolling the tool out to all evaluations.
The QA prompt (run it on every draft)
Act as an independent reviewer of an AI tool test.
Compare the summary below with the agreement table.
List: every percentage that differs from the table, every conclusion that relies on overall agreement while a code is below the bar, and every recommendation the test does not support.
Do not rewrite the summary. List the problems only.
Table: [paste]
Summary: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"Reliable for all codes" rests on the overall 85%; Gender norms is below the 80% bar.
Gender norms is 55% (33 of 60), not 75%.
Rolling out to all evaluations goes beyond the test, which covered one codebook and one set of transcripts.
Corrected output
Overall agreement is 85% (293 of 345 segments). Three codes meet the 80% bar. Gender norms (55%) does not: it stays with human coders.
The tool may be used for the three tested codes in this evaluation. Any other use needs its own test.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Quality: a tool that works on someone else's data may fail on yours. Test it where you already know the answers.
Step 6 of 6Quick check
Quick check+25 XP
Overall agreement is 85%, but one code is at 55%. What do you approve?
Show answer
Approve it for the codes that pass, and leave the weak code to human coders.
Try it this week
Take 30 segments your team coded by hand last year. Run them through an AI assistant with your codebook and calculate agreement code by code.
From the profile: reviewing an evaluation team's inception report and asking how they will check AI coding of 60 KII transcripts; reviewing teams' use of AI for transcription, translation and coding, and requiring disclosure in reports.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Inception reports mention "AI-assisted analysis" in one line.
Nobody asks how transcription, translation or coding will be checked.
AI use is not disclosed in the final report.
How it works with AI
You ask which AI tools will be used, on which data, and how each will be checked.
You require samples: back-translation, double-coding, quotation checks.
You make disclosure in the final report a condition of approval.
Step 2 of 6Step by step
YouList every AI step in the plan
Transcription, translation, coding, summarising: note each one and the data it touches.
AIAsk the AI to find the gapsClaude · ChatGPT
Give the AI the methods section, without personal data, and ask what is missing for each AI step (prompt below).
YouRequire a check for each step
For example: compare transcripts with the audio, back-translate a sample, double-code a sample of segments, check every quotation used.
YouCheck data protection
Where will recordings and transcripts be processed? Does that fit the consent given and your organisation's rules?
Your teamAgree the changes with the team
Send your comments and agree the checks, sample sizes and timing before fieldwork starts.
YouMake disclosure a condition
The final report must say which AI tools were used, for what, and how they were checked.
Step 3 of 6Worked example
What you give the AI Illustrative
Methods excerpt: 60 KIIs will be recorded, transcribed with Whisper, translated from Hausa with an LLM and coded with AI in NVivo. Findings will be validated by the team leader. [No personal data]
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
You are reviewing the methods section of an evaluation inception report.
For each step that uses AI (transcription, translation, coding, summarising), list: the data it uses, what could go wrong, and what check the report should describe.
Then say what the report states about data protection and about disclosing AI use.
Do not rewrite the methods. List gaps only.
[paste the methods section, no personal data]
What comes back
AI step
What could go wrong
Check to require
Transcription (Whisper)
Words missed or misheard, especially names and local terms
Compare a sample of transcripts with the audio
Translation (LLM)
Meaning shifts; idioms lost
Back-translate a sample of five
Coding (AI in NVivo)
Codes applied wrongly at scale
Double-code a sample and report agreement
Data protection and disclosure
Not mentioned
Add both before approval
"Validated by the team leader" is not a check. Ask how, on what sample, and with what result.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
AI step
Check proposed in the draft review
Transcription (Whisper)
None needed: Whisper is accurate
Translation (LLM)
Back-translate a sample of five
Coding (AI in NVivo)
Team leader validates the findings
The QA prompt (run it on every draft)
Act as an independent evaluation quality reviewer.
Check this list of AI checks against the inception report's methods.
List: every AI step with no real check, every check that is a statement rather than a method with a sample, and every missing topic such as data protection or disclosure.
Do not rewrite the list. List the problems only.
Methods: [paste]
Checks: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
Transcription has no check: accuracy must be tested on a sample against the audio.
"Team leader validates the findings" is not a method: it needs a double-coded sample and an agreement score.
Data protection and disclosure of AI use are missing.
Corrected output
AI step
Check required before approval
Transcription
Compare five transcripts with the audio
Translation
Back-translate a sample of five
Coding
Double-code 10% of segments; report agreement
Data and disclosure
Where recordings are processed; AI use stated in the final report
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Accountability: approve the plan only when it says who checks each AI step, how, and how AI use will be disclosed.
Step 6 of 6Quick check
Quick check+25 XP
The inception report says AI findings "will be validated by the team leader". Is that enough?
Show answer
No. Ask for a named method, a sample size and a reported result.
Try it this week
Open the last inception report you reviewed. List each AI step and write the check you would now require for it.
From the profile: an AI assistant reads twelve evaluation reports from the last four years and lists recurring findings on partner capacity; the senior specialist checks each against the source report before presenting it to the management team.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Lessons from past evaluations sit in PDFs that nobody rereads.
Management asks what keeps coming up, and the answer relies on memory.
The same problems are found again in each new evaluation.
How it works with AI
An AI assistant reads the reports and lists recurring findings with citations.
You check every finding against its source report.
Management gets a short, verified list with the evidence behind it.
Step 2 of 6Step by step
YouCollect the reportsSharePoint
Gather the final evaluation reports in one folder (twelve from four years in the profile's example).
YouAsk one clear question
For example: what do these evaluations find about partner capacity? A narrow question gives answers you can check.
AIAsk for findings with citationsChatGPT
The AI lists recurring findings, each with the reports and pages it comes from (prompt below).
YouCheck every citation
Open each cited page and confirm it says what the AI claims. Remove anything you cannot find.
YouWeigh the evidence
Note how strong each source is. A finding from one weak evaluation is not a pattern.
YouBrief management
Present the verified findings, how many reports support each one, and what acting on them would mean.
Step 3 of 6Worked example
What you give the AI Illustrative
Report
Year
Relevant finding (excerpt)
Evaluation 3
2023
Partners lacked staff to run monthly data checks (p. 14)
Evaluation 7
2024
Two of five partners had no M&E officer (p. 9)
Evaluation 11
2025
Partner reports arrived late in 6 of 8 quarters (p. 22)
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Below are excerpts from evaluation reports, each with the report name and page.
List the findings about partner capacity that appear in more than one report.
For each finding, cite every report and page it comes from.
Use only the excerpts. If a finding appears in only one report, say so. Do not add recommendations.
[paste excerpts with report names and pages]
What comes back
Recurring finding: partners have limited M&E staff capacity (Evaluation 3, p. 14; Evaluation 7, p. 9).
Single report: late partner reporting (Evaluation 11, p. 22). Possibly linked to capacity, but only one report raises it.
Each citation is checked against its page before the finding goes to management.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Partners lack M&E capacity in all twelve evaluations (Evaluation 3, p. 14; Evaluation 7, p. 9; Evaluation 9, p. 31). Late reporting is caused by low partner capacity.
The QA prompt (run it on every draft)
Act as an independent reviewer of an evidence synthesis.
Compare the synthesis below with the source excerpts.
List: every citation that does not appear in the excerpts, every count of reports that the citations do not support, and every causal link the sources do not state.
Do not rewrite the synthesis. List the problems only.
Excerpts: [paste]
Synthesis: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"All twelve evaluations": only two of the excerpts support the finding.
Evaluation 9, p. 31 is not in the excerpts: an invented citation.
"Caused by low partner capacity" is a link no report makes.
Corrected output
Recurring finding: partners have limited M&E staff capacity (Evaluation 3, p. 14; Evaluation 7, p. 9).
Single report: late partner reporting (Evaluation 11, p. 22). It may be linked to capacity, but no report says so.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Independence: AI lists recurring findings. You check each one against its source report before management sees it.
Step 6 of 6Quick check
Quick check+25 XP
The AI says a finding appears in "most evaluations" and cites four of twelve reports. What do you tell management?
Show answer
Report the checked count: four of twelve reports, each citation verified.
Try it this week
Pick three past evaluations of one programme. Ask an AI assistant for recurring findings with page citations, then check every citation.
Advise on a 15% drop: real change or new data source?
From the profile: advising the country director on whether a 15% drop in a reported outcome reflects real change or a new data source; checking AI-generated performance analyses for overstated causal claims before they reach senior management.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
The drop appears in the dashboard and alarms the country director.
Explanations come quickly, before anyone checks the data.
The programme is blamed, or the data is dismissed, without evidence.
How it works with AI
You check what changed in the data before anything else.
AI helps you describe the change by data source and site, without guessing causes.
You tell the director what the figure can and cannot support.
Step 2 of 6Step by step
YouCheck what changed in the data
Did the source, form, definition, sample or set of reporting sites change when the drop began?
AutomatedCompare like with likePython (pandas) · Power BI
Break the indicator down by data source and by sites that reported the same way in both periods.
AIAsk the AI to describe the patternClaude
Give the AI the anonymised table and ask it to describe the change by group, without explaining causes (prompt below).
YouRule out the data explanation
If the drop sits only where the new source started, it is most likely a measurement change, not a real one.
Your teamAsk the programme team
Check whether anything changed on the ground in the same period.
YouAdvise in plain language
Tell the director what the drop most likely reflects, how sure you are, and what to report in the meantime.
Step 3 of 6Worked example
What you give the AI Illustrative
Sites
Data source
Before
After
Change
Sites 1–8
Paper registers (unchanged)
64%
63%
−1 point
Sites 9–12
New mobile survey from Q3
66%
44%
−22 points
All sites
Mixed
65%
55%
−15%
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Below is an outcome indicator by group of sites and data source, before and after Q3.
Describe where the change happens and how large it is in each group.
Note any change of data source that coincides with the drop.
Do not explain causes and do not recommend actions.
[paste anonymised table]
What comes back
Pattern: sites 1–8, with an unchanged source, are stable (−1 point). Sites 9–12, which switched to a new mobile survey in Q3, drop by 22 points and account for almost all of the overall 15% fall.
Coincides with: the change of data source at sites 9–12.
The likely answer is a measurement change. Confirm with the programme team before you advise the director.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
The outcome fell 25% across all sites, which shows the programme is losing effect. Sites 9–12 should be reviewed for poor performance.
The QA prompt (run it on every draft)
Act as an independent M&E reviewer.
Compare the summary below with the table by site and data source.
List: every figure that differs from the table, every conclusion that ignores the change of data source, and every recommendation the data does not support.
Do not rewrite the summary. List the problems only.
Table: [paste]
Summary: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
The overall fall is 15% (65% to 55%), not 25%.
"The programme is losing effect" ignores that the drop sits where the data source changed.
Reviewing sites 9–12 "for poor performance" assumes a real drop the data does not show.
Corrected output
Sites 1–8, with an unchanged source, are stable (−1 point). Sites 9–12 switched to a new mobile survey in Q3 and drop by 22 points, which accounts for almost all of the 15% fall.
Advice: report the figure with a note on the change of data source, and run both sources side by side at sites 9–12 next quarter.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Independence: you set the standard of proof: a change in data source is ruled out before a drop is called real.
Step 6 of 6Quick check
Quick check+25 XP
A reported outcome drops 15% in the quarter a new data source starts. What do you advise first?
Show answer
Compare like with like by data source first, then advise on what the figure can support.
Try it this week
Find one indicator whose data source or method changed in the last year. Compare it by source and write the two sentences you would say to the country director.
Decide whether an AI-derived score can be reported
From the profile: asked to report an AI-derived satisfaction score from 6,000 feedback messages, the senior specialist checks it against a hand-rated sample of 300 and advises reporting it as indicative only.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
An AI tool produces a satisfaction or sentiment score from feedback.
It goes into the management report because it is a number.
Nobody knows how it relates to what people actually said.
How it works with AI
You check the score against a hand-rated sample.
You report it as indicative unless it holds up.
Management sees the score with a plain note on how reliable it is.
Step 2 of 6Step by step
YouDefine what the score claims to measure
Write one sentence: what does "satisfied" mean here, and which messages count?
AutomatedDraw a random samplePython (pandas)
Pick a random sample of messages (300 of 6,000 in the profile's example) across sites, languages and groups.
Your teamRate the sample by hand
Two people rate each sampled message without seeing the AI score, then agree any differences.
AutomatedCompare the twoExcel · Power BI
Compare the share rated satisfied by people with the AI score, overall and by group and language.
YouDecide how to report it
If the AI score is far from the hand-rated sample, or differs by group, report it as indicative only, or not at all.
YouExplain it to management
One plain sentence: what the score is, how it was checked and how far to trust it.
Step 3 of 6Worked example
What you give the AI Illustrative
Measure
Result
AI satisfaction score, 6,000 messages
78% satisfied
Hand-rated random sample, 300 messages
64% satisfied
Messages in Hausa
AI score 15 points above the hand rating
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Below are an AI-derived satisfaction score and the results of a hand-rated sample.
State the difference between them, overall and for the group shown.
Then draft one plain sentence for senior management on how this score should be reported.
Do not call the score accurate or inaccurate.
[paste results]
What comes back
Difference: the AI score (78%) is 14 points above the hand-rated sample (64%), and the gap is 15 points for messages in Hausa.
For management: "An AI tool rates 78% of feedback as satisfied, but a hand-rated sample puts it at 64%. We report the score as indicative only."
This is the profile's advice in this case: report it as indicative only.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Beneficiary satisfaction is 78%, confirmed by a sample check. Satisfaction is equally high across languages.
The QA prompt (run it on every draft)
Act as an independent reviewer of a reported metric.
Compare the statement below with the validation results.
List: every figure presented as confirmed that the sample does not confirm, every claim about groups that the results contradict, and any missing note on how the score should be read.
Do not rewrite the statement. List the problems only.
Results: [paste]
Statement: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
The sample does not confirm 78%: it found 64%.
"Equally high across languages" contradicts the 15-point gap for messages in Hausa.
There is no note that the score is indicative only.
Corrected output
An AI tool rates 78% of feedback messages as satisfied. A hand-rated random sample of 300 found 64%, and the gap is larger for messages in Hausa. The score is reported as indicative only.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Quality: an AI-derived score is reported as indicative until a hand-rated sample shows it can be trusted.
Step 6 of 6Quick check
Quick check+25 XP
An AI score says 78% satisfied; your hand-rated sample says 64%. What goes in the management report?
Show answer
Report it as indicative only, with the hand-rated result next to it.
Try it this week
Take one AI-derived metric your team reports. Ask what sample it was checked against. If none, plan a hand-rated sample of 100.
From the profile: drafting a two-page guidance note on when staff may paste beneficiary data into AI assistants; deciding which M&E tasks AI may support and where human analysis is required.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Staff paste data into AI assistants without clear rules.
Each evaluation team decides for itself.
Problems are found only after data has left the organisation.
How it works with AI
AI drafts the structure from your policy and real cases.
You decide the rules, with the data protection focal point.
Staff get two pages they can follow, with examples.
Step 2 of 6Step by step
YouCollect the inputs
Your data protection policy, consent wording, the AI tools staff actually use, and two or three real, anonymised cases.
AIDraft the structureClaude
Ask the AI for a two-page outline with data categories and examples (prompt below).
YouSet the rules yourself
Decide which data may go into which approved tool, after which anonymisation. The AI suggests; you decide.
Your teamAgree with the data protection focal point
Review the draft together and record their sign-off.
YouAdd what evaluation teams must disclose
External teams state which AI tools they use on project data and how they protect it.
Your teamTrain and review
Walk staff through the note with examples. Review it every year, or when a new tool arrives.
Step 3 of 6Worked example
What you give the AI Illustrative
Inputs: the data protection policy (8 pages), consent form wording, tools in use (Claude and ChatGPT on organisational accounts, Whisper), and three anonymised cases: an enumerator pasting survey answers with names, a consultant translating FGD transcripts with an LLM, and a staff member summarising a public report.
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Using the policy excerpts and cases below, draft an outline for a two-page guidance note on using AI tools with programme and beneficiary data.
Include: purpose, data categories with examples, which categories may go into approved AI tools and under which conditions, what external evaluation teams must disclose, who to ask, and what to do after a mistake.
Where the policy does not state a rule, write [TO DECIDE] instead of inventing one.
[paste policy excerpts and anonymised cases]
What comes back
Section
Draft content (excerpt)
Data categories
Public reports; aggregated figures; anonymised text; personal data; sensitive data
Approved tools
Claude and ChatGPT on organisational accounts, for public and aggregated data
Personal data
[TO DECIDE] with the data protection focal point
Evaluation teams
State which AI tools are used on project data, and how it is protected
The [TO DECIDE] lines are the decisions that are yours.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Section
Draft content
Personal data
May be used in AI tools if names are removed
Approved tools
Any AI tool with a paid licence
Evaluation teams
(section missing)
The QA prompt (run it on every draft)
Act as an independent data protection reviewer.
Compare the draft guidance note below with the policy excerpts.
List: every rule the policy does not state, every rule that is weaker than the policy, and every required section that is missing.
Do not rewrite the note. List the problems only.
Policy: [paste]
Draft: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"If names are removed" is a rule the policy does not state, and removing names alone may not anonymise data.
"Any AI tool with a paid licence" is weaker than the policy, which approves named tools only.
The section on what evaluation teams must disclose is missing.
Corrected output
Section
Corrected content
Personal data
Not in AI tools unless a DPIA and the data protection focal point approve
Approved tools
Claude and ChatGPT on organisational accounts only
Evaluation teams
Disclose the AI tools used on project data, and how the data is protected
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Ethics: the note protects participants first: no personal data in unapproved AI tools.
Step 6 of 6Quick check
Quick check+25 XP
The AI draft says personal data may go into AI tools "if names are removed". What do you do?
Show answer
Replace it with a rule you agree with the data protection focal point.
Try it this week
Write down three ways colleagues used AI with programme data last month. Use them as the cases in your guidance note.
Pick a recurring task that is not covered above, such as peer-reviewing a sampling design for a 1,200-household endline, and use this prompt to draft your own step-by-step guide.
I am a senior M&E specialist. Here is a recurring task I do: [describe the task, how often, who is involved, which tools and data].
Break it into numbered steps. For each step, say who should do it (me, a colleague, an evaluation team, an automated workflow or an AI assistant), which tool, and what could go wrong.
Mark every step where a person must check the result before it is used, and say what they should check.
Do not suggest putting beneficiaries' personal or sensitive data into an AI tool.
End of Mission 04
MISSION05
The skills you need to develop
Skill inventory · Badge: Skill Builder
1 challenge
Mission goal: find out which of the 14 profile skills you already have and which need new learning.
Step 1 of 5Guess first
Your guess+20 XP
Which skill is the first learning priority for this role in the career profile?
Show answer
Evaluating AI systems, followed by responsible AI, mixed-methods integration, AI-assisted qualitative analysis and communicating evidence.
Step 2 of 5Skills you bring with you
Tick the ones you already have. Open a skill to see how to build it.
Your skills match
0 of 14
A. Existing M&E and professional competencies
Transfers from your role
Evaluation questions, OECD-DAC criteria, evaluation matrices and design choice: the frame for judging any AI-assisted analysis plan.
Build it: Add one question on AI checks to the next evaluation matrix you review.
Transfers from your role
Logframes, results chains and assumptions: the test for any causal claim an AI draft makes.
Build it: Check one AI-drafted summary against its results chain.
Transfers from your role
SMART indicators, baselines, targets and reference sheets. Clear definitions show when a change in data source alters a figure.
Build it: Note the data source in every indicator reference sheet you approve.
Transfers from your role
Combining qualitative and quantitative evidence. AI-coded findings need triangulation with survey and monitoring data (a learning priority).
Build it: Triangulate one AI-coded theme with survey data.
Transfers from your role
Validity, reliability, timeliness, precision and integrity. The same thinking applies to AI outputs.
Build it: Add an AI-output check to your next DQA.
Transfers from your role
Sample sizes and designs. You need them to set double-coding and hand-rating samples for AI checks.
Build it: Set the sample size for one AI validation check.
Transfers from your role
Questioning evidence and spotting bias and weak reasoning, including in AI outputs.
Build it: Review one AI-drafted report for overstated causal claims.
Transfers from your role
Training and mentoring staff and partners, now including how to check AI output.
Build it: Add an AI-check question to every peer review you run.
Transfers from your role
Briefs and presentations for decision-makers. Management needs plain explanations of how reliable AI-supported figures are (a learning priority).
Build it: Write one plain paragraph on how far to trust an AI-supported figure.
Step 3 of 5New skills for the AI era
These are the ones most senior specialists still need to learn.
Your skills match
0 of 14
B. New AI competencies
Needs new learning
Test sets, accuracy and error analysis, bias testing and monitoring of AI tools in use. The profile's first learning priority.
Build it: Run the pilot test from guide 1 on 30 hand-coded segments.
Needs new learning
Checking AI results against source data, sampling for errors and keeping outputs traceable.
Build it: Check every citation in one AI synthesis (guide 3).
Needs new learning
Transcription, translation, first-pass coding and theme extraction with human review: where they go wrong (a learning priority).
Build it: Back-translate five AI-translated transcripts.
D. Ethics, governance and quality assurance
Needs new learning
Fairness, transparency, explainability and human oversight. Senior staff are asked to decide what AI use is acceptable in evaluations (a learning priority).
Build it: Write the disclosure rule for evaluation reports.
Needs new learning
Consent, anonymisation, DPIAs and safe handling of personal and sensitive data.
Build it: Draft the guidance note from guide 6 with your data protection focal point.
Step 4 of 5Your five learning priorities
In this order, from the career profile:
Evaluating AI systems. Approving AI tools for team use requires testing their accuracy on the organisation's own data.
Responsible AI. Senior M&E staff are increasingly asked to decide what AI use is acceptable in evaluations.
Mixed-methods integration. AI-coded qualitative data needs careful triangulation with survey and monitoring evidence.
AI-assisted qualitative analysis. Supervising AI-assisted evaluations means knowing where transcription, translation and coding go wrong.
Communicating evidence. Management needs plain explanations of how reliable AI-supported figures are.
You don't need to build AI tools. Your job is to test them on your own data, set the rules for checking them, and explain the results. An AI assistant can help with the calculations; you decide what passes.
End of Mission 05
MISSION06
Interactive skills gap assessment
Know your gaps · Badge: Self-Aware Analyst
1 challenge
Mission goal: measure your readiness in four areas and find your top three gaps.
Step 1 of 3How it works
10 quick statements, about 3 minutes. Rate yourself honestly: this is a learning tool, not a test, and your answers stay in your browser.
Beginner: little or no experience
Developing: I understand it but need practice
Competent: I can apply it independently
Step 2 of 3Questions 1 to 5
Step 3 of 3Questions 6 to 10
Prefer to score it yourself?
Beginner = 0, Developing = 1, Competent = 2. Add them up (maximum 20): 0–7 Getting started · 8–14 Developing · 15–20 Ready to lead AI integration. For every Beginner answer, start with the matching action:
Run the pilot test in mission 4, guide 1.
Draft the rules as part of the mission 8 project.
Use the inception report review in mission 4, guide 2.
Check one AI-coded theme against survey data.
Review one AI-drafted report for overstated causal claims (mission 3).
Use the routine in mission 4, guide 4.
Use the routine in mission 4, guide 5.
Draft the guidance note in mission 4, guide 6.
Write one paragraph on a figure you reported last quarter.
Add one AI-check question to every peer review you run.
Mission goal: commit to a realistic 12-week plan and track it week by week.
Step 1 of 5Weeks 1–2: Understand AI
About 3–4 hours a week (a planning suggestion). Tick steps as you do them: +5 XP each, saved in this browser. The mission completes when every step in all four stages is ticked.
Stage 1 · Weeks 1–20 / 5 done
Understand AI
Build AI literacy and see where AI already touches your portfolio.
Loot: an error log, a map of AI use and a list of review tasks.
Step 2 of 5Weeks 3–4: Apply AI
Stage 2 · Weeks 3–40 / 4 done
Apply AI
Practise on past data, where you already know the answers.
Loot: a pilot result, a checked synthesis and three tested prompt templates.
Step 3 of 5Weeks 5–8: Integrate AI
Stage 3 · Weeks 5–80 / 5 done
Integrate AI
Turn what you tested into standards others follow.
Loot: validation rules, an updated review process, a guidance note and a management briefing.
Step 4 of 5Weeks 9–12: Demonstrate competence
Stage 4 · Weeks 9–120 / 5 done
Demonstrate competence
Show you can lead AI integration responsibly.
Loot: a portfolio project, one supervised evaluation and a development plan.
Step 5 of 5Check
Mission check+20 XP
In which stage do you first test AI coding on segments your team already coded by hand?
Show answer
Stage 2 (weeks 3–4).
End of Mission 07
MISSION08
Your practical learning project
Boss level · Badge: Standard Setter
1 challenge + 8 deliverables
Mission goal: produce a portfolio piece that proves you can lead AI integration across M&E responsibly.
Step 1 of 6Your boss-level project
Boss level project
Write an AI quality standard for M&E across your portfolio
Objective: show you can decide where AI may support M&E work, test it, set the checks others follow, and explain to management what the results can support.
Case study · Multi-Donor Resilience Portfolio Illustrative
No portfolio of your own to use? Build the standard for this one, using the examples from mission 4.
Context: use your own programmes and one AI-assisted analysis or evaluation, past or planned. Anonymise anything you share.
Resources: transcripts or data your team already coded or checked by hand, two or three past evaluation reports, your data protection policy, an approved AI assistant, and Excel.
Step 2 of 6Deliverables, one by one
Deliverables checklist0 / 8 done
8 steps to a portfolio piece
The mission and the certificate unlock only when all 8 deliverables are ticked.
Step 3 of 6Templates to start from
Four ready-made files for your deliverables. Download, adapt and keep them.
Before you share it: what should a person check in your standard?
Think about it for a moment, then reveal.
Every figure in the pilot and the synthesis, against source.
The guidance note, reviewed by your data protection focal point.
The validation rules, tested on one real analysis.
Any AI-generated text, reviewed and marked as AI-assisted.
No personal or confidential data left in the portfolio sample.
Step 5 of 6How it's judged
Criterion
What good looks like
Evidence-based choices
AI is approved only for tasks where a test on your own data passed.
Defensible validation
Every AI step has a method, a sample size and a stop rule.
Standard of proof
Causal claims need evidence, whatever the AI drafts.
Data protection
Rules name the data type, the tool and the condition.
Transparency
Evaluation reports must disclose AI use and its checks.
People
The mentoring plan shows how others learn to judge AI output.
Step 6 of 6Check
Mission check+20 XP
Which deliverable shows management that an AI tool works on your organisation's own data?
Show answer
The pilot test.
End of Mission 08
MISSION09
Responsible AI: what you must never delegate blindly
Guardrails · Badge: Responsible AI Guardian
10 challenges
Mission goal: know exactly which judgements stay with you, and the control for each AI risk.
Step 1 of 6Would you approve this report?
Click every sentence you would not approve. There are five problems: an invented figure, an unsupported causal claim, an incorrect percentage, a missing indicator and a misleading interpretation.
Case study · Multi-Donor Resilience Portfolio Illustrative
An AI assistant drafted the regional management brief from the three country reports. You are asked to approve it today.
Source (country IPTTs)
Result
Country A: improved food consumption
62% (baseline 48%)
Country B: school attendance
84% (last year 81%)
Country B: children reading at grade level
41% (target 50%)
Country C: clinic visits
14,200 against a target of 18,000 (79%)
Country C: outcome 2.3
−15%; new data source at four sites from Q3
Women's leadership in committees
No indicator
Problems found: 0 of 5+40 XP
Show answer
Five problems: the 89% (it is 79%), the invented leadership figure, the unsupported claim about school feeding, the misleading reading of the 15% drop (a new data source), and the claim that every outcome is reported (reading at grade level is missing).
Step 2 of 6Your decision
Your decision+20 XP
You found the problems. Can you approve this brief for the management team?
Show answer
Return it for correction. Fix each problem, recheck every figure against the country IPTTs, then approve.
Step 3 of 6Five questions before you approve
The brief is corrected. Before you approve it, answer the five questions. Tick each one you can honestly answer yes to.
Step 4 of 6Mini-game: delegate or never?
AI can read, code and draft. It cannot be accountable. For each task: can AI help (with checks), or must you never delegate it?
Delegate?+10 XP
Listing recurring findings across twelve evaluations, with every citation checked
Show answer
AI can help. AI reads and lists; you verify each finding against its source.
Delegate?+10 XP
Judging whether findings are credible enough for donor decisions
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
First-pass coding of KII transcripts, after a pilot test on your data
Show answer
AI can help. With a passed pilot and a double-coded sample, a good use of AI.
Delegate?+10 XP
Setting the standard of proof for causal claims
Show answer
Never delegate. Yours to set, whatever the AI drafts.
Delegate?+10 XP
Transcribing and translating FGDs, with a back-translated sample
Show answer
AI can help. Useful when samples are checked against the audio and back-translated.
Delegate?+10 XP
Deciding which M&E tasks AI may support
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Drafting prompt templates for indicator narratives
Show answer
AI can help. AI can help draft them; you test them before the team uses them.
Delegate?+10 XP
Deciding how to report a figure when teams push for a positive result
Show answer
Never delegate. Weighing evidence quality against pressure for good news stays with you.
Step 5 of 6Six risks, six controls
What can go wrong when AI shapes evaluation findings? Name three risks before you look.
Think about it for a moment, then reveal.
Risk
What it looks like
Your control
Invented citations
A finding cited to a page that doesn't say it
Check every citation against the source
Overstated causes
A change credited to an activity without evidence
Your standard of proof for causal claims
Mistranslation
Meaning lost between Hausa and English
Back-translate a sample; check quotations
Unreliable metrics
An AI score that runs higher than people's ratings
Hand-rated sample; report as indicative
Privacy
Recordings or personal data in an unapproved tool
Guidance note; approved tools only
Hidden AI use
Readers don't know AI shaped the findings
Disclosure in every evaluation report
Step 6 of 6Tell readers how AI was used
Transparency is a control too. Require a short statement in every evaluation report that used AI, and adapt it to what was actually done.
AI tools were used to transcribe and translate interviews, to code qualitative data and to draft parts of the synthesis. Transcription and translation were checked on a sample of [X] interviews, coding was double-checked on [Y]% of segments ([Z]% agreement), every cited finding was verified against its source, and the final text was reviewed by the Senior M&E Specialist.
The AI-Enabled M&E Manager is covered in Tutorial 01, the AI-Enabled M&E Specialist in Tutorial 02, the AI-Assisted Evaluation Specialist in Tutorial 08 and the AI-Enabled Evaluation Manager in Tutorial 09.
Step 2 of 5What you need to show
Strengths you build on
Evaluation design and technical oversight
Setting M&E standards across programmes
Mixed methods and critical appraisal
Advising management and donors
What you need to show
A pilot test of an AI tool on your own data
Validation rules other teams follow
An evaluation where AI use was checked and disclosed
A management note on how far to trust an AI figure
Specialists you mentored to check AI output
Step 3 of 5Tool: build your LinkedIn headline
Pick the pieces that are true for you. Only claim what you have actually done.
Your headline
Step 4 of 5CV bullets and qualifications
CV bullet templates
Piloted AI coding on [80] hand-coded transcripts before approving it for a mid-term evaluation; approved [X] of [Y] codes after code-by-code agreement testing.
Wrote validation rules for AI-supported analysis (double-coding, source tracing, back-translation) applied across [N] evaluations.
Checked an AI-derived satisfaction score against a hand-rated sample of [300] and advised management on how to report it.
Drafted the organisation's guidance note on AI tools and beneficiary data with the data protection focal point.
Replace the brackets with what you have actually done.
Qualifications in the profile
Master's degree in development studies, evaluation, statistics, economics, public policy or a related field.
6–10 years in programme M&E, including leading evaluations and providing technical oversight to other M&E staff.
Useful certifications: IPDET evaluation training programme; MEAL DPro (PM4NGOs).
Step 5 of 5Check
Mission check+20 XP
What's the best way to use "AI-Enabled Senior M&E Specialist" on your CV?
Show answer
Use it to describe your capabilities, backed by evidence, unless your employer actually uses the title.
End of Mission 10
MISSION11
Your next steps
Commit and celebrate · Badge: Transition Ready
1 challenge
Mission goal: turn what you learned into a two-week plan and unlock your certificate.
Step 1 of 2Commit to your next two weeks
Pick the actions you commit to for the next two weeks. Your plan builds itself below.
Start learning
Do this
Show this
My plan
Recommended first project: the AI quality standard from mission 8, starting with a pilot test of AI coding on data your team already coded by hand.
Step 2 of 2Your completion certificate
Complete all 11 missions to unlock it. Add your name, take a screenshot and share your progress.
AI & M&E Career Transition Series · Tutorial 03
Your name
has completed the learning path
From Senior M&E Specialist to AI-Enabled Senior M&E Specialist
This is a learning-progress certificate. It records what you completed in this free tool, in your own browser; it is not independently verified and not a formal qualification. Your mission 8 project is the evidence to show an employer.
End of Mission 11
Questions and answers
What does an AI-Enabled Senior M&E Specialist do?
Acts as the technical lead for monitoring and evaluation across a country office, a large multi-donor programme or a thematic portfolio, setting methods and standards that other M&E staff apply. Leads the integration of AI across M&E functions, supervises AI-assisted evaluations and advises senior management on how far AI-supported performance measures can be trusted.
How is AI changing the Senior M&E Specialist role?
The senior specialist moves from doing most analysis personally to designing how AI-assisted analysis is done and checked across the portfolio. More time goes to setting review standards, testing AI outputs on real programme data and explaining to management what the results can and cannot support.
Which judgements should an AI-Enabled Senior M&E Specialist never hand over to AI?
The career profile lists these as human judgement:
Deciding which M&E tasks AI may support and where human analysis is required.
Judging whether evaluation findings are credible enough to inform management and donor decisions.
Weighing evidence quality against pressure for positive results from programme teams.
Setting the standard of proof for causal claims in AI-supported analysis.
Which skills should I learn first?
The learning priorities in the career profile, in order:
Evaluating AI systems. Approving AI tools for team use requires testing their accuracy on the organization's own data.
Responsible AI. Senior M&E staff are increasingly asked to decide what AI use is acceptable in evaluations.
Mixed-methods integration. AI-coded qualitative data needs careful triangulation with survey and monitoring evidence.
AI-assisted qualitative analysis. Supervising AI-assisted evaluations means knowing where transcription, translation and coding go wrong.
Communicating evidence. Management needs plain explanations of how reliable AI-supported figures are.
The profile lists 14 skills in all: Evaluation design, Results frameworks and theory of change, Indicator design, Validating AI outputs, Mixed-methods integration, Critical appraisal, AI-assisted qualitative analysis, Data quality assessment, Sampling, Evaluating AI systems, Responsible AI, Capacity strengthening, Communicating evidence and Data protection and privacy.
Which tools does this role use?
The career profile names: Claude, ChatGPT, LLM APIs (OpenAI, Anthropic, Google), NVivo, MAXQDA, Whisper (speech-to-text), Python (pandas), Power BI, SharePoint and OECD-DAC evaluation criteria. The tutorial shows how to check every AI output, whichever tool you use.
What qualifications and experience does the profile list?
Master's degree in development studies, evaluation, statistics, economics, public policy or a related field.
6–10 years in programme M&E, including leading evaluations and providing technical oversight to other M&E staff.
Useful certification: IPDET evaluation training programme and MEAL DPro (PM4NGOs).
How long does the transition take?
This tutorial suggests a 12-week plan of about 3 hours a week, in four stages: understand AI (weeks 1–2), apply AI (weeks 3–4), integrate AI (weeks 5–8) and demonstrate competence (weeks 9–12). It is a planning suggestion, not a requirement.
Is "AI-Enabled Senior M&E Specialist" an official job title?
No. It is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy. Use it to describe your capabilities, backed by evidence, unless your employer actually uses the title.
