Mission goal: understand what the AI-enhanced version of your role is and why it is emerging.
Step 1 of 5Where are you starting from?
Choose your starting point+10 XP
Where is your organisation today with AI in its M&E work?
Show answer
Every starting point is valid. Follow the mission suggested for your situation.
Step 2 of 5Meet your new role
Today's role
M&E Director
AI-enhanced role
Director of AI-Enabled M&E
Level
Director · 12+ years
How the role uses AI
Integrates AI: sets the strategy, governance and quality standards for AI in the organisation's evidence work
"Director of AI-Enabled M&E" is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy.
Step 3 of 5What changes most in your job?
Your guess+10 XP
When AI enters your organisation's evidence system, what changes most on your agenda?
Show answer
In the profile, the director's agenda now includes deciding where AI fits in the organisation's evidence system, what it may be trusted with and how it is governed. Success depends less on technical detail and more on investment choices, policies, staff capability and keeping the credibility of the organisation's evidence with donors and communities.
Step 4 of 5AI-Enabled, not AI-Dependent
The goal is not to hand your work to AI. It is to become a stronger M&E leader who uses AI and stays in charge of it. Here are the five questions, asked about your own work:
1QualityDoes AI make your organisation's evidence more credible, and do you know how it was tested?No AI tool is approved for evidence work until it has been tested on questions with known answers.
2AccountabilityWhen an AI-supported figure reaches the board or a donor, who answers for it?Every headline figure has a named person who verified it, and the policy says who reviews what.
3IndependenceDo evaluation findings stay independent when AI summarises them?AI summaries are inputs to strategy, checked by staff. You defend the integrity of the findings.
4EthicsAre communities' data protected, and is AI use disclosed?The policy bans identifiable beneficiary data in consumer tools and requires disclosure in every report.
5Professional developmentAre your teams learning to judge AI, or only to use it?Regional leads own the QA standards, and staff capability is part of the roadmap and its budget.
The five questions as the series states them
Quality: Does AI improve the quality and credibility of evaluation findings?
Accountability: Who is responsible when AI produces an incorrect interpretation?
Independence: Are evaluators maintaining their ability to think critically and make independent judgments?
Ethics: Are confidentiality, human rights and the interests of affected populations protected?
Professional development: Are we developing stronger evaluators who understand AI, or creating professionals who increasingly depend on it?
Step 5 of 5Your goal for this tutorial
By the end, you will know how to decide where AI fits in your organisation's evidence work, govern it, and keep your evidence credible, which skills to build, and you will have a 12-week plan.
Next: see which of your tasks change, and how.
End of Mission 01
MISSION02
Traditional role vs AI-enhanced role
Before and after · Badge: Role Shifter
8 challenges
Mission goal: see which responsibilities become AI-assisted, which need new skills and which stay human.
Step 1 of 4Meet your case study
Case study · Tamarind Development Alliance Illustrative
A fictional international NGO you will follow through missions 2, 3, 4, 8 and 9.
The organisation
An international NGO with programmes in 30 countries, regional M&E leads and one global results framework.
The data
22 country dashboards linked to the results framework, a library of 350 past evaluations, and a survey of the 30 country M&E teams.
The tools
Power BI, Microsoft 365 Copilot, Claude, SharePoint, LLM APIs and Excel, with the NIST AI RMF, ISO/IEC 42001, GDPR and the OECD-DAC criteria as frameworks.
Your objective
Decide where AI fits in the organisation's evidence system, govern it, and keep the evidence credible with the board, donors and communities.
Step 2 of 4Flip your tasks: today and with AI
Less compiling and firefighting, more deciding and governing. Tap each card to see how it changes.
Step 3 of 4Mini-game: sort the work
For each task, choose: AI-assisted, a new skill for you, or stays human.
Sort it+10 XP
Summarising organisational performance evidence for strategy
Show answer
AI-assisted. AI summaries are inputs to strategy, checked by staff (the profile's wording).
Sort it+10 XP
Drafting the board results summary from country dashboards
Show answer
AI-assisted. Copilot drafts; every headline figure is verified by a person.
Sort it+10 XP
Finding evidence across 350 past evaluations, with citations
Show answer
AI-assisted. A retrieval-based assistant, approved only after a test.
Sort it+10 XP
Structuring an AI policy around the NIST AI RMF
Show answer
New skill. The NIST AI RMF is one of the profile's learning priorities.
Sort it+10 XP
Assessing where AI could cause harm before you invest
Show answer
New skill. AI risk assessment is a learning priority in the profile.
Sort it+10 XP
Setting how much the organisation relies on AI for funding decisions
Show answer
Human. Listed in the profile as human judgement.
Sort it+10 XP
Making investment and staffing choices with limited budgets
Show answer
Human. Listed in the profile as human judgement.
Sort it+10 XP
Defending the independence and integrity of evaluation findings
Show answer
Human. Listed in the profile as human judgement.
Step 4 of 4Your turn: two minutes
Think about your last quarter. Which of your decisions and tasks would go in each pile?
Think about it for a moment, then reveal.
A quick way to do it: three columns on a sheet of paper, AI-assisted, new skill, stays human. Keep the list: you will use it in mission 4 (to pick a task to try) and mission 6 (to rate your gaps).
End of Mission 02
MISSION03
What AI actually changes in your daily work
Checks, risks and red flags · Badge: Red Flag Spotter
2 challenges
Mission goal: learn the three checks every AI decision needs at your level, and practise spotting problems in an AI draft.
Step 1 of 4What can AI really do for you?
Before you look: which of your tasks could AI help with, and which must stay with you?
Think about it for a moment, then reveal.
AI can assist with
Summarising survey responses from country teams
Drafting board results summaries
Finding evidence across past evaluations
Drafting policy outlines and audit terms
Can be partly automated
Compiling results from country dashboards
Searching the evaluation library
Suggesting codes for qualitative data
Tracking audit actions
New responsibilities
Setting AI policy and approved tools
Requiring a test before any approval
Defining human review and disclosure
Commissioning audits of AI workflows
Stays with you
How far to rely on AI for funding decisions
Innovation against the risk of harm
Investment and staffing choices
The independence of evaluation findings
Step 2 of 4The three checks you must always build in
At your level, one weak rule is repeated in every country. Here are the checks, one at a time.
Test before you approve. No AI tool is used for evidence until your team has tested it on questions with known answers.
Name who verifies. Every AI-supported figure or finding that leaves the organisation has a named person who checked it.
Govern the data and disclose. Approved tools only, no identifiable beneficiary data in consumer tools, and AI use stated in every report.
Step 3 of 4Mini-game: spot the red flags
Case study · Tamarind Development Alliance Illustrative
Copilot summarised the survey of country M&E teams for the executive team. Here is the source and the draft.
Click every sentence you would not send to the executive team.
Survey of country M&E teams
Teams that responded
30 of 30
Use AI assistants for drafting
21
Ask for automated partner reporting
18
Have data agreements that cover AI tools
9
Reported an AI error that reached a report
4
Red flags found: 0 of 3+30 XP
Show answer
Three sentences should not be sent: "most teams are ready to scale AI" (the survey shows use, not readiness), "AI errors are rare, so review controls can be light" (four teams reported errors), and the 30% time saving (nobody measured it).
Step 4 of 4Mission check
Your call+20 XP
A vendor's demo of an evidence assistant answers ten questions impressively. The vendor asks for approval. What do you do?
Show answer
Test it on questions with known answers, against a pass rule you set before the test.
End of Mission 03
MISSION04
Your daily tasks, step by step with AI
Six practical guides · Badge: Workflow Builder
6 challenges
Mission goal: learn how to set strategy, governance and quality standards for AI in six of your real tasks.
Case study · Tamarind Development Alliance Illustrative
Every worked example below uses this organisation's data, so you can follow one evidence system from survey to board report.
Six real tasks from your working month. Pick one and go through it in six short steps: a task prompt, then a QA prompt that checks the AI's work.
YouAIYour teamAutomated shows who does each step. Examples are illustrative.
Choose a task
Guide 1 of 6 From the career profile
Set the AI-in-M&E roadmap and its funding priorities
From the profile: the director reviews a survey of 30 country M&E teams on their current AI use, then sets three priorities for funding, such as automated partner reporting, with defined review controls for each, and presents a three-year roadmap to the executive team, with costs and risks.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
AI use grows country by country, with no shared plan.
Investment follows the most enthusiastic team, not the biggest need.
Risks come up only after something goes wrong.
How it works with AI
AI summarises the survey of country teams; an analyst checks the counts.
You choose three priorities, each with a review control.
The roadmap shows costs, risks and staffing over three years.
Step 2 of 6Step by step
Your teamSurvey the country teamsExcel
What they use AI for today, what they need, and which data agreements cover AI tools.
AIAsk AI to summarise the responsesMicrosoft 365 Copilot
Counts by use case, needs and risks raised, with no recommendations (prompt below).
Your teamCheck the countsExcel · Power BI
An analyst checks the summary against the raw responses.
YouScore the options
Value, risk to communities and data, and team readiness. You set the weights before you look at the results.
YouChoose three priorities, each with a review control
No priority is funded without a named human check.
YouPresent the roadmapPower BI
Three years, with costs, risks, staffing and what you will stop doing.
Step 3 of 6Worked example
What you give the AI Illustrative
AI use case
Teams already using
Teams asking for it
Main risk raised
Drafting reports with AI assistants
21
6
Unchecked figures
Automated partner reporting
4
18
Errors spreading across reports
AI coding of qualitative data
7
12
Codes nobody reviewed
Searchable evaluation library
0
15
Answers without sources
Chatbot for community feedback
2
3
Personal data and consent
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Summarise the survey responses below from our country M&E teams.
For each AI use case, give: teams already using it, teams asking for it, and the risks they raised, in their words.
Do not recommend priorities and do not estimate costs or savings.
Flag any count you could not confirm from the responses.
[paste responses, with no names of staff or beneficiaries]
What comes back
Automated partner reporting: 18 teams asking, 4 using. Risk raised: errors spreading across reports.
Act as an independent reviewer of a draft list of AI investment priorities.
Compare the list with the survey summary and the scoring.
List: every priority with no review control, every claim (such as savings) the evidence does not support, every priority ranked high despite a high risk to communities, and every need from the survey left out without a reason.
Do not rewrite the list. List the problems only.
Survey summary: [paste]
Scoring: [paste]
Priorities: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
None of the three priorities has a review control.
The 30% saving is not in the survey or any test. Remove it.
The chatbot ranks first although only 3 teams asked for it, and it carries personal data and consent risks.
AI coding of qualitative data (12 teams asking, 7 already using it) is left out without a reason.
Corrected output
Priority
Why
Review control
1. Automated partner reporting
18 teams asking; repetitive work
Every new indicator mapping reviewed before data enters the tracker
2. Searchable evaluation library
15 teams asking; evidence reuse
Approved only after a test on 50 questions with known answers
3. QA standard for AI-coded qualitative data
12 teams asking; 7 already using it
Minimum human review agreed with regional leads
The chatbot is not funded this cycle. It is revisited after a data protection impact assessment.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Quality: no priority is funded until it has a named review control. A fast tool that spreads errors is not an investment.
Step 6 of 6Quick check
Quick check+25 XP
The AI summary says AI "will save 30% of staff time". No team measured this. What do you do?
Show answer
Remove it, and measure the saving in a pilot before you claim it.
Try it this week
Send three questions to your regional M&E leads: which AI tools do your teams use, for which tasks, and with which data? Summarise the answers and check the counts yourself.
From the profile: the director leads a policy, mapped to the NIST AI RMF and GDPR, that lists approved tools, bans uploading identifiable beneficiary data to consumer tools and requires disclosure of AI use in evaluation reports, and reviews the draft with the legal and data protection leads.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Each country decides which AI tools to use.
Staff don't know what data they may put into a tool.
Reports rarely say how AI was used.
How it works with AI
You decide the rules; AI drafts the policy from them.
Legal and data protection leads check every legal reference.
The rules are short enough for country teams to apply.
Step 2 of 6Step by step
YouDecide the rules first
Approved tools, banned data, human review, disclosure. The AI writes them up; it does not decide them.
AIAsk AI to draft the policy outlineClaude
From your rules, linked to the NIST AI RMF functions, with gaps marked for legal review (prompt below).
Your teamReview it with the legal and data protection leads
Every legal reference is checked against the law itself, not against the AI's summary.
Your teamTest it with two country teams
Can they tell, from the policy alone, whether a task is allowed?
YouApprove and publishSharePoint
With an owner, a review date and a way to report problems.
YouTell donors and partners
What the policy covers and how AI use is disclosed.
Step 3 of 6Worked example
What you give the AI Illustrative
Your decisions (from the leadership meeting): approved tools are Microsoft 365 Copilot and Claude under organisational accounts; no identifiable beneficiary data in consumer AI tools; every AI-supported figure or finding is checked by a named person; AI use is disclosed in every evaluation report.
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Draft a short AI use policy for our M&E and evaluation work from the decisions below.
Use these sections: purpose and scope, approved tools, data rules, human review, disclosure, roles, reporting problems, review date.
For each rule, name the NIST AI RMF function it supports (Govern, Map, Measure or Manage).
Do not cite articles of any law; write [LEGAL TO CONFIRM] wherever a legal basis is needed.
Write in plain language for country M&E teams.
Decisions: [paste]
What comes back
Section
Rule (draft)
NIST AI RMF
Approved tools
Copilot and Claude under organisational accounts only
Govern
Data rules
No identifiable beneficiary data in consumer AI tools [LEGAL TO CONFIRM]
Map · Manage
Human review
Every AI-supported figure or finding is checked by a named person before use
Measure
Disclosure
Every evaluation report states how AI was used
Govern
Your legal and data protection leads confirm the legal references, not the AI.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Data rules: Anonymised data may be used in any AI tool. Under data protection law, research use of beneficiary data in AI tools is always permitted.
Human review: Recommended where possible.
Disclosure: Optional, at the evaluation manager's discretion.
The QA prompt (run it on every draft)
Act as an independent reviewer of a draft AI use policy.
Compare the draft with the decisions below.
List: every rule weaker than the decisions, every legal claim a lawyer must check, every term that is not defined (such as "anonymised"), and every rule a country team could not apply without asking.
Do not rewrite the policy. List the problems only.
Decisions: [paste]
Draft: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"Any AI tool" contradicts the approved-tools list.
"Always permitted" is a legal claim made by the AI. Your legal lead must check it, and it is not what you decided.
"Anonymised" is not defined: people in small groups can be re-identified.
"Recommended where possible" and "optional" weaken your decisions: review and disclosure are required.
Corrected output
Data rules: Only approved tools, under organisational accounts. No identifiable beneficiary data in consumer AI tools. [Definition of identifiable data agreed with the data protection lead.]
Human review: Every AI-supported figure or finding is checked by a named person before use.
Disclosure: Required in every evaluation report.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Ethics: the policy protects communities first: no identifiable beneficiary data in consumer tools, and AI use disclosed in every report.
Step 6 of 6Quick check
Quick check+25 XP
The AI's draft cites data protection law to allow beneficiary data in any AI tool. What do you do?
Show answer
Remove it. Legal claims are checked by your legal and data protection leads.
Try it this week
Write down, in five lines, the rules your teams follow today on AI and beneficiary data. Where there is no rule, that is your first policy decision.
Test an evaluation evidence assistant before you approve it
From the profile: the director sponsors a retrieval-based assistant over 350 past evaluations that answers questions with citations; it is approved only after the team tests it on 50 questions with known answers.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Past evaluations sit in folders nobody searches.
Lessons are rediscovered, or repeated, in each new programme.
Evidence for strategy comes from what people remember.
How it works with AI
An assistant searches 350 evaluations and answers with citations.
Your team tests it on 50 questions with known answers before approval.
Staff open every citation before they use an answer.
Step 2 of 6Step by step
YouDecide what it may be used for
For example: finding evidence and lessons. Not scoring programmes, and not replacing an evaluation.
Your teamPrepare the librarySharePoint
Final reports only, with sensitive annexes and personal data removed.
Your teamWrite 50 test questions with known answersExcel
From evaluations the team knows well, including questions the library cannot answer.
AutomatedRun the testLLM APIs · Claude
The assistant answers each question, with citations.
Your teamScore every answerExcel
Correct? Citation real and relevant? Did it say "not found" when it should?
YouApprove, limit or reject
Against a pass rule you set before the test.
Step 3 of 6Worked example
What you give the AI Illustrative
Test question
Known answer
Source
What did the WASH evaluation in Country 4 find on latrine use?
Use rose where households built their own latrines
WASH evaluation, Country 4, p. 22
Which evaluations assessed cash transfers for women?
Three: Countries 2, 7 and 12
Library index
What did we learn about mobile money in Country 9?
No evaluation covers it
None: it should say "not found"
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Answer the question below using only the evaluation reports in the library.
Give the answer in two sentences or fewer, then cite each report you used, with its title and page.
If the library does not answer the question, say "Not found in the library" and do not guess.
Question: [paste]
What comes back
Question
Assistant's answer
Your score
Latrine use, Country 4
Use rose where households built their own latrines (WASH evaluation, Country 4, p. 22)
Correct; citation opened and checked
Cash transfers for women
Countries 2 and 7 (both cited)
Incomplete: Country 12 is missing
Mobile money, Country 9
"Not found in the library"
Correct: it did not guess
Score all 50 before you decide. One good demo is not a test.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Test summary: 46 of 50 answers correct (92%). Recommendation: approve for all staff, including for drafting evaluation findings.
The QA prompt (run it on every draft)
Act as an independent reviewer of a test summary for an AI evidence assistant.
Compare the summary with the test log.
List: every answer counted as correct that the log marks incomplete or wrong, every citation not checked against the report, every type of question the test left out, and every use recommended beyond the agreed scope.
Do not rewrite the summary. List the problems only.
Agreed scope: [paste]
Test log: [paste]
Summary: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
Six answers the log marks incomplete are counted as correct.
Three citations point to pages that do not say what the answer claims.
No question without an answer in the library was tested, so nobody knows whether it guesses.
"Drafting evaluation findings" is outside the agreed scope.
Corrected output
Test result: 40 of 50 fully correct (80%), 6 incomplete, 4 wrong; 3 citations did not support the answer. The pass rule set before the test was not met.
Decision: not approved yet. Fix the search, add ten questions with no answer in the library, and test again. The scope stays: finding evidence and lessons, with every citation checked by the user.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Quality: the assistant is approved only after it answers known questions correctly, with citations your team opened and checked.
Step 6 of 6Quick check
Quick check+25 XP
The test shows 40 of 50 answers fully correct, and three citations that do not support the answer. Your pass rule was 45. What do you do?
Show answer
Not approved yet. Fix it and test again against the same rule.
Try it this week
Write ten questions your team can answer from past evaluations, including two the evaluations cannot answer. Keep them: they start your test set.
Verify the board results report before it goes out
From the profile: Copilot drafts a summary of organisational results from 22 country dashboards for the board; the director has the global M&E lead verify every headline figure and removes claims not backed by evaluation evidence.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Country results are compiled by hand for each board meeting.
Headline figures get rounded, merged or repeated.
Success stories stand in for evaluation evidence.
How it works with AI
Copilot drafts the summary from the 22 dashboards.
The global M&E lead verifies every headline figure.
You remove every claim no evaluation supports.
Step 2 of 6Step by step
AutomatedClose the dashboards for the periodPower BI
Note which countries have not updated, so incomplete totals are labelled.
AIAsk Copilot for a figures-only draftMicrosoft 365 Copilot
Results against targets, with [EVIDENCE] where a claim would need an evaluation (prompt below).
Your teamVerify every headline figurePower BI · Excel
The global M&E lead checks each one against the dashboards.
YouRemove claims without evidence
Keep a claim about causes or outcomes only if an evaluation supports it, and cite it.
YouAdd what the board needs to decide
Risks, gaps and what the evidence does not show.
YouSay how AI was used
One line in the report.
Step 3 of 6Worked example
What you give the AI Illustrative
Strategic goal
Result
Target
Note
Health
1,240,000 people reached
1,500,000
Education
640 schools supported
800
Livelihoods
52,000 households in savings groups
60,000
2 countries not updated
Women's economic empowerment
No data yet
New indicator
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Using only the dashboard figures below, draft a one-page results summary for the board: one paragraph per strategic goal, with result, target and percentage of target.
Label incomplete figures.
Where a sentence would need evaluation evidence (such as a cause or an outcome), write [EVIDENCE] instead.
Do not add figures, causes or comparisons that are not in the table.
[paste figures]
What comes back
Health: 1,240,000 people reached against a target of 1,500,000 (83%).
Education: 640 of 800 schools supported (80%). [EVIDENCE]
Livelihoods: 52,000 of 60,000 households in savings groups (87%); incomplete, two countries have not updated.
Women's economic empowerment: no data yet; the indicator is new this year.
Each [EVIDENCE] gap is filled from an evaluation, with its citation, or the sentence is removed.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Health programmes reached 1.4 million people (93% of target), driven by our new AI-assisted monitoring. Education and livelihoods are on track.
The QA prompt (run it on every draft)
Act as an independent reviewer of a board results summary.
Compare the summary with the dashboard figures and the list of evaluations.
List: every figure that differs from the dashboards, every causal or outcome claim no evaluation supports, every strategic goal left out, and every "on track" the figures do not support.
Do not rewrite the summary. List the problems only.
Dashboards: [paste]
Evaluations: [paste]
Summary: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
1.4 million and 93% are wrong: the dashboards show 1,240,000 of 1,500,000 (83%).
"Driven by AI-assisted monitoring" is a causal claim no evaluation supports.
Women's economic empowerment is left out.
"Livelihoods on track" ignores that two countries have not updated their dashboards.
Corrected output
Health: 1,240,000 people reached against a target of 1,500,000 (83%). Education: 640 of 800 schools supported (80%). Livelihoods: 52,000 of 60,000 households in savings groups (87%), incomplete: two countries have not updated.
Women's economic empowerment: no data yet (new indicator). Figures verified by the global M&E lead; the draft was prepared with AI assistance.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Accountability: a named person verifies every headline figure before the board sees it, and claims without evaluation evidence come out.
Step 6 of 6Quick check
Quick check+25 XP
Copilot writes that results improved "thanks to AI-assisted monitoring". No evaluation says so. What do you do?
Show answer
Remove it, and commission evidence if the question matters.
Try it this week
Take your last board report. Pick five headline figures and trace each one to its source. Note how long it took and what you found.
Agree minimum QA and disclosure standards for AI-coded data
From the profile: meeting regional M&E leads to agree minimum QA standards for AI-coded qualitative data; defining minimum human-review and disclosure standards for AI-supported evaluations; discussing with a major donor how AI-assisted analysis will be disclosed in its evaluations.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Each region checks qualitative coding its own way.
Nobody knows how much of the AI coding was checked.
Donors learn about AI use after the fact, if at all.
How it works with AI
AI suggests codes for interviews and open answers.
Regional leads agree a minimum human review for every study.
Every report says how AI was used and checked, as agreed with the donor.
Step 2 of 6Step by step
AIDraft options for the standardClaude
From your principles: review levels, handling disagreements, disclosure wording (prompt below).
Your teamAgree it with regional leads
What is the minimum human review? Who resolves disagreements?
YouSet the minimum, not the maximum
Regions can do more; nobody does less.
Your teamPilot it on one studyExcel
Record how often people disagreed with the AI codes.
YouAgree disclosure with the donor
What the report will say about AI use, before the evaluation starts.
YouPublish the standardSharePoint
With an owner and a review date.
Step 3 of 6Worked example
What you give the AI Illustrative
Your principles: people write the codebook; a person reviews AI codes before they are used; people resolve disagreements; AI use is disclosed in the report; no transcripts with names go into consumer AI tools.
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Draft a one-page minimum standard for AI-assisted coding of qualitative data in our evaluations, from the principles below.
Cover: who writes the codebook, what share of excerpts a person double-codes, what happens when the person and the AI disagree, data rules for transcripts, and the disclosure sentence for the report.
Where a number is needed, write [AGREE WITH REGIONAL LEADS] instead of choosing one.
Principles: [paste]
What comes back
Topic
Draft rule
Codebook
Written and approved by the evaluation team
Human double-coding
[AGREE WITH REGIONAL LEADS] of excerpts, chosen at random
Disagreements
Resolved by a person; patterns recorded and the codebook revised
Transcripts
Names and identifying details removed before any AI tool
Disclosure
"AI suggested codes; [share] were double-coded by the team; all themes were reviewed by people."
The numbers are agreed with your regional leads, not chosen by the AI.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Standard: AI codes may be used without review where the tool's confidence score is above 90%. Disclosure is needed only if the donor asks.
The QA prompt (run it on every draft)
Act as an independent reviewer of a draft QA standard for AI-coded qualitative data.
Compare the draft with the principles.
List: every case where AI codes are used without human review, every rule that relies on the tool's own confidence, every gap in data protection, and every disclosure rule weaker than the principles.
Do not rewrite the standard. List the problems only.
Principles: [paste]
Draft: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"Without review" breaks the first principle: a person reviews AI codes before they are used.
A tool's own confidence score is not a check by a person.
Transcripts are not mentioned: names and identifying details must be removed before any AI tool.
Disclosure "only if the donor asks" contradicts the principle: AI use is always disclosed.
Corrected output
Every AI code is reviewed by a person before use; [AGREE WITH REGIONAL LEADS] of excerpts are double-coded at random, and disagreements are resolved by people.
Transcripts have names and identifying details removed before any AI tool. Every report says how AI was used and how it was checked.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Independence: people write the codebook and resolve disagreements with the AI. A confidence score is not a review.
Step 6 of 6Quick check
Quick check+25 XP
A regional lead says AI codes with a high confidence score don't need review. What do you say?
Show answer
The minimum human review applies to every study, whatever the tool's confidence.
Try it this week
Ask one regional lead how AI-coded data was checked in their last study. Write down the answer: it is your baseline.
Commission an audit of AI-supported workflows and act on it
From the profile: commissioning audits of AI-supported M&E workflows and acting on their findings; commissioning periodic reviews of compliance with quality standards; making sure management responses are followed up.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Nobody checks whether the AI rules are followed.
Problems surface when a donor finds an error.
Audit reports are filed, not acted on.
How it works with AI
AI helps draft the terms of reference for the audit.
An independent team tests workflows against the policy.
You agree a management response and follow it up.
Step 2 of 6Step by step
YouDecide what to audit
For example: a sample of AI-supported reports from three regions over the last year.
AIDraft the terms of referenceClaude
Questions, sample, evidence and independence (prompt below).
YouMake it independent
Auditors who did not build or run the workflows.
Your teamRun the audit
Trace AI-supported figures to source; check review records and disclosure.
YouWrite the management responseExcel
Accept, partly accept or reject each finding, with an owner and a date.
YouFollow it upPower BI
Report progress to the executive team until every action is closed.
Step 3 of 6Worked example
What you give the AI Illustrative
Scope you set: 20 AI-supported reports from three regions, last 12 months. Questions: were AI-supported figures checked by a named person? Was AI use disclosed? Did any personal data go into unapproved tools?
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Draft terms of reference for an audit of our AI-supported M&E workflows, from the scope below.
Include: audit questions, how reports and workflows are sampled, the evidence auditors should collect, independence requirements, and the format for findings (finding, evidence, risk, recommendation).
Do not draft findings or conclusions.
Scope: [paste]
What comes back
Audit question
Evidence to collect
Were AI-supported figures verified by a named person?
Review records for each sampled report
Was AI use disclosed?
The disclosure note in each report
Did personal data go into unapproved tools?
Tool access records and interviews with staff
Do the review controls work?
A sample of figures re-checked against source
The findings come from the auditors. The responses come from you.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Audit finding
Draft response (AI)
6 of 20 reports did not disclose AI use
Noted.
Two teams pasted interview transcripts into a consumer chatbot
Teams reminded of the policy.
AI-coded themes in 4 studies had no human review
Accepted.
The QA prompt (run it on every draft)
Act as an independent reviewer of a draft management response to an audit.
Check each response below against its finding.
List: every response with no action, owner or date; every serious finding (such as personal data in unapproved tools) answered only with a reminder; and every finding that needs a decision from leadership.
Do not rewrite the responses. List the problems only.
Findings: [paste]
Responses: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"Noted" and "Accepted" name no action, owner or date.
Transcripts in a consumer chatbot may be a data breach. A reminder is not enough: the data protection lead must assess it.
Missing human review in 4 studies needs a decision: are the findings in those reports still reliable?
Corrected output
Finding
Response
Owner · date
6 of 20 reports did not disclose AI use
Accept. Add the disclosure note to the report template; check the next 10 reports.
Global M&E lead · next quarter
Transcripts in a consumer chatbot
Accept. The data protection lead assesses the breach and any duty to notify; access to unapproved tools is blocked.
Data protection lead · now
No human review in 4 studies
Accept. Themes are re-checked before the findings are used again; the teams are trained.
Regional M&E leads · next quarter
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Professional development: audit findings become changes and training that your teams own, with dates, not a report on a shelf.
Step 6 of 6Quick check
Quick check+25 XP
The audit finds that interview transcripts were pasted into a consumer chatbot. What is your response?
Show answer
Treat it as a possible breach, with the data protection lead, and fix the cause.
Try it this week
Pick three recent reports that used AI. Check: was AI use disclosed, and who verified the figures? That is a one-hour pilot audit.
Pick a recurring task that is not covered above, such as preparing a donor working group briefing on AI and evidence, and use this prompt to draft your own step-by-step guide.
I am the director of M&E in an international organisation. Here is a recurring task I do: [describe the task, how often, who is involved, which tools and data].
Break it into numbered steps. For each step, say who should do it (me, a regional lead, an analyst, the legal or data protection lead, or an AI assistant), which tool, and what could go wrong.
Mark every step where a person must check the result before it is used, and say what they should check.
Do not suggest putting personal data into an AI tool.
End of Mission 04
MISSION05
The skills you need to develop
Skill inventory · Badge: Skill Builder
1 challenge
Mission goal: find out which of the 13 profile skills you already have and which need new learning.
Step 1 of 5Guess first
Your guess+20 XP
Which skill is the first learning priority for this role in the career profile?
Show answer
Responsible AI, followed by data governance, the NIST AI RMF, AI risk assessment and change management.
Step 2 of 5Skills you bring with you
Tick the ones you already have. Open a skill to see how to build it.
Your skills match
0 of 13
A. Existing M&E and leadership competencies
Transfers from your role
Long-term priorities, phased plans and trade-offs, now applied to where AI fits in the evidence system.
Build it: Write the three questions your AI-in-M&E roadmap must answer.
Transfers from your role
Leading regional and thematic M&E leads through change.
Build it: Ask each regional lead for one AI risk they worry about.
Transfers from your role
The organisation-wide results framework and the systems that report against it.
Build it: Trace one board figure from the framework down to country data.
Transfers from your role
Knowing what evidence can and cannot show, so AI summaries don't overclaim.
Build it: Check one AI summary for claims the evaluation design cannot support.
Transfers from your role
Making evaluation findings feed strategic decisions. The evidence library builds on it.
Build it: List five past evaluations your next strategy should draw on.
Transfers from your role
Investment and staffing choices with limited budgets.
Build it: Cost one AI priority, including staff time for human review.
Transfers from your role
Boards, donors, peers and communities, who all need to trust your evidence.
Build it: Ask one donor what they need to know about your AI use.
Transfers from your role
Clear results reporting to boards and donors, without overclaiming.
Build it: Remove every unsupported claim from one results summary.
Transfers from your role
Judging the strength of evidence, including AI outputs.
Build it: Appraise one AI-drafted summary against its sources.
Step 3 of 5New skills for the AI era
These are the ones most M&E directors still need to learn.
Your skills match
0 of 13
B. New AI and governance competencies
Needs new learning
Fairness, transparency and oversight. The profile's first learning priority: you set the rules others follow.
Build it: Write the five rules your teams must follow when they use AI.
Needs new learning
Data ownership, access and sharing agreements across countries, which AI use depends on (a learning priority).
Build it: Check which country data agreements cover AI tools.
Needs new learning
Seeing where AI could cause harm before you invest (a learning priority).
Build it: Assess one AI priority for risks to communities and to credibility.
Needs new learning
Bringing regional teams and partners along with new rules and tools (a learning priority).
Build it: Plan how two regions will adopt one new standard.
Step 4 of 5Your five learning priorities
In this order, from the career profile:
Responsible AI. The director sets the rules others follow and must understand fairness, transparency and oversight issues.
Data governance. AI use depends on clear data ownership, access and sharing agreements across countries.
NIST AI Risk Management Framework. A recognised risk framework gives AI governance a structure donors and boards understand.
AI risk assessment. Investment and policy decisions need a clear view of where AI could cause harm.
Change management. Organisation-wide adoption depends on bringing regional teams and partners along.
You don't need to build AI systems. You need to know enough to ask the right questions, set the rules and judge the evidence. The NIST AI RMF and ISO/IEC 42001 give your governance a structure boards and donors recognise; the OECD-DAC criteria still frame your evaluations.
End of Mission 05
MISSION06
Interactive skills gap assessment
Know your gaps · Badge: Self-Aware Analyst
1 challenge
Mission goal: measure your readiness in four areas and find your top three gaps.
Step 1 of 3How it works
10 quick statements, about 3 minutes. Rate yourself honestly: this is a learning tool, not a test, and your answers stay in your browser.
Beginner: little or no experience
Developing: I understand it but need practice
Competent: I can apply it independently
Step 2 of 3Questions 1 to 5
Step 3 of 3Questions 6 to 10
Prefer to score it yourself?
Beginner = 0, Developing = 1, Competent = 2. Add them up (maximum 20): 0–7 Getting started · 8–14 Developing · 15–20 Ready to lead AI-enabled M&E. For every Beginner answer, start with the matching action:
Use the routine in mission 4, guide 1.
Cost one priority in the roadmap template.
Write them down (mission 4, guide 2).
Use the routine in mission 4, guide 2.
Ask your regional leads and your data protection lead.
Use the routine in mission 4, guide 3.
Use the routine in mission 4, guide 4.
Use the routine in mission 4, guide 5.
Plan the adoption of one standard with two regions.
Mission goal: commit to a realistic 12-week plan and track it week by week.
Step 1 of 5Weeks 1–2: Understand AI
About 3 hours a week (a planning suggestion). Tick steps as you do them: +5 XP each, saved in this browser. The mission completes when every step in all four stages is ticked.
Stage 1 · Weeks 1–20 / 5 done
Understand AI
See where AI already is in your evidence system, and what it gets wrong.
Loot: a map of current AI use, an error log and a list of governance gaps.
Step 2 of 5Weeks 3–4: Apply AI
Stage 2 · Weeks 3–40 / 4 done
Apply AI
Try the first governance steps, with checks.
Loot: a checked survey summary, a policy outline and a first test set.
Step 3 of 5Weeks 5–8: Integrate AI
Stage 3 · Weeks 5–80 / 5 done
Integrate AI
Make the rules and controls part of how the organisation works.
Loot: funded priorities, an approved policy and agreed standards.
Step 4 of 5Weeks 9–12: Demonstrate competence
Stage 4 · Weeks 9–120 / 5 done
Demonstrate competence
Show the strategy works and others can carry it.
Loot: a governance package, a roadmap decision and a development plan.
Step 5 of 5Check
Mission check+20 XP
In which stage do you first draft the AI use policy outline?
Show answer
Stage 2 (weeks 3–4).
End of Mission 07
MISSION08
Your practical learning project
Boss level · Badge: AI Strategy Lead
1 challenge + 8 deliverables
Mission goal: produce a portfolio piece that proves you can lead AI in M&E responsibly.
Step 1 of 6Your boss-level project
Boss level project
Lead one AI-in-M&E governance cycle, from priorities to audit
Objective: show you can decide where AI fits in your organisation's evidence system, govern it, test it before approval, and keep the evidence that reaches the board credible.
Case study · Tamarind Development Alliance Illustrative
No organisational data of your own to use? Build the cycle for this NGO, using the examples from mission 4.
Context: your organisation, or one region of it. Use aggregated figures and anonymised examples only.
Resources: your results framework, the last board report, country dashboards, your data protection rules, the NIST AI RMF, and an approved AI assistant.
Step 2 of 6Deliverables, one by one
Deliverables checklist0 / 8 done
8 steps to a portfolio piece
The mission and the certificate unlock only when all 8 deliverables are ticked.
Step 3 of 6Templates to start from
Four ready-made files for your deliverables. Download, adapt and keep them.
Before you share it: what should a person check in your governance package?
Think about it for a moment, then reveal.
Every legal reference, by the legal or data protection lead.
Every count in the survey summary, against the responses.
Every headline figure, against the dashboards.
Any AI-generated text, reviewed and marked as AI-assisted.
No personal data in any AI step or shared file.
Step 5 of 6How it's judged
Criterion
What good looks like
Evidence-based priorities
Priorities follow the survey and a scoring set in advance, not the loudest request.
Controls before funding
Every funded priority and approved tool has a named review control.
Tested, not trusted
No AI tool approved without a test on known answers.
Credible reporting
Every board figure verified; no claim without evaluation evidence.
Protects communities
No identifiable beneficiary data in consumer tools; AI use disclosed.
Acted on
Audit findings have owners, dates and follow-up.
Step 6 of 6Check
Mission check+20 XP
Which deliverable shows that an AI tool earned its approval?
Show answer
The test plan and results.
End of Mission 08
MISSION09
Responsible AI: what you must never delegate blindly
Guardrails · Badge: Responsible AI Guardian
10 challenges
Mission goal: know exactly which judgements stay with you, and the control for each AI risk.
Step 1 of 6Would you approve this report?
Click every sentence you would not send. There are five problems: an invented figure, an unsupported causal claim, an incorrect percentage, a missing indicator and a misleading interpretation.
Case study · Tamarind Development Alliance Illustrative
Copilot drafted the annual results summary for the board from the country dashboards. The board papers go out today.
Source (dashboards and evaluations, year to date)
Result
Country dashboards updated
20 of 22
Health: people reached
1,240,000 of 1,500,000 (83%)
Education: schools supported
640 of 800 (80%)
Evaluations completed
14 (3 rated results "partly achieved")
Women's economic empowerment
No data yet (new indicator)
Household income
No indicator
Problems found: 0 of 5+40 XP
Show answer
Five problems: the 93% (it is 83%), the invented income figure, the unsupported claim that AI-assisted monitoring caused the results, the misleading reading of the evaluations (three rated results "partly achieved"), and the claim that all strategic goals are covered (women's economic empowerment is missing).
Step 2 of 6Your decision
Your decision+20 XP
You found the problems. Can the summary go to the board?
Show answer
Correct it first. Fix each problem, have every figure verified, then send it.
Step 3 of 6Five questions before you send it
The summary is corrected. Before it goes to the board, answer the five questions. Tick each one you can honestly answer yes to.
Step 4 of 6Mini-game: delegate or never?
AI can summarise, draft and search. It cannot be accountable. For each task: can AI help (with checks), or must you never delegate it?
Delegate?+10 XP
Summarising survey responses from country teams
Show answer
AI can help. With the counts checked against the responses.
Delegate?+10 XP
Setting how much the organisation relies on AI for funding decisions
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Drafting the board results summary from the dashboards
Show answer
AI can help. With every headline figure verified by a named person.
Delegate?+10 XP
Making investment and staffing choices with limited budgets
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Finding evidence across past evaluations, with citations
Show answer
AI can help. After a test on known answers, with every citation checked.
Delegate?+10 XP
Defending the independence and integrity of evaluation findings
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Drafting the outline of the AI use policy
Show answer
AI can help. From rules you decided, with legal references checked by the legal lead.
Delegate?+10 XP
Balancing innovation against the risk of harm to communities
Show answer
Never delegate. Listed in the profile as human judgement.
Step 5 of 6Six risks, six controls
What can go wrong when AI enters your organisation's evidence system? Name three risks before you look.
Think about it for a moment, then reveal.
Risk
What it looks like
Your control
Untested tools
An assistant approved after a vendor demo
A test on known answers, with a pass rule set first
Unverified figures
A wrong headline figure in the board report
A named person verifies every headline figure
Invented legal claims
A policy that cites law the AI got wrong
Legal and data protection leads check every reference
Personal data exposed
Interview transcripts in a consumer chatbot
Approved tools only; identifiable data banned from consumer tools
Hidden AI use
Donors learn about AI use after the fact
Disclosure in every report, agreed with donors in advance
Weakened independence
AI summaries replacing evaluation findings
AI summaries are inputs, checked by staff; you defend the findings
Step 6 of 6Tell readers how AI was used
Transparency is a control too. Add a short note to every board report and evaluation, and adapt it to what was actually done.
An AI assistant (Microsoft 365 Copilot) drafted this summary from the organisation's country dashboards. Every headline figure was verified by [name], Global M&E Lead, and claims about causes or outcomes are based only on the evaluations cited. No identifiable beneficiary data was processed by AI tools. AI use follows our AI use policy for evidence work.
This is a learning-progress certificate. It records what you completed in this free tool, in your own browser; it is not independently verified and not a formal qualification. Your mission 8 project is the evidence to show an employer.
End of Mission 11
Questions and answers
What does a Director of AI-Enabled M&E do?
Leads monitoring, evaluation and learning for the whole organization, setting its M&E strategy, policies and quality standards and representing it to boards, donors and peers. Develops the organizational strategy for AI in M&E, oversees the intelligent monitoring systems that result, and establishes the governance and quality rules that make AI-supported evidence trustworthy.
How is AI changing the M&E Director role?
The director's agenda now includes deciding where AI fits in the organization's evidence system, what it may be trusted with and how it is governed. Success depends less on technical detail and more on investment choices, policies, staff capability and keeping the credibility of the organization's evidence with donors and communities.
Which judgements should a Director of AI-Enabled M&E never hand over to AI?
The career profile lists these as human judgement:
Setting how much the organization relies on AI for evidence that informs funding and programme decisions.
Balancing innovation against the risk of harm to communities and to the organization's credibility.
Making investment and staffing choices with limited budgets.
Defending the independence and integrity of evaluation findings.
Which skills should I learn first?
The learning priorities in the career profile, in order:
Responsible AI. The director sets the rules others follow and must understand fairness, transparency and oversight issues.
Data governance. AI use depends on clear data ownership, access and sharing agreements across countries.
NIST AI Risk Management Framework. A recognised risk framework gives AI governance a structure donors and boards understand.
AI risk assessment. Investment and policy decisions need a clear view of where AI could cause harm.
Change management. Organization-wide adoption depends on bringing regional teams and partners along.
The profile lists 13 skills in all: Strategic planning, Team leadership, Responsible AI, Data governance, AI risk assessment, Performance and results-based management, Evaluation design, Learning and knowledge management, Change management, Budget and resource management, Stakeholder engagement, Communicating evidence and Critical appraisal.
Which tools does this role use?
The career profile names: Power BI, Microsoft 365 Copilot, Claude, SharePoint, LLM APIs (OpenAI, Anthropic, Google), OECD-DAC evaluation criteria, NIST AI Risk Management Framework, ISO/IEC 42001 (AI management systems), GDPR and Excel. The tutorial shows how to check every AI output, whichever tool you use.
What qualifications and experience does the profile list?
Master's degree or PhD in evaluation, development studies, economics, public policy, statistics or a related field.
12+ years in M&E, including senior leadership of a multi-country M&E function and organizational policy development.
Useful certification: IPDET evaluation training programme and IAPP AI Governance Professional (AIGP).
How long does the transition take?
This tutorial suggests a 12-week plan of about 3 hours a week, in four stages: understand AI (weeks 1–2), apply AI (weeks 3–4), integrate AI (weeks 5–8) and demonstrate competence (weeks 9–12). It is a planning suggestion, not a requirement.
Is "Director of AI-Enabled M&E" an official job title?
No. It is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy. Use it to describe your capabilities, backed by evidence, unless your employer actually uses the title.
