Mission goal: understand what the AI-enhanced version of your role is and why it is emerging.
Step 1 of 5Where are you starting from?
Choose your starting point+10 XP
Where are you today with AI in your evaluation portfolio?
Show answer
Every starting point is valid. Follow the mission suggested for your situation.
Step 2 of 5Meet your new role
Today's role
Evaluation Manager
AI-enhanced role
AI-Enabled Evaluation Manager
Level
Manager · 7–12 years
How the role uses AI
Integrates AI: coordinates AI-supported evaluation processes, manages shared analytical workflows and is accountable for the quality of AI-supported results
"AI-Enabled Evaluation Manager" is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy.
Step 3 of 5What changes most in your job?
Your guess+10 XP
Evaluation teams now use AI for transcription, coding, synthesis and drafting. What changes most for the manager?
Show answer
In the profile, the manager must assess both the evaluation and the AI-supported process behind it: set the rules for AI use in commissioned work, run the shared workflows and hold the line on evidence quality when faster analysis makes weak findings easier to produce.
Step 4 of 5AI-Enabled, not AI-Dependent
The goal is not to hand your work to AI. It is to become a stronger evaluation manager who uses AI and stays in charge of it. Here are the five questions, asked about your own work:
1QualityDo AI-supported evaluations in your portfolio produce stronger evidence, or only faster reports?No report is accepted until its findings are checked against the evidence annex and its AI validation plan was followed.
2AccountabilityWhen a commissioned finding turns out to be wrong, can you show who checked what?Contracts require a methods annex on AI use, and your quality review of each report is on file.
3IndependenceDo you still judge methods and evidence yourself, including when findings are unwelcome?You decide whether a report is good enough, and an AI check never becomes a way to soften a finding.
4EthicsDo interview data go only where your contracts allow?Every ToR sets data handling rules for AI, shared workflows have restricted access, and consent covers AI processing.
5Professional developmentIs your evaluation team getting stronger, or only more dependent on tools?Guidance is updated from what your reviews find, and staff learn to check AI work, not only to run it.
The five questions as the series states them
Quality: Does AI improve the quality and credibility of evaluation findings?
Accountability: Who is responsible when AI produces an incorrect interpretation?
Independence: Are evaluators maintaining their ability to think critically and make independent judgments?
Ethics: Are confidentiality, human rights and the interests of affected populations protected?
Professional development: Are we developing stronger evaluators who understand AI, or creating professionals who increasingly depend on it?
Step 5 of 5Your goal for this tutorial
By the end, you will know how to set rules for AI in the evaluations you commission and check the AI-supported work before you accept it, which skills to build, and you will have a 12-week plan.
Next: see which of your tasks change, and how.
End of Mission 01
MISSION02
Traditional role vs AI-enhanced role
Before and after · Badge: Role Shifter
8 challenges
Mission goal: see which responsibilities become AI-assisted, which need new skills and which stay human.
Step 1 of 4Meet your case study
Case study · Foundation evaluation portfolio Illustrative
A fictional evaluation portfolio you will follow through missions 2, 3, 4, 8 and 9.
The portfolio
The evaluation unit of a fictional foundation: five evaluations this year, three commissioned to external firms, and a team of three.
The workflows
A shared transcription and translation workflow used by four evaluations, a library of 46 past evaluation reports, and a tracker of 120 recommendations from eight evaluations.
The tools
Claude, Microsoft 365 Copilot, Whisper, LLM APIs, MAXQDA, SharePoint, Excel and Power BI, with the OECD-DAC criteria.
Your objective
Accept only reports whose methods and evidence hold up, with every AI step in the portfolio set by rules, checked and disclosed.
Step 2 of 4Flip your tasks: today and with AI
Less chasing and reading, more setting rules and checking. Tap each card to see how it changes.
Step 3 of 4Mini-game: sort the work
For each task, choose: AI-assisted, a new skill for you, or stays human.
Sort it+10 XP
Listing every finding in a draft report and the evidence it cites
Show answer
AI-assisted. One of the profile's AI tasks, with every finding without a source reviewed by you.
Sort it+10 XP
Summarising status updates on recommendations
Show answer
AI-assisted. Copilot summarises; you confirm overdue items with programme leads.
Sort it+10 XP
Transcribing and translating interviews across evaluations
Show answer
AI-assisted. A shared workflow, with restricted access and a monthly quality sample.
Sort it+10 XP
Judging whether evaluators checked their AI-supported analysis
Show answer
New skill. Validating AI outputs is the profile's first learning priority.
Sort it+10 XP
Understanding what an AI-searchable library can miss
Show answer
New skill. Retrieval-augmented knowledge systems are a learning priority in the profile.
Sort it+10 XP
Deciding whether a report's methods and evidence are good enough to accept
Show answer
Human. Listed in the profile as human judgement.
Sort it+10 XP
Protecting evaluation independence when findings are unwelcome
Show answer
Human. Listed in the profile as human judgement.
Sort it+10 XP
Choosing which programmes to evaluate and with what budget
Show answer
Human. Listed in the profile as human judgement.
Step 4 of 4Your turn: two minutes
Think about the evaluations you manage now. Which tasks would go in each pile?
Think about it for a moment, then reveal.
A quick way to do it: three columns on a sheet of paper, AI-assisted, new skill, stays human. Keep the list: you will use it in mission 4 (to pick a task to try) and mission 6 (to rate your gaps).
End of Mission 02
MISSION03
What AI actually changes in your daily work
Checks, risks and red flags · Badge: Red Flag Spotter
2 challenges
Mission goal: learn the three checks every AI-supported evaluation needs, and practise spotting problems in an AI summary.
Step 1 of 4What can AI really do for you?
Before you look: which of your tasks could AI help with, and which must stay with you?
Think about it for a moment, then reveal.
AI can assist with
Listing findings and their evidence in draft reports
Summarising recommendation status updates
Drafting ToR clauses
Searching past evaluations
Can be partly automated
Transcription and translation across evaluations
Cost and usage reports
Tracker dashboards
Reminders to programme leads
New responsibilities
Setting AI rules in ToRs and contracts
Assessing evaluators' AI validation plans
Running shared workflows: access, costs, quality
Updating guidance from what reviews find
Stays with you
Accepting or returning a report
Protecting independence
Choosing what to evaluate, and the budget
Negotiating management responses
Step 2 of 4The three checks you must always build in
Faster analysis makes weak findings easier to produce. Here are the checks, one at a time.
Check the process, not only the report. Ask how AI was used and validated: the sample, the agreement scores, the methods annex.
Check against the evidence. Findings against the evidence annex, statuses against the updates, library answers against the reports.
Control where data goes. Contracts say which AI services may receive interview data, access is restricted, and AI use is disclosed.
Step 3 of 4Mini-game: spot the red flags
Case study · Foundation evaluation portfolio Illustrative
An AI assistant summarised the AI section of a firm's inception report for your review. Here is the source and the summary.
Click every sentence you would not rely on.
Inception report, AI section
Interviews
90 KIIs
Coding
LLM, with 15% double-coded by a team member (14 transcripts)
Agreement
To be reported for each code
Translation
Machine translation; no check described
Red flags found: 0 of 3+30 XP
Show answer
Three sentences are wrong: "all transcripts will be double-coded" (only 15%, 14 of 90), "translations will be checked by a fluent speaker" (no check is described), and "the approach meets the foundation's AI requirements" (the report doesn't say so, and that judgement is yours).
Step 4 of 4Mission check
Your call+20 XP
A firm tells you its AI coding is "fully validated". What do you ask for?
Show answer
Ask for the sample, the agreement scores for each code and a methods annex describing prompts and models.
End of Mission 03
MISSION04
Your daily tasks, step by step with AI
Six practical guides · Badge: Workflow Builder
6 challenges
Mission goal: learn how to set rules for AI in your evaluations and check AI-supported work, with a human check at every step.
Case study · Foundation evaluation portfolio Illustrative
Every worked example below uses this portfolio, so you can follow one evaluation unit from the ToR to the board report.
Six real tasks from managing evaluations. Pick one and go through it in six short steps: a task prompt, then a QA prompt that checks the AI's work.
YouAIYour teamAutomated shows who does each step. Examples are illustrative.
Choose a task
Guide 1 of 6 From the career profile
Write AI clauses into ToRs and contracts
From the profile: adding AI use, data handling and disclosure clauses to evaluation ToRs and contracts; setting requirements for AI use in commissioned evaluations, such as data handling, validation and disclosure; scoring technical proposals against the ToR criteria.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
ToRs and contracts say nothing about AI.
Firms use whatever tools they like.
Nobody knows where interview data went.
How it works with AI
AI drafts clauses from your organisation's requirements.
You check them with the data protection focal point and procurement.
Proposals are scored on their AI validation plan.
Step 2 of 6Step by step
YouList your requirements
Data handling, validation and disclosure, from your organisation's AI and data protection rules.
AIAsk Claude to draft the clausesClaude
One clause per requirement, each one checkable (prompt below).
YouCheck them with the data protection focal point
Is the law right? Are the approved services named?
Your teamReview the wording with procurement
So the clauses fit your contract templates.
YouAdd a scoring criterion for the AI validation planExcel
Proposals that use AI must say how they will check it.
YouBrief bidders
Explain the clauses at the bidders' meeting or in the Q&A.
Step 3 of 6Worked example
What you give the AI Illustrative
Your requirements (excerpt): interview data may be processed only in AI services the foundation approves; AI coding and translation must be validated on a sample; AI use must be disclosed in the inception and final reports.
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Draft contract clauses for an evaluation ToR from the requirements below.
Write one clause per requirement. Each clause must say what is required, who does it and how it is shown (for example, in which report).
Do not add requirements that are not listed. Mark any legal reference with [CHECK].
[paste requirements]
What comes back
Clause
What it requires
Data handling
Interview data only in AI services approved in writing; never in consumer tools
Validation
15% of AI-coded transcripts double-coded, agreement reported for each code; translation checked on a sample
Disclosure
A methods annex in the inception and final reports: tools, models, prompts and checks
Three clauses, each one you can check in the inception report.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Draft clauses (AI): The evaluator may use AI tools as appropriate. AI use must comply with GDPR. AI outputs will be reviewed by the team. The evaluator is responsible for the accuracy of AI outputs.
The QA prompt (run it on every draft)
Act as an independent reviewer of draft contract clauses on AI use in an evaluation.
Compare the clauses with the requirements below.
List: every clause that sets no rule anyone could check, every requirement with no clause, every legal reference that may not apply to this contract, and every clause that does not say what must be reported and where.
Do not rewrite the clauses. List the problems only.
Requirements: [paste]
Clauses: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"As appropriate" sets no rule: it doesn't say which AI services are approved or where data may be processed.
"Must comply with GDPR" may not be the law that applies. Check with your data protection focal point.
"Reviewed by the team" doesn't say how much, by whom, or how agreement is reported.
There is no disclosure clause: nothing requires a methods annex in the inception or final report.
Corrected output
Data handling: interview audio and transcripts are processed only in AI services the foundation has approved in writing, and never in consumer tools.
Validation: any AI coding is validated by double-coding at least 15% of transcripts, with agreement reported for each code; machine translation is checked on a sample.
Disclosure: the inception and final reports include a methods annex describing the tools, models, prompts and checks. Reviewed by the data protection focal point on [date].
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Ethics: every ToR and contract says where interview data may go, and which AI services are approved.
Step 6 of 6Quick check
Quick check+25 XP
Of three proposals, the cheapest proposes LLM coding but says nothing about checking it. What do you do?
Show answer
Score it on the validation plan like any other criterion, and ask for the plan.
Try it this week
Take your next ToR and add three clauses: data handling, validation and disclosure. Ask your data protection focal point to review them.
From the profile: an evaluation firm proposes LLM coding of 90 interviews; the manager requires double-coding of 15%, a list of codes with agreement scores and a methods annex describing prompts and models; assessing evaluators' AI validation plans during inception review.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Inception reviews look at the design, not at the AI steps.
"Quality assured by the team leader" is accepted as it is.
Problems with AI coding appear only in the draft report.
How it works with AI
AI lists every AI step in the inception report and what it says about checks.
You compare each step with your contract clauses.
You ask for what is missing before you approve.
Step 2 of 6Step by step
YouRead the design against the ToROECD-DAC
Questions, the OECD-DAC criteria, methods and budget. That stays with you.
AIAsk Claude to list every AI stepClaude
Tool, data, check, and how it is reported (prompt below).
YouCompare each step with your clauses
Approved service? Validation sample? Disclosure?
YouAsk for what is missing
For example: double-coding of 15%, agreement scores for each code, a methods annex.
Your teamShare it with the reference group
Collect their comments on the design.
YouApprove, or return with comments
Your decision, in writing.
Step 3 of 6Worked example
What you give the AI Illustrative
Inception report (excerpt): We will use an LLM to code 90 KII transcripts against the codebook, saving about 20 days. Transcripts in two languages will be machine-translated. Quality will be assured by the team leader.
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
From the inception report attached, list every step where AI tools are used.
For each step give: the tool or model, the data it receives, where the data is processed, how the output will be checked (by whom, and on how much), and where this will be reported.
Write NOT STATED where the report says nothing. Do not judge the approach.
[attach inception report]
What comes back
AI step
Data
Check
Reported
LLM coding
90 KII transcripts
"Team leader": sample NOT STATED
NOT STATED
Machine translation
Transcripts in two languages
NOT STATED
NOT STATED
Every NOT STATED becomes a comment to the firm before you approve.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
AI review of the inception report: The AI approach is robust and follows good practice. Quality assurance by the team leader is adequate. Translation is handled. Recommend approval.
The QA prompt (run it on every draft)
Act as an independent reviewer of an AI assessment of an inception report.
Compare the assessment with the inception report and the contract clauses on AI.
List: every judgement the report does not back with a described check, every AI step with no sample size or agreement measure, every clause the report does not meet, and every decision the assessment makes that belongs to the evaluation manager.
Do not rewrite the assessment. List the problems only.
Contract clauses: [paste]
Inception report: [attach]
Assessment: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"Robust" is not backed by anything in the report: no validation sample is described.
"Quality assurance by the team leader" has no sample size and no agreement measure.
"Translation is handled": no check of the machine translation is described.
"Recommend approval" is your decision, and the report doesn't yet meet the validation and disclosure clauses.
Corrected output
Comments to the firm, before approval: double-code 15% of transcripts (14 of 90) by a second team member, with agreement reported for each code; check the machine translation on a sample, with a fluent reviewer; confirm the AI service used is on the approved list; add a methods annex describing prompts and models.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Quality: no AI coding is accepted without a validation plan: a double-coded sample, agreement scores and a methods annex.
Step 6 of 6Quick check
Quick check+25 XP
The firm says double-coding 15% of transcripts adds three days. What do you do?
Show answer
Keep the validation. Agree the days, and see the agreement scores in the methods annex.
Try it this week
Take one current inception report. Ask an AI assistant to list every AI step and its check, then write one comment for every NOT STATED.
Check a draft report's findings against the evidence
From the profile: the manager asks Claude to list each finding in a 70-page draft and the evidence it cites, then reviews the six findings with no linked source and returns them for revision; quality-assuring draft reports against the ToR, OECD-DAC criteria and evidence standards.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Reading a 70-page draft from start to finish.
Unsupported findings are noticed late, or not at all.
Comments say "strengthen the evidence" without saying where.
How it works with AI
Claude lists every finding and the evidence it cites.
You review every finding with no source, and a sample of the rest.
You return specific comments, finding by finding.
Step 2 of 6Step by step
YouUse an approved toolSharePoint
Draft reports can contain sensitive material. Use the organisation's approved AI service.
AIAsk Claude to list findings and evidenceClaude
Finding by finding, with the annex, table or page cited (prompt below).
YouReview every finding with no source
Is the evidence in the annexes, or is the finding unsupported?
YouCheck a sample of cited findings
Does the annex say what the finding says?
YouAssess the report against the ToR and criteriaOECD-DAC
Draft final report: 70 pages, 42 findings, evidence annexes A to F (survey tables, KII and FGD summaries, document review).
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
From the draft evaluation report attached, list every finding.
For each finding, give its number, a one-line summary and the evidence it cites (annex, table or page).
Write NONE where the finding cites no evidence. Do not judge whether the evidence is strong enough.
[attach draft report and annexes]
What comes back
Finding
Evidence cited
F3: Women's savings rose
Annex C, table 4
F17: Partners report strong ownership
Annex E (KIIs 4, 9, 12)
F22: Training improved market access
NONE
Six of 42 findings cite NONE (F11, F14, F22, F29, F33, F40). You review each one, then a sample of the others.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Finding
Evidence cited
F3: Women's savings rose by 30%
Annex C, table 4
F17: Partners report strong ownership
Annex E (KIIs 4, 9, 12)
F22: Training improved market access
None
AI note: All 42 findings are supported by the annexes.
The QA prompt (run it on every draft)
Act as an independent reviewer of an AI list of report findings and their evidence.
Open each cited annex at the cited table or page.
List: every finding with no evidence cited, every finding whose cited evidence says something different, and every summary claim the list itself does not show.
Do not judge whether the report should be accepted. List the problems only.
Draft report and annexes: [attach]
AI list: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"All 42 findings are supported" contradicts the list: F22 and five others cite no evidence.
F3 cites Annex C, table 4, which shows a 13% rise, not 30%.
A citation shows where to look, not how strong the evidence is: F17 rests on three KIIs, all with partner staff.
Whether a finding is supported, and the report is acceptable, is your judgement.
Corrected output
Comments to the evaluation team: six findings (F11, F14, F22, F29, F33, F40) cite no evidence: link them to the annex or remove them. F3: Annex C, table 4 shows 13%, not 30%: correct it. F17: say that all three interviews were with partner staff.
Also checked: a sample of ten other findings against the annex; no further problems.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Independence: you decide whether each finding is supported. A check returns unsupported findings; it never removes unwelcome ones.
Step 6 of 6Quick check
Quick check+25 XP
A well-supported finding is critical of a programme team, and its director asks you to remove it. What do you do?
Show answer
Keep it. Disagreement goes into the management response, not into the findings.
Try it this week
Take one draft report you are reviewing. Ask an AI assistant to list every finding and its evidence, then review every finding marked NONE.
Run a shared transcription and translation workflow
From the profile: interviews from four evaluations go through a Whisper and LLM translation pipeline with restricted access; the manager reviews monthly costs and a five-transcript quality sample from each evaluation; managing shared AI workflows for transcription, translation and coding, including access and costs.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Each evaluation buys its own transcription.
Nobody knows who can open which recordings.
Quality problems surface in the draft report.
How it works with AI
One shared workflow transcribes and translates for every evaluation.
Access is restricted to each evaluation's own team.
You review costs and a five-transcript sample from each evaluation every month.
Step 2 of 6Step by step
YouSet the access rulesSharePoint
Each evaluation's team sees only its own files. Log every change.
AutomatedTranscribe and translateWhisper · LLM APIs
Whisper transcribes; an LLM translates, in an approved service.
Your teamCheck a sample each month
Five transcripts from each evaluation, against the audio; key passages checked by a fluent speaker.
AISummarise the monthClaude
Hours, costs, sample results and access changes (prompt below).
YouReview costs and qualityExcel
Against each evaluation's budget and the sample results.
YouAct and update the guidance
Pause what fails the sample, and record what you changed.
Step 3 of 6Worked example
What you give the AI Illustrative
Evaluation
Audio hours
Sample (5 transcripts)
Cost vs budget
A
22
1 error that changes meaning
92%
B
30
20 errors, all in translated interviews
98%
C
12
Not done
75%
D
26
2 errors
140%
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
From this month's workflow log, summarise by evaluation: hours processed, cost against budget, quality sample results (errors that change meaning, per transcript) and access changes.
Flag every evaluation with no quality sample, every cost over budget and every access change outside the evaluation's own team.
Do not judge whether quality is acceptable.
[paste log, samples, budget and access log]
What comes back
Flags: Evaluation B: 4 errors that change meaning per transcript, all in translated interviews. Evaluation C: no quality sample. Evaluation D: 40% over budget. Access: one consultant from Evaluation A added to Evaluation C's folder.
Four flags. Each one is yours to act on before you approve the month.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Monthly workflow summary (AI): Quality is good across all four evaluations (1.2 errors per transcript on average). Costs are within budget. No access issues.
The QA prompt (run it on every draft)
Act as an independent reviewer of a monthly summary of a shared transcription and translation workflow.
Compare the summary with the workflow log, the quality samples, the budget and the access log.
List: every average that hides a problem in one evaluation, every evaluation with no quality sample, every cost over its budget, and every access change outside the evaluation's own team.
Do not judge whether quality is acceptable. List the problems only.
Log, samples, budget and access log: [attach]
Summary: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
The 1.2 average hides Evaluation B: 4 errors that change meaning per transcript, all in translated interviews.
Evaluation C has no quality sample this month, so its quality is unknown, not "good".
Evaluation D's cost is 40% over its monthly budget.
The access log shows a consultant from Evaluation A added to Evaluation C's folder.
Corrected output
Actions: Evaluation B: pause machine translation for that language until a fluent reviewer checks key passages. Evaluation C: five-transcript sample due this week. Evaluation D: cost reviewed with the team leader. Evaluation C folder: access removed and logged.
Approved: this month's cost for Evaluations A and B, after the checks above.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Accountability: you own the shared workflow: who has access, what it costs and what the monthly quality sample shows.
Step 6 of 6Quick check
Quick check+25 XP
The sample shows many errors that change meaning in one translated language. What do you do?
Show answer
Pause it, have key passages checked by a fluent speaker, and record the change in the guidance.
Try it this week
Pick one month of transcripts from your evaluations. Check five against the audio, and record every error that changes meaning.
From the profile: Copilot summarises status updates on 120 recommendations from eight evaluations into the tracker; the manager confirms overdue items with programme leads before the annual report to the board; updating the recommendations tracker after a management response meeting.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Chasing status updates by email before the board.
"We have started" gets recorded as done.
The board sees figures nobody confirmed.
How it works with AI
Copilot summarises updates into the tracker, quoting each one.
You check every status change against its update.
You confirm overdue and completed items with programme leads.
Step 2 of 6Step by step
Your teamProgramme leads send updates
After management response meetings, or every quarter.
AISummarise updates into the trackerMicrosoft 365 Copilot
A proposed status for each, quoting the update (prompt below).
YouCheck every status changeExcel
Do the quoted words support it?
YouConfirm with programme leads
Every overdue and every completed item, before reporting.
AutomatedUpdate the dashboardPower BI
Status by evaluation and by owner.
YouWrite the board section
Only confirmed figures, with what is overdue and why.
Step 3 of 6Worked example
What you give the AI Illustrative
R-14 (Evaluation 3): "We have started discussions with the ministry on the handover." R-27 (Evaluation 5): "Training plan finalised; first of four sessions held in March." R-52 (Evaluation 6): no update received.
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
For each recommendation below, propose a status: Not started, In progress, Completed, Overdue or No update.
Quote the words from the update that support the status.
If there is no update, write No update. Do not infer completion from plans or intentions.
[paste recommendations, due dates and updates]
What comes back
Recommendation
Proposed status
Quoted words
R-14
In progress
"started discussions"
R-27
In progress
"first of four sessions held"
R-52
No update
None received
Three proposals. You confirm them with the programme leads before they enter the board report.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Recommendation
AI status
R-14
Completed ("started discussions with the ministry")
R-27
Completed ("first training session held")
R-52
In progress
AI summary: 85% of recommendations are implemented.
The QA prompt (run it on every draft)
Act as an independent reviewer of AI-proposed statuses in a recommendation tracker.
Compare each status with the update it quotes and the recommendation's wording.
List: every status the quoted words don't support, every status given where no update was received, and every summary figure the tracker doesn't produce.
Do not change the statuses. List the problems only.
Tracker and updates: [paste]
AI statuses: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
R-14: "started discussions" is in progress, not completed.
R-27: one of four planned sessions is in progress, not completed.
R-52: no update was received, so its status is unknown, not in progress.
"85% implemented" is not what the tracker shows: 61 of 120 are completed (51%).
Corrected output
Status after confirmation with programme leads on [date]: 61 of 120 recommendations completed (51%), 38 in progress, 12 overdue and 9 with no update. The 12 overdue items and their owners are listed for the board.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Accountability: overdue and completed items are confirmed with programme leads before anything reaches the board.
Step 6 of 6Quick check
Quick check+25 XP
The AI marks a recommendation Completed because the update says "we have started". What do you do?
Show answer
Change it to In progress and confirm it with the programme lead.
Try it this week
Take ten recommendations from your tracker. Ask an AI assistant to propose statuses from the latest updates, then check every one.
Search your evaluation library and check the answers
From the profile: maintaining a searchable library of past evaluations and recommendations; reviewing the reliability of AI-supported analyses across the portfolio and updating guidance. Searchable libraries depend on retrieval systems whose limits the manager should understand.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Past reports sit in folders nobody searches.
Lessons are remembered, not found.
The same recommendations come back in every evaluation.
How it works with AI
An AI assistant searches the library and cites the reports it used.
You open every cited page.
You look for relevant reports the search missed, and log errors.
Step 2 of 6Step by step
YouDecide what goes in the librarySharePoint
Final reports and management responses. No transcripts or personal data.
AIAsk a question, with citationsMicrosoft 365 Copilot · Claude
Only from the library, with report and page for each point (prompt below).
YouOpen every cited page
Is the report in the library, and does the page say that?
Share what the library can and can't do with the evaluation team.
Step 3 of 6Worked example
What you give the AI Illustrative
Library: 46 final evaluation reports (2016–2026) and their management responses. Question: What have our past evaluations found about savings groups after funding ends?
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Answer the question below using only the reports in the library.
For each point, give the report ID, year and page.
If the library has no relevant report, say so. List every report you used.
Do not add evidence from outside the library.
Question: [paste]
What comes back
Point
Source
Most groups still active after funding ended
E-2019-04, p. 31
About half still active after two years
E-2022-02, p. 18
Most groups still active
E-2024-07, p. 12
Three points, each with a page you can open. A manual check comes next: the search may have missed reports.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Library answer (AI): Past evaluations consistently find that savings groups stay active after funding ends (E-2019-04, E-2022-02, E-2024-07, E-2025-01).
The QA prompt (run it on every draft)
Act as an independent reviewer of an answer from an AI-searchable evaluation library.
Open every cited report at the cited page.
List: every citation that is not in the library, every claim a cited report doesn't make, every word such as "consistently" the reports don't support, and every way to check for relevant reports the search may have missed.
Do not rewrite the answer. List the problems only.
Library list: [attach]
Answer: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
E-2025-01 is not in the library: the citation is invented.
E-2022-02, p. 18 found only half of the groups still active after two years: "stay active" misreports it.
"Consistently" overstates three reports with different results.
A manual search by year and theme found E-2021-06, which the search missed: most groups there stopped meeting after funding ended.
Corrected output
Of four relevant reports (three found by the search, one by a manual check), two found most savings groups still active after funding ended (E-2019-04, p. 31; E-2024-07, p. 12), one found about half still active after two years (E-2022-02, p. 18), and one found most had stopped meeting (E-2021-06, p. 22). The evidence is mixed. The invented citation was logged and reported to the library administrator.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Quality: every library answer is checked against the reports it cites, and you look for the reports the search missed.
Step 6 of 6Quick check
Quick check+25 XP
The library answer cites a report you can't find in the library. What do you do?
Show answer
Remove it, log it, and check every other citation in that answer.
Try it this week
Ask your library, or an approved AI tool with five past reports, one question. Open every cited page, and note anything it got wrong.
Pick a recurring task that is not covered above, such as planning the annual evaluation portfolio or preparing a reference group meeting, and use this prompt to draft your own step-by-step guide.
I am an evaluation manager. Here is a task I do regularly: [describe the task, when, who is involved, which tools and data].
Break it into numbered steps. For each step, say who should do it (me, my team, the evaluation firm, programme staff, an automated tool or an AI assistant), which tool, and what could go wrong.
Mark every step where a person must check the result before it is used, and say what they should check.
Do not suggest putting personal data or identifiable interview content into an AI tool.
End of Mission 04
MISSION05
The skills you need to develop
Skill inventory · Badge: Skill Builder
1 challenge
Mission goal: find out which of the 14 skills for this role you already have and which need new learning.
Step 1 of 5Guess first
Your guess+20 XP
Which skill is the first learning priority for this role in the career profile?
Show answer
Validating AI outputs, followed by data protection, retrieval-augmented knowledge systems, learning and knowledge management, and MAXQDA.
Step 2 of 5Skills you bring with you
Tick the ones you already have. Open a skill to see how to build it.
Your skills match
0 of 14
A. Existing evaluation management competencies
Transfers from your role
Judging whether a design fits the questions, budget and context. Now including how AI is used in it.
Build it: Review the AI steps in one inception report (guide 2).
Transfers from your role
Supervising evaluation staff and managing external evaluators.
Build it: Agree with your team who checks which AI step.
Transfers from your role
Keeping evaluations on time, from ToR to management response.
Build it: Add the AI checks to one evaluation's workplan.
Transfers from your role
The annual portfolio budget, now with shared workflow costs.
Build it: Review one month of shared workflow costs (guide 4).
Transfers from your role
Judging whether findings combine survey, interview and document evidence well.
Build it: Check how one draft report triangulates its main finding.
Transfers from your role
Seeing when a fluent report overstates its evidence.
Build it: Mark every overstatement in one draft report.
Transfers from your role
Reference groups, validation and management responses.
Build it: Plan how you will present one contested finding to the reference group.
Transfers from your role
Clear comments, quality reviews and board reports.
Build it: Write comments on one draft, finding by finding (guide 3).
Step 3 of 5New skills for the AI era
These are the ones most evaluation managers still need to learn, or to apply in a new way.
Your skills match
0 of 14
B. New AI and evaluation management competencies
Needs new learning
Judging whether evaluators checked their AI-supported analysis properly. The profile's first learning priority.
Build it: Assess one AI validation plan: sample, agreement, annex (guide 2).
Needs new learning
Commissioned evaluators may send interview data to AI services, which contracts must control (a learning priority).
Build it: Review one contract's data clauses with your focal point (guide 1).
Needs new learning
Searchable evaluation libraries depend on retrieval, and its limits (a learning priority).
Build it: Check one library answer against every cited page (guide 6).
Needs new learning
Evaluations are only useful when findings feed into decisions and are tracked (a learning priority).
Build it: Confirm ten recommendation statuses with programme leads (guide 5).
Needs new learning
Shared transcription, translation and coding workflows used across evaluations.
Build it: Map one shared workflow: data, access, checks and costs.
Needs new learning
Consent, confidentiality and independence when AI enters commissioned work.
Build it: Check that one evaluation's consent covers AI processing.
Step 4 of 5Your five learning priorities
In this order, from the career profile:
Validating AI outputs. You must judge whether evaluators have checked their AI-supported analysis properly.
Data protection and privacy. Commissioned evaluators may send interview data to AI services, which contracts must control.
Retrieval-augmented knowledge systems. Searchable evaluation libraries depend on retrieval systems whose limits you should understand.
Learning and knowledge management. Evaluations are only useful when findings feed into decisions and are tracked.
MAXQDA. Reviewing coded qualitative evidence requires familiarity with common analysis software.
The tools change; the method doesn't. Whichever tools your evaluators use, ask how they were checked, check a sample yourself, and keep the decision to accept a report with you.
End of Mission 05
MISSION06
Interactive skills gap assessment
Know your gaps · Badge: Self-Aware Analyst
1 challenge
Mission goal: measure your readiness in four areas and find your top three gaps.
Step 1 of 3How it works
10 quick statements, about 3 minutes. Rate yourself honestly: this is a learning tool, not a test, and your answers stay in your browser.
Beginner: little or no experience
Developing: I understand it but need practice
Competent: I can apply it independently
Step 2 of 3Questions 1 to 5
Step 3 of 3Questions 6 to 10
Prefer to score it yourself?
Beginner = 0, Developing = 1, Competent = 2. Add them up (maximum 20): 0–7 Getting started · 8–14 Developing · 15–20 Ready to manage AI-supported evaluations. For every Beginner answer, start with the matching action:
Use the routine in mission 4, guide 1.
Add a validation criterion to your next proposal scoring grid.
Use the routine in mission 4, guide 2.
Use the routine in mission 4, guide 3.
Plan how you will handle one contested finding with the reference group.
Use the routine in mission 4, guide 4.
Check one current contract's data clauses with your data protection focal point.
Use the routine in mission 4, guide 5.
Use the routine in mission 4, guide 6.
Write one guidance update from your last three reviews.
Mission goal: commit to a realistic 12-week plan and track it week by week.
Step 1 of 5Weeks 1–2: Understand AI
About 3 hours a week (a planning suggestion). Tick steps as you do them: +5 XP each, saved in this browser. The mission completes when every step in all four stages is ticked.
Stage 1 · Weeks 1–20 / 5 done
Understand AI
Map where AI is already used in your portfolio and learn what it gets wrong.
Loot: a map of AI use in your portfolio, an error log and the rules you must apply.
Step 2 of 5Weeks 3–4: Apply AI
Stage 2 · Weeks 3–40 / 4 done
Apply AI
Try four management tasks with AI, with checks.
Loot: draft clauses, one inception review and one findings check.
Step 3 of 5Weeks 5–8: Integrate AI
Stage 3 · Weeks 5–80 / 5 done
Integrate AI
Make the rules and checks routine across the portfolio.
Loot: AI rules and checks running across the portfolio.
Step 4 of 5Weeks 9–12: Demonstrate competence
Stage 4 · Weeks 9–120 / 5 done
Demonstrate competence
Show the approach is reliable and your team can run it.
Loot: a portfolio project, evidence of what the checks caught and a development plan.
Step 5 of 5Check
Mission check+20 XP
In which stage do the AI clauses go into every new ToR and contract?
Show answer
Stage 3 (weeks 5–8).
End of Mission 07
MISSION08
Your practical learning project
Boss level · Badge: Quality Gatekeeper
1 challenge + 8 deliverables
Mission goal: produce a portfolio piece that proves you can manage AI-supported evaluations responsibly.
Step 1 of 6Your boss-level project
Boss level project
Set AI rules and checks for one evaluation, from ToR to recommendations
Objective: show you can set the rules for AI in commissioned work, check the AI-supported process and the report, and keep the decision to accept it with you.
Case study · Foundation evaluation portfolio Illustrative
No live evaluation to use? Work through this portfolio, using the examples from mission 4.
Context: one evaluation you manage, from the ToR to the recommendations. Use documents you are allowed to share, and no personal data in AI tools.
Resources: the ToR and contract, the inception report, a draft report with its annexes, the shared workflow log, the recommendations tracker, your evaluation library and an approved AI assistant.
Step 2 of 6Deliverables, one by one
Deliverables checklist0 / 8 done
8 steps to a portfolio piece
The mission and the certificate unlock only when all 8 deliverables are ticked.
Step 3 of 6Templates to start from
Four ready-made files for your deliverables. Download, adapt and keep them.
Before you share it: what should a person check in your project?
Think about it for a moment, then reveal.
Every clause, against your organisation's rules and the law that applies.
Every AI step in the inception report, against the clauses.
Every finding with no source, and a sample of the rest, against the annex.
Every status in the board report, against a confirmed update.
No personal data in any AI step or shared file.
Step 5 of 6How it's judged
Criterion
What good looks like
Enforceable rules
Clauses anyone can check in the inception report.
Validated process
A double-coded sample with agreement scores.
Evidence-checked
Every finding with no source reviewed by you.
Confirmed status
Board figures confirmed with programme leads.
Protected data
Approved services and restricted access only.
Transparent
AI use disclosed in methods annexes and board reports.
Step 6 of 6Check
Mission check+20 XP
Which deliverable shows that a firm's AI coding can be trusted before the draft report arrives?
Show answer
The inception review, with the validation plan agreed.
End of Mission 08
MISSION09
Responsible AI: what you must never delegate blindly
Guardrails · Badge: Responsible AI Guardian
10 challenges
Mission goal: know exactly which judgements stay with you, and the control for each AI risk.
Step 1 of 6Would you approve this report?
Click every sentence you would not send. There are five problems: an invented figure, an unsupported causal claim, an incorrect percentage, a missing indicator and a misleading interpretation.
Case study · Foundation evaluation portfolio Illustrative
An AI assistant drafted the evaluation section of the annual report to the board, from the recommendations tracker. It is due to the director today.
Source (recommendations tracker, confirmed)
Result
Recommendations tracked
120, from 8 evaluations
Completed
61 (51%)
In progress
38
Overdue (confirmed with programme leads)
12
No update received
9
Cost data
Not part of the tracker
Problems found: 0 of 5+40 XP
Show answer
Five problems: the 72% (99 of 120 is 83%), the invented 30% cost saving, the unsupported claim that AI makes evaluations more credible, the claim that recommendations with no update are on track, and the claim that every status is reported (the 12 overdue recommendations are missing).
Step 2 of 6Your decision
Your decision+20 XP
You found the problems. Can the section go to the director?
Show answer
Correct it first. Fix each problem, recheck every figure, then send it.
Step 3 of 6Five questions before you send it
The section is corrected. Before you send it, answer the five questions. Tick each one you can honestly answer yes to.
Step 4 of 6Mini-game: delegate or never?
AI can list, summarise, transcribe and search. It cannot be accountable. For each task: can AI help (with checks), or must you never delegate it?
Delegate?+10 XP
Listing findings and the evidence they cite in a draft report
Show answer
AI can help. With every finding without a source reviewed by you.
Delegate?+10 XP
Deciding whether a report's methods and evidence are good enough to accept it
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Summarising recommendation status updates
Show answer
AI can help. With overdue and completed items confirmed with programme leads.
Delegate?+10 XP
Protecting evaluation independence when findings are unwelcome
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Transcribing and translating interviews across evaluations
Show answer
AI can help. With restricted access and a monthly quality sample.
Delegate?+10 XP
Choosing which programmes to evaluate and with what budget
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Searching past evaluations in the library
Show answer
AI can help. With every cited page opened.
Delegate?+10 XP
Negotiating management responses to difficult recommendations
Show answer
Never delegate. Listed in the profile as human judgement.
Step 5 of 6Six risks, six controls
What can go wrong when AI enters your evaluation portfolio? Name three risks before you look.
Think about it for a moment, then reveal.
Risk
What it looks like
Your control
Unvalidated AI coding
"Quality assured" with no sample or agreement score
A validation plan in every inception report
Data in the wrong place
Transcripts in a consumer tool used by a subcontractor
Transparency is a control too. Add a short note to the board report or the evaluation policy, and adapt it to what was actually done.
AI tools supported the evaluation function this year. Commissioned evaluations used AI for transcription, translation and coding under contract clauses on data handling, validation and disclosure; each inception report included a validation plan, and agreement scores are reported in the methods annexes. An AI assistant listed findings and their evidence in draft reports, and the evaluation manager reviewed every finding without a linked source. Microsoft 365 Copilot summarised recommendation status updates; overdue items were confirmed with programme leads on [date]. [name] is responsible for these checks.
The AI-Enabled M&E Manager is covered in Tutorial 01, the AI-Enabled Senior M&E Specialist in Tutorial 03, the Director of AI-Enabled M&E in Tutorial 06 and the AI-Assisted Evaluation Specialist in Tutorial 08.
Step 2 of 5What you need to show
Strengths you build on
Commissioning and managing evaluations
Quality assurance with the OECD-DAC criteria
Reference groups and management responses
Budgets and evaluation staff
What you need to show
AI clauses in ToRs and contracts
Validation plans required and reviewed
Findings checked against evidence annexes
Shared workflows with access, cost and quality checks
Recommendation status confirmed before the board
Step 3 of 5Tool: build your LinkedIn headline
Pick the pieces that are true for you. Only claim what you have actually done.
Your headline
Step 4 of 5CV bullets and qualifications
CV bullet templates
Added AI use, data handling and disclosure clauses to [x] evaluation ToRs and contracts, reviewed with the data protection focal point.
Required and reviewed AI validation plans in [x] inception reports: double-coding of [15]% of transcripts, with agreement scores for each code.
Checked [x] draft reports' findings against their evidence annexes with AI support, returning [x] unsupported findings for revision.
Ran a shared transcription workflow for [4] evaluations, with restricted access, monthly cost reviews and quality samples.
Tracked [120] recommendations across [8] evaluations, confirming overdue items with programme leads before board reporting.
Replace the brackets with what you have actually done.
Qualifications in the profile
Master's degree in evaluation, development studies, public policy, social sciences or a related field.
7–12 years in evaluation, including managing commissioned evaluations, budgets and evaluation staff.
Useful certifications: IPDET evaluation training programme; PMD Pro (PM4NGOs).
Step 5 of 5Check
Mission check+20 XP
What's the best way to use "AI-Enabled Evaluation Manager" on your CV?
Show answer
Use it to describe your capabilities, backed by evidence, unless your employer actually uses the title.
End of Mission 10
MISSION11
Your next steps
Commit and celebrate · Badge: Transition Ready
1 challenge
Mission goal: turn what you learned into a two-week plan and unlock your certificate.
Step 1 of 2Commit to your next two weeks
Pick the actions you commit to for the next two weeks. Your plan builds itself below.
Start learning
Do this
Show this
My plan
Recommended first project: the evaluation from mission 8, starting with the AI clauses in your next ToR.
Step 2 of 2Your completion certificate
Complete all 11 missions to unlock it. Add your name, take a screenshot and share your progress.
AI & M&E Career Transition Series · Tutorial 09
Your name
has completed the learning path
From Evaluation Manager to AI-Enabled Evaluation Manager
This is a learning-progress certificate. It records what you completed in this free tool, in your own browser; it is not independently verified and not a formal qualification. Your mission 8 project is the evidence to show an employer.
End of Mission 11
Questions and answers
What does an AI-Enabled Evaluation Manager do?
Manages an organization's evaluation portfolio, commissioning and overseeing evaluations conducted by internal teams or external consultants so they are credible, useful and delivered on time. Coordinates AI-supported evaluation processes, manages the automated analytical workflows evaluators use, and reviews methodology, evidence quality and findings before reports are accepted.
How is AI changing the Evaluation Manager role?
Evaluation teams now use AI for transcription, coding, synthesis and drafting, so the manager must assess both the evaluation and the AI-supported process behind it. The manager sets the rules for AI use in commissioned work, runs the shared workflows and holds the line on evidence quality when faster analysis makes weak findings easier to produce.
Which judgements should an AI-Enabled Evaluation Manager never hand over to AI?
The career profile lists these as human judgement:
Deciding whether an evaluation's methods and evidence are good enough to accept the report.
Protecting evaluation independence when findings are unwelcome.
Choosing which programmes to evaluate and with what budget.
Negotiating management responses to difficult recommendations.
Which skills should I learn first?
The learning priorities in the career profile, in order:
Validating AI outputs. The manager must judge whether evaluators have checked their AI-supported analysis properly.
Data protection and privacy. Commissioned evaluators may send interview data to AI services, which contracts must control.
Retrieval-augmented knowledge systems. Searchable evaluation libraries depend on retrieval systems whose limits the manager should understand.
Learning and knowledge management. Evaluations are only useful when findings feed into decisions and are tracked.
MAXQDA. Reviewing coded qualitative evidence requires familiarity with common analysis software.
The profile lists 13 skills in all: Evaluation design, Team leadership, Project management, Budget and resource management, Validating AI outputs, Critical appraisal, Mixed-methods integration, Learning and knowledge management, Stakeholder engagement, Workflow automation, Research and evaluation ethics, Data protection and privacy and Analytical report writing.
Which tools does this role use?
The career profile names: OECD-DAC evaluation criteria, Claude, Microsoft 365 Copilot, LLM APIs (OpenAI, Anthropic, Google), Whisper (speech-to-text), MAXQDA, SharePoint, Excel and Power BI. The tutorial shows how to check every AI output, whichever tool you use.
What qualifications and experience does the profile list?
Master's degree in evaluation, development studies, public policy, social sciences or a related field.
7–12 years in evaluation, including managing commissioned evaluations, budgets and evaluation staff.
Useful certification: IPDET evaluation training programme and PMD Pro (PM4NGOs).
How long does the transition take?
This tutorial suggests a 12-week plan of about 3 hours a week, in four stages: understand AI (weeks 1–2), apply AI (weeks 3–4), integrate AI (weeks 5–8) and demonstrate competence (weeks 9–12). It is a planning suggestion, not a requirement.
Is "AI-Enabled Evaluation Manager" an official job title?
No. It is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy. Use it to describe your capabilities, backed by evidence, unless your employer actually uses the title.
