Mission goal: understand what the AI-enhanced version of your role is and why it is emerging.
Step 1 of 5Where are you starting from?
Choose your starting point+10 XP
Where are you today with AI in your monitoring work?
Show answer
Every starting point is valid. Follow the mission suggested for your situation.
Step 2 of 5Meet your new role
Today's role
Monitoring Specialist
AI-enhanced role
AI-Powered Monitoring Specialist
Level
Specialist · 3–6 years
How the role uses AI
Integrates AI: analyses monitoring data, detects trends and raises early warnings
"AI-Powered Monitoring Specialist" is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy.
Step 3 of 5What changes most in your job?
Your guess+10 XP
When AI scans routine data and field notes continuously and raises alerts, where does your time go?
Show answer
In the profile, monitoring used to depend on someone noticing problems weeks after they occurred. AI-assisted analysis can now scan routine data and field notes continuously, so the specialist spends more time setting sensible thresholds, separating real problems from noise and making sure alerts lead to action.
Step 4 of 5AI-Enabled, not AI-Dependent
The goal is not to hand your work to AI. It is to become a stronger monitoring specialist who uses AI and stays in charge of it. Here are the five questions, asked about your own work:
1QualityDo your alerts catch real problems, and do you check how often they were right?Every rule is tested on past weeks, and each quarter you review whether the warnings were accurate and useful.
2AccountabilityWhen an alert reaches a programme manager, who stands behind it?You review each alert before it is sent, and every action is tracked until it is closed.
3IndependenceCan you still tell a real problem from noise or a seasonal dip?You call the field before you decide, instead of trusting the flag.
4EthicsAre people's feedback and sensitive findings handled safely?Hotline messages are anonymised before any AI step, and possible diversion or exclusion goes through the right channel.
5Professional developmentAre you learning how your checks work, or only reading their output?You can explain every line of your script and every threshold change, and you teach field teams to read the alerts.
The five questions as the series states them
Quality: Does AI improve the quality and credibility of evaluation findings?
Accountability: Who is responsible when AI produces an incorrect interpretation?
Independence: Are evaluators maintaining their ability to think critically and make independent judgments?
Ethics: Are confidentiality, human rights and the interests of affected populations protected?
Professional development: Are we developing stronger evaluators who understand AI, or creating professionals who increasingly depend on it?
Step 5 of 5Your goal for this tutorial
By the end, you will know how to build early warnings that programme teams can trust and act on, which skills to build, and you will have a 12-week plan.
Next: see which of your tasks change, and how.
End of Mission 01
MISSION02
Traditional role vs AI-enhanced role
Before and after · Badge: Role Shifter
8 challenges
Mission goal: see which responsibilities become AI-assisted, which need new skills and which stay human.
Step 1 of 4Meet your case study
Case study · Riverbend Food and Health Response Illustrative
A fictional programme you will follow through missions 2, 3, 4, 8 and 9.
The programme
Food distributions at 24 sites, cash payments and 8 health clinics, with a hotline for feedback and a third-party monitor in areas staff cannot reach.
The data
Weekly ActivityInfo exports, monitoring visit reports, hotline messages, and third-party forms with GPS and timestamps.
The tools
KoboToolbox, SurveyCTO, ActivityInfo, Excel, Python (pandas), Power BI, Power Automate, Claude, ChatGPT and an LLM API.
Your objective
Spot implementation problems while there is still time to fix them, and make sure every alert leads to a proportionate response.
Step 2 of 4Flip your tasks: today and with AI
Less scanning by eye, more judging and following up. Tap each card to see how it changes.
Step 3 of 4Mini-game: sort the work
For each task, choose: AI-assisted, a new skill for you, or stays human.
Sort it+10 XP
Scanning weekly data for sites with falling attendance
Show answer
AI-assisted. Automated trend and threshold checks are one of the profile's AI tasks.
Sort it+10 XP
Grouping issues from monitoring visit reports
Show answer
AI-assisted. Claude groups them; you confirm each group against the reports.
Sort it+10 XP
Checking GPS and timestamps on third-party forms
Show answer
AI-assisted. An AI step flags inconsistencies; you investigate them.
Sort it+10 XP
Writing a script that checks every new batch of data
Show answer
New skill. Python is one of the profile's learning priorities.
Sort it+10 XP
Categorising visit notes and feedback consistently
Show answer
New skill. Text classification is a learning priority in the profile.
Sort it+10 XP
Deciding whether an alert is a real problem or a seasonal effect
Show answer
Human. Listed in the profile as human judgement.
Sort it+10 XP
Raising sensitive findings such as diversion or exclusion
Show answer
Human. Listed in the profile as human judgement.
Sort it+10 XP
Judging what follow-up is proportionate for each issue
Show answer
Human. Listed in the profile as human judgement.
Step 4 of 4Your turn: two minutes
Think about last month's monitoring. Which tasks would go in each pile?
Think about it for a moment, then reveal.
A quick way to do it: three columns on a sheet of paper, AI-assisted, new skill, stays human. Keep the list: you will use it in mission 4 (to pick a task to try) and mission 6 (to rate your gaps).
End of Mission 02
MISSION03
What AI actually changes in your daily work
Checks, risks and red flags · Badge: Red Flag Spotter
2 challenges
Mission goal: learn the three checks every alert needs, and practise spotting problems in an AI draft.
Step 1 of 4What can AI really do for you?
Before you look: which of your tasks could AI help with, and which must stay with you?
Think about it for a moment, then reveal.
AI can assist with
Grouping issues from visit reports
Tagging hotline messages by site and theme
Drafting monitoring briefs
Writing scripts for routine checks
Can be partly automated
Weekly threshold checks on routine data
GPS and timestamp checks on forms
Sending reviewed alerts
Follow-up reminders on open actions
New responsibilities
Testing rules on past weeks
Reviewing every alert before it is sent
Logging every threshold change
Checking how often alerts were right
Stays with you
Real problem or seasonal effect?
Thresholds that don't overwhelm teams
Raising diversion or exclusion safely
Proportionate follow-up
Step 2 of 4The three checks you must always build in
An alert nobody checked can send a team in the wrong direction. Here are the checks, one at a time.
Test it on known cases. Every script and rule is run on past weeks where you already know what happened.
Confirm before you send. Every alert and every issue is checked against field evidence before it reaches a programme manager.
Protect people. Feedback is anonymised before any AI step, and sensitive findings go through the agreed channel.
Step 3 of 4Mini-game: spot the red flags
Case study · Riverbend Food and Health Response Illustrative
An AI assistant summarised this week's alert check for programme managers. Here is the source and the draft.
Click every sentence you would not send.
Weekly alert check
Sites flagged
4 of 24 (Sites 3, 7, 11 and 15)
Site 15
No report received yet
Site 7
Planting season: the same dip happened last year
Sites 3 and 11
No explanation yet
Red flags found: 0 of 3+30 XP
Show answer
Three sentences should not be sent: "attendance is collapsing across the programme" (4 of 24 sites), "Site 15 had no attendance" (its report is missing), and the claim about Site 3's committee (no evidence).
Step 4 of 4Mission check
Your call+20 XP
The script flags four sites. Programme managers want the alerts now. What happens first?
Show answer
Review each alert first, and call the field where the cause is unclear.
End of Mission 03
MISSION04
Your daily tasks, step by step with AI
Six practical guides · Badge: Workflow Builder
6 challenges
Mission goal: learn how to build early warnings and checks in six of your real tasks, with a human check at every step.
Case study · Riverbend Food and Health Response Illustrative
Every worked example below uses this programme's data, so you can follow one monitoring system from weekly data to the brief.
Six real tasks from your working week. Pick one and go through it in six short steps: a task prompt, then a QA prompt that checks the AI's work.
YouAIYour teamAutomated shows who does each step. Examples are illustrative.
Choose a task
Guide 1 of 6 From the career profile
Set up weekly early warning alerts and review each one
From the profile: a Python script runs on weekly ActivityInfo exports, flags sites where attendance falls more than 25% below their four-week average and emails programme managers; the specialist reviews each alert before it is sent.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Problems show up in a spreadsheet weeks after they start.
Someone has to notice a falling number by eye.
When there are alerts, they go out without anyone checking them.
How it works with AI
A script checks every site in each weekly export.
Sites more than 25% below their four-week average are flagged.
You review each alert before it reaches a programme manager.
Step 2 of 6Step by step
YouAgree the rule with programme teams
For example: attendance more than 25% below the site's four-week average.
AIAsk AI to write the scriptClaude · Python (pandas)
It reads the weekly export and lists the sites that break the rule (prompt below).
YouTest the script on past weeksPython (pandas)
Use weeks where you know which sites had problems. It must flag them, and only them.
AutomatedRun it on every new exportActivityInfo · Python (pandas)
Each week, on the ActivityInfo export.
YouReview each alert before it is sent
Real problem, data error or seasonal effect? Call the field if you are not sure.
AutomatedSend the reviewed alertsPower Automate
To the programme manager for each site, with the figures.
Step 3 of 6Worked example
What you give the AI Illustrative
Site
Week 1
Week 2
Week 3
Week 4
This week
Site 3
410
395
420
405
290
Site 7
220
230
210
225
205
Site 11
150
160
140
150
100
Site 15
300
310
290
300
0
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Write a Python (pandas) script that reads our weekly attendance export (CSV with columns: site, week, attendance).
For each site, compare this week's attendance with the average of the four weeks before it.
List every site where this week is more than 25% below that average, with the average, this week's figure and the percentage change.
List sites with no figure this week separately, as missing.
Do not change or fill in any figure. Add a comment that explains each step.
What comes back
Site
Four-week average
This week
Change
Site 3
408
290
−29%
Site 11
150
100
−33%
Site 15
300
0
−100%
Three flags. Site 15 shows 0: a closed site, or a report entered as 0? You check before any alert goes out.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
avg = df.groupby('site')['attendance'].mean()
df = df.dropna()
flag = this_week < avg * 0.25
The QA prompt (run it on every draft)
Act as an independent reviewer of a monitoring script.
Compare the code below with the rule.
List: every line where the calculation differs from the rule, every place the code drops or changes data silently (such as missing figures), and a small test table, with the right answer, that would show each problem.
Do not rewrite the script. List the problems only.
Rule: [paste]
Code: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
The average uses every week, including this one. The rule is the four weeks before this one.
dropna() silently removes sites with no report this week, so missing sites never appear.
avg * 0.25 flags sites below a quarter of their average, not sites more than 25% below it. It should be avg * 0.75.
Corrected output
# prev = the four weeks before this week, by site
avg = prev.groupby('site')['attendance'].mean()
flag = this_week < avg * 0.75
Sites with no report this week are listed separately as missing. Tested on four past weeks: it flagged the two sites with known problems, and no others.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Accountability: you review each alert before it is sent. An automated email is still your message.
Step 6 of 6Quick check
Quick check+25 XP
The script flags Site 15 at 0 this week. What do you do before the alert goes out?
Show answer
Call the field first. Then send the alert, or correct the data.
Try it this week
Take last month's attendance data. Work out each site's four-week average in Excel and see which sites the 25% rule would have flagged. Were they real problems?
Group issues from visit reports into the action tracker
From the profile: Claude reads 40 monitoring visit reports from a quarter and groups the issues raised, such as late payments and missing supplies; the specialist confirms each group against the reports and enters them in the action tracker.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Issues are buried in 40 separate visit reports.
The same problem at several sites goes unnoticed.
Actions are agreed in meetings and forgotten.
How it works with AI
Claude groups the issues across reports, with the report each one came from.
You confirm every group against the reports.
Each confirmed issue goes into the action tracker with an owner and a date.
Step 2 of 6Step by step
YouRemove names before you upload
Replace names of staff and participants with roles; check for phone numbers and household details.
AIAsk Claude to group the issuesClaude
With the report and site for every issue (prompt below).
YouConfirm each group against the reports
Open the reports cited. Is the issue there, and is it the same issue?
YouEnter confirmed issues in the action trackerExcel
One row per issue: sites, source, action, owner, due date.
Your teamAgree actions with programme managers
They own the response; you track it.
YouTrack every action until it is closedExcel
Update the tracker with responses each week.
Step 3 of 6Worked example
What you give the AI Illustrative
Report
Site
Issue noted (excerpt)
Visit 4
Site 2
Cash payment two weeks late for 30 households
Visit 9
Site 5
Payment delayed; committee says the list is not approved
Visit 12
Site 8
Hygiene kits missing soap
Visit 17
Site 5
Long queues; elderly people waiting in the sun
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Below are excerpts from monitoring visit reports, each with its report number and site.
Group the issues by theme (for example late payments, missing supplies, access).
For each group, list every report and site it comes from.
Use the reports' own words. Do not add causes or recommendations.
If an issue fits no group, list it on its own.
[paste excerpts, names removed]
What comes back
Theme
Reports and sites
Late payments
Visit 4 (Site 2), Visit 9 (Site 5)
Missing supplies
Visit 12 (Site 8)
Access and waiting conditions
Visit 17 (Site 5)
Each group is checked against the reports cited before it enters the tracker.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Act as an independent reviewer of an issue summary from monitoring visit reports.
Compare the summary with the excerpts.
List: every cited report that does not mention the issue, every cause the reports do not state for all the sites in the group, and every issue in the excerpts the summary leaves out.
Do not rewrite the summary. List the problems only.
Excerpts: [paste]
Summary: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
Visit 21 (Site 6) does not mention payments.
"Caused by slow committee approval" comes from one report (Site 5). Site 2 gives no cause.
Long queues with elderly people waiting in the sun (Visit 17) are left out, and they may need a quick response.
Corrected output
Theme
Reports and sites
Late payments
Visit 4 (Site 2); Visit 9 (Site 5: committee says the list is not approved)
Missing supplies
Visit 12 (Site 8)
Access and waiting conditions
Visit 17 (Site 5): elderly people waiting in the sun
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Quality: every issue group is confirmed against the visit reports before it enters the action tracker.
Step 6 of 6Quick check
Quick check+25 XP
Claude's summary cites Visit 21 for late payments. You open it, and it doesn't mention payments. What do you do?
Show answer
Remove it, re-check the group, and keep a note of the error.
Try it this week
Take last month's visit reports. Remove names, ask an AI assistant to group the issues with report numbers, then check three groups against the reports.
Compare feedback and complaints with monitoring data
From the profile: an LLM categorises hotline messages by site and theme, and the specialist compares them with distribution data, finding that complaints about short rations match two sites with unusual stock figures. Raising sensitive findings, such as diversion or exclusion, through the right channels is human judgement.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Hotline messages and monitoring data are read by different teams.
A pattern across both is noticed late, or never.
Sensitive complaints sit in a spreadsheet with everything else.
How it works with AI
An LLM tags each message by site and theme.
You compare the themes with distribution and stock data, site by site.
Sensitive findings go through the agreed channel, not a dashboard.
Step 2 of 6Step by step
Your teamAnonymise the messagesKoboToolbox
Remove names, phone numbers and household details. The AI step gets site, date and text only.
AITag each message by site and themeLLM API
From a fixed list of themes, with UNSURE where nothing fits (prompt below).
YouCheck a sample of tags
Read 20 messages and compare them with the AI's themes.
AutomatedPut the themes next to monitoring dataPower BI
Complaints by theme and site, beside distribution and stock figures.
YouLook for sites where both point the same way
For example, short-ration complaints at sites with unusual stock figures.
YouRaise sensitive findings through the right channel
Possible diversion or exclusion goes to the agreed focal point, confidentially.
Step 3 of 6Worked example
What you give the AI Illustrative
Site
Short-ration complaints
Stock received
Stock distributed
Households served
Site 4
9
1,000 kg
1,000 kg
400
Site 9
11
1,200 kg
700 kg
480
Site 12
1
800 kg
800 kg
320
Site 16
8
900 kg
500 kg
360
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Tag each hotline message below with one site and one theme from this list: short rations, late distribution, exclusion from the list, staff behaviour, other.
If the site or theme is unclear, write UNSURE.
Do not summarise, judge or add anything to the messages.
The messages contain no names or phone numbers.
[paste messages: site, date, text]
What comes back
Site
Theme
Messages
Site 9
Short rations
11
Site 4
Short rations
9
Site 16
Short rations
8
Not clear
UNSURE
3
Sites 9 and 16 also distributed far less than they received. That match is yours to check, not the AI's conclusion.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Sites 4, 9 and 16 show diversion by distribution staff.
A caller from Site 9, the committee chair's neighbour, says the chair sells rations.
Recommendation: name the three sites in the monthly brief.
The QA prompt (run it on every draft)
Act as an independent reviewer of an analysis that combines complaints and monitoring data.
Compare the analysis with the tables.
List: every conclusion the data does not support (such as a cause), every detail that could identify a caller or an accused person, every site grouped with others when its data differs, and every sensitive finding sent to the wrong audience.
Do not rewrite the analysis. List the problems only.
Tables: [paste]
Analysis: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"Diversion by distribution staff" is a conclusion the data cannot show. It is a concern to raise, not a finding.
"The committee chair's neighbour" could identify the caller, and the accusation points at an identifiable person.
Site 4 distributed all the stock it received; its complaints need a different follow-up.
Possible diversion goes to the agreed focal point, not into a general monthly brief.
Corrected output
Sites 9 and 16: short-ration complaints match stock distributed well below stock received. Raised confidentially with the [agreed focal point] on [date].
Site 4: short-ration complaints, but stock received and distributed match. Follow up with the field team on how the ration size is explained to households.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Ethics: messages are stripped of names before any AI step, and possible diversion goes to the agreed focal point, not into a brief.
Step 6 of 6Quick check
Quick check+25 XP
The data suggests possible diversion at two sites. Where does it go?
Show answer
To the agreed focal point, confidentially, with the evidence you have.
Try it this week
Ask your accountability colleagues for last month's complaint themes by site. Put them next to your monitoring data for the same sites. Do any point the same way?
Check third-party monitoring forms for inconsistencies
From the profile: a third-party monitor submits 150 photo-verified visit forms from areas staff cannot reach; an AI step checks GPS coordinates and timestamps for inconsistencies, and the specialist investigates the 12 flagged forms.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Third-party forms are accepted as they arrive.
Nobody checks where and when a form was filled in.
Problems surface only if someone visits.
How it works with AI
An AI step checks GPS and timestamps on every form.
Forms far from the site, too short or on the wrong date are flagged.
You investigate every flagged form before you conclude anything.
Step 2 of 6Step by step
YouSet the rules for a flag
For example: more than 2 km from the site, under 10 minutes, or not on the planned visit date.
AIAsk AI to write the checkChatGPT · Python (pandas)
A pandas script or spreadsheet formulas that apply your rules (prompt below).
YouTest it on forms you know
Include one form you know is fine and one you know is wrong.
AutomatedRun it on every batchSurveyCTO · Python (pandas)
Each new batch of forms from the monitor.
YouInvestigate each flagged form
Photos, notes and a call to the monitor. A flag is a question, not proof.
Your teamAgree what happens next
Accept, ask for a re-visit, or raise it with the monitor's manager.
Step 3 of 6Worked example
What you give the AI Illustrative
Form
Site
Distance from site
Duration
Date
F-021
Site 30
0.3 km
42 min
12 May
F-047
Site 31
14.8 km
38 min
12 May
F-088
Site 34
0.5 km
4 min
13 May
F-112
Site 35
0.2 km
35 min
2 June (planned 14 May)
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Write a Python (pandas) script for our third-party monitoring forms (CSV with: form ID, site, site latitude, site longitude, form latitude, form longitude, start time, end time, planned visit date).
Flag each form where: the form is more than 2 km from the site; the interview took less than 10 minutes; the form date differs from the planned visit date; or the GPS reading or a timestamp is missing.
List each flagged form with the reason.
Do not delete or change any form. Add a comment that explains each step.
What comes back
Form
Reason for the flag
F-047
14.8 km from the site
F-088
4 minutes
F-112
Date differs from the planned visit
Three flags to investigate. None of them proves a form was faked.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Result: 12 of 150 forms were faked by the monitor. Forms with no GPS reading were accepted. Recommendation: withhold payment for all 150 forms.
The QA prompt (run it on every draft)
Act as an independent reviewer of a report on third-party monitoring checks.
Compare the report with the flagged list and the rules.
List: every conclusion that goes beyond what the checks show, every form the rules should have flagged but did not (such as forms with no GPS), and every recommendation that is not proportionate to the evidence.
Do not rewrite the report. List the problems only.
Rules: [paste]
Flagged list: [paste]
Report: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
"Faked" is a conclusion. A flag can have other causes: a GPS reading taken at the road, a visit moved to another day, a short follow-up interview.
Forms with no GPS reading were accepted. They should be flagged too.
Withholding payment for all 150 forms is not proportionate to 12 flags nobody has investigated yet.
Corrected output
12 of 150 forms were flagged (distance, duration or date), and 4 forms have no GPS reading. All 16 are being investigated with the monitor: photos, notes and a call for each.
Payment for the 16 forms is on hold until the investigation ends. The other forms are accepted.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Independence: a flag is a question, not a verdict. You investigate before you conclude that a form is wrong.
Step 6 of 6Quick check
Quick check+25 XP
Form F-047 was filled in 14.8 km from the site. What is it?
Show answer
A question. Investigate before you conclude.
Try it this week
Take the last batch of third-party forms. Check five by hand: is the GPS near the site, and is the interview length plausible?
Tune alert thresholds and check whether warnings were right
From the profile: adjusting alert thresholds after a seasonal drop in attendance triggered false warnings; tuning thresholds to reduce false alarms and documenting every change; reviewing periodically whether early warnings were accurate and useful.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
Thresholds are set once and never revisited.
Teams get so many alerts they stop reading them.
Nobody knows how many alerts were right.
How it works with AI
AI helps you test a new rule on past weeks.
You compare false alarms and missed problems before changing anything.
Every change is logged, with the reason and the date.
Step 2 of 6Step by step
Your teamRecord what each alert turned out to beExcel
Real problem, data error or seasonal effect, from the field's reply.
YouCount right and wrong alerts
Over the last quarter: real problems caught, false alarms, problems missed.
AIAsk AI to test options on past weeksClaude · Python (pandas)
The same weeks, with the current rule and a new one (prompt below).
YouChoose the change
Fewer false alarms, without missing real problems. Agree it with programme teams.
YouLog the changeExcel
Old rule, new rule, reason, date, who agreed.
YouReview again next quarter
Were the warnings accurate and useful?
Step 3 of 6Worked example
What you give the AI Illustrative
Last quarter, current rule (more than 25% below the four-week average)
Alerts sent
20
Real problems
7
Seasonal (planting season)
11
Data errors
2
Real problems missed (found on visits)
1
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Below are 13 weeks of attendance by site, and what each past alert turned out to be.
Test two rules on these weeks: (A) more than 25% below the site's four-week average; (B) the same, but compared with the same weeks last year during the planting season.
For each rule, count: alerts, real problems caught, false alarms and real problems missed.
Show the counts only. Do not recommend a rule.
[paste data and alert outcomes]
What comes back
Rule
Alerts
Real problems caught
False alarms
Missed
A (current)
20
7
13
1
B (seasonal comparison)
10
7
3
1
Rule B removes most false alarms without missing more problems. The choice, and the one missed problem, are yours to look at.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Recommendation: raise the threshold to 50% below the four-week average. This removes all false alarms.
The QA prompt (run it on every draft)
Act as an independent reviewer of a proposed change to an alert threshold.
Compare the proposal with the test results.
List: every real problem the new rule would miss, every claim not shown in the results, and anything missing from the change record (old rule, new rule, reason, date, who agreed).
Do not rewrite the proposal. List the problems only.
Test results: [paste]
Proposal: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
At 50%, the test catches only 3 of the 7 real problems: four would be missed.
"Removes all false alarms" is not what the test shows: the 2 data errors still trigger alerts.
There is no change record: no date, no reason, no one who agreed.
Corrected output
Change log
Old rule
More than 25% below the four-week average
New rule
The same, compared with the same weeks last year during the planting season
Reason
11 of 20 alerts last quarter were seasonal
Tested on
13 past weeks: false alarms 13 → 3; real problems caught 7 → 7
Agreed by and date
Programme managers, [date]
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Quality: every threshold change is tested on past weeks and logged, so fewer false alarms never means more missed problems.
Step 6 of 6Quick check
Quick check+25 XP
Teams complain about too many alerts during the planting season. What do you do?
Show answer
Test a seasonal rule on past weeks, agree it, and log it.
Try it this week
Ask programme managers what last quarter's alerts turned out to be. Count the real problems and the false alarms: that is your baseline.
Draft the monitoring brief and confirm every issue
From the profile: drafting monitoring briefs with AI support and confirming each issue against field evidence; calling a field team lead to confirm whether a stock-out at two clinics is real or a reporting error; updating the action tracker with responses from programme managers.
Step 1 of 6Today, and with AI
How it is usually done today Illustrative
The brief is written from scratch each month.
Issues go in before anyone has checked them.
Recommendations are not followed up.
How it works with AI
AI drafts the brief from your reviewed alerts and the tracker.
You confirm each issue against field evidence.
Each follow-up goes into the action tracker.
Step 2 of 6Step by step
YouCollect what you haveExcel
Reviewed alerts, confirmed visit issues, the action tracker.
AIAsk AI for a draftClaude
One section per issue, with source and status, and [CONFIRM] where evidence is missing (prompt below).
Your teamCall the field where needed
Is the stock-out real or a reporting error?
YouFill or remove every [CONFIRM]
Only issues with field evidence go in as confirmed.
YouWrite the follow-up
Proportionate to each issue, with an owner.
AutomatedSend it and track responsesPower Automate · Excel
Programme managers' responses go into the action tracker.
Step 3 of 6Worked example
What you give the AI Illustrative
Issue
Source
Status
Attendance down at Sites 3 and 11
Alerts, reviewed 12 May
Site 3: road closed (field team). Site 11: not yet known
Stock-out at Clinics 2 and 6
Weekly report
Not yet confirmed
Late payments at Sites 2 and 5
Visit reports 4 and 9
Confirmed; action open since 2 May
What would you ask the AI to do with this?
Think about it for a moment, then reveal.
The task prompt (copy and adapt)
Draft a one-page monitoring brief for programme managers from the table below.
One short section per issue: what was found, the source, what is confirmed and what is not.
Where an issue is not confirmed, write [CONFIRM] instead of a conclusion.
Do not add causes, figures or recommendations that are not in the table.
[paste table]
What comes back
Attendance, Sites 3 and 11: more than 25% below the four-week average (alerts reviewed 12 May). Site 3: road closed (field team). Site 11: [CONFIRM].
Stock-out, Clinics 2 and 6: reported in the weekly report. [CONFIRM]
Late payments, Sites 2 and 5: confirmed in visit reports 4 and 9; action open since 2 May.
Step 4 of 6Check the AI output
Independence first: read the draft and note the problems you see yourself, before you run the QA prompt. Then compare.
Another draft came back like this Illustrative
Stock-outs at Clinics 2 and 6 show a supply chain failure: an emergency order is recommended. Attendance at Site 11 fell because of the road closure. The late payments are resolved.
The QA prompt (run it on every draft)
Act as an independent reviewer of a monitoring brief.
Compare the brief with the evidence table and the action tracker.
List: every issue presented as confirmed that is not, every cause applied to a site it was not reported for, every status that differs from the tracker, and every recommendation not proportionate to the evidence.
Do not rewrite the brief. List the problems only.
Evidence: [paste]
Tracker: [paste]
Brief: [paste]
What should the check catch?
Think about it for a moment, then reveal.
What the check finds
The stock-outs are not confirmed yet: they may be a reporting error. An emergency order is not proportionate before a call.
The road closure was reported for Site 3, not Site 11.
The late payments are still open in the tracker, not resolved.
Corrected output
Stock-out: the field team lead confirmed on 14 May that Clinic 2 is out of stock; Clinic 6 was a reporting error. Follow-up: supply request for Clinic 2 (owner: health programme manager).
Attendance, Site 11: cause not yet known; field visit planned. Late payments, Sites 2 and 5: still open since 2 May.
Step 5 of 6Your checks
Tick each check as you do it on your own work. Each one earns +5 XP.
Accountability: every issue in the brief is confirmed against field evidence, and the follow-up you recommend is yours.
Step 6 of 6Quick check
Quick check+25 XP
The weekly report shows a stock-out at two clinics. Before the brief goes out, what do you do?
Show answer
Call the field team lead first, then report what is confirmed.
Try it this week
Take your last monitoring brief. For each issue, note the evidence behind it. Mark any issue that went in unconfirmed.
Pick a recurring task that is not covered above, such as post-distribution monitoring, and use this prompt to draft your own step-by-step guide.
I am a monitoring specialist. Here is a recurring task I do: [describe the task, how often, who is involved, which tools and data].
Break it into numbered steps. For each step, say who should do it (me, a field team, a programme manager, an automated check or an AI assistant), which tool, and what could go wrong.
Mark every step where a person must check the result before it is used, and say what they should check.
Do not suggest putting personal data into an AI tool.
End of Mission 04
MISSION05
The skills you need to develop
Skill inventory · Badge: Skill Builder
1 challenge
Mission goal: find out which of the 12 profile skills you already have and which need new learning.
Step 1 of 5Guess first
Your guess+20 XP
Which skill is the first learning priority for this role in the career profile?
Show answer
Predictive modelling (basic anomaly detection and trend analysis), followed by Python, text classification, adaptive management and Power Automate.
Step 2 of 5Skills you bring with you
Tick the ones you already have. Open a skill to see how to build it.
Your skills match
0 of 12
A. Existing M&E and monitoring competencies
Transfers from your role
Monitoring plans, indicators and reporting flows. Early warnings need to know what is planned.
Build it: Write down the five implementation indicators you most need to watch.
Transfers from your role
Definitions and units, so an alert compares like with like.
Build it: Check that attendance is counted the same way at every site.
Transfers from your role
Telling a real change from a reporting error.
Build it: Trace three flagged figures back to the site records.
Transfers from your role
Charts that show trends by site at a glance.
Build it: Chart twelve weeks of attendance for three sites.
Transfers from your role
Hotlines and complaints, handled safely and confidentially.
Build it: Ask how sensitive complaints are routed today.
Transfers from your role
Forms with GPS and timestamps that make checks possible.
Build it: Check which of your forms record GPS and start and end times.
Transfers from your role
Briefs that separate what is confirmed from what is not.
Build it: Mark every unconfirmed issue in your last brief.
Step 3 of 5New skills for the AI era
These are the ones most monitoring specialists still need to learn.
Your skills match
0 of 12
B. New AI and data competencies
Needs new learning
Basic anomaly detection and trend analysis underpin useful early warning. The profile's first learning priority.
Build it: Work out a four-week average and a 25% rule for five sites.
Needs new learning
Sending reviewed alerts and follow-up reminders to the right people.
Build it: Send one reviewed alert by email with Power Automate.
Needs new learning
Consistent categories for visit notes and feedback, which contain early signs of problems (a learning priority).
Build it: Tag 20 anonymised messages yourself, then compare with the AI.
Needs new learning
Checking AI groupings, tags and scripts against source data.
Build it: Check three AI-grouped issues against the visit reports (guide 2).
Needs new learning
Connecting early warnings to clear decision points (a learning priority).
Build it: Agree with one programme manager what each alert should trigger.
Step 4 of 5Your five learning priorities
In this order, from the career profile:
Predictive modelling and forecasting. Basic anomaly detection and trend analysis underpin useful early warning.
Python for data. Scripts make it possible to run routine checks on every new batch of data.
Text classification. Visit notes and feedback contain early signs of problems that need consistent categorising.
Adaptive management. Early warnings are only useful if they connect to clear decision points.
Power Automate. Alerts and follow-up reminders can be sent automatically to the right people.
You don't need to be a data scientist. A four-week average and a percentage rule catch a lot. An AI assistant can write the script; what matters is that you test it on weeks you know, and can explain every line.
End of Mission 05
MISSION06
Interactive skills gap assessment
Know your gaps · Badge: Self-Aware Analyst
1 challenge
Mission goal: measure your readiness in four areas and find your top three gaps.
Step 1 of 3How it works
10 quick statements, about 3 minutes. Rate yourself honestly: this is a learning tool, not a test, and your answers stay in your browser.
Beginner: little or no experience
Developing: I understand it but need practice
Competent: I can apply it independently
Step 2 of 3Questions 1 to 5
Step 3 of 3Questions 6 to 10
Prefer to score it yourself?
Beginner = 0, Developing = 1, Competent = 2. Add them up (maximum 20): 0–7 Getting started · 8–14 Developing · 15–20 Ready to run AI-powered monitoring. For every Beginner answer, start with the matching action:
Use the routine in mission 4, guide 1.
Ask AI to explain each line of the guide 1 script, then check it.
Use the routine in mission 4, guide 5.
Use the routine in mission 4, guide 2.
Use the routine in mission 4, guide 4.
Write the steps you follow and check them with your data protection focal point.
Mission goal: commit to a realistic 12-week plan and track it week by week.
Step 1 of 5Weeks 1–2: Understand AI
About 3 hours a week (a planning suggestion). Tick steps as you do them: +5 XP each, saved in this browser. The mission completes when every step in all four stages is ticked.
Stage 1 · Weeks 1–20 / 5 done
Understand AI
Map your monitoring data and learn what AI gets wrong.
Loot: a data map, an error log and a baseline of how late problems were found.
Step 2 of 5Weeks 3–4: Apply AI
Stage 2 · Weeks 3–40 / 4 done
Apply AI
Try one alert and two AI steps, with checks.
Loot: a tested alert script, a confirmed issue list and a checked brief.
Step 3 of 5Weeks 5–8: Integrate AI
Stage 3 · Weeks 5–80 / 5 done
Integrate AI
Make the checks part of every week.
Loot: weekly alerts, a combined view of feedback and monitoring, and an action tracker.
Step 4 of 5Weeks 9–12: Demonstrate competence
Stage 4 · Weeks 9–120 / 5 done
Demonstrate competence
Show the alerts are accurate and lead to action.
Loot: a portfolio project, a quarter of evidence and a development plan.
Step 5 of 5Check
Mission check+20 XP
In which stage do you first write and test the alert script?
Show answer
Stage 2 (weeks 3–4).
End of Mission 07
MISSION08
Your practical learning project
Boss level · Badge: Early Warning Lead
1 challenge + 8 deliverables
Mission goal: produce a portfolio piece that proves you can run AI-powered monitoring responsibly.
Step 1 of 6Your boss-level project
Boss level project
Build an early warning system for one programme, with a check at every step
Objective: show you can use AI to spot implementation problems early, check every alert and issue against field evidence, and make sure each one leads to a proportionate response.
Case study · Riverbend Food and Health Response Illustrative
No programme data of your own to use? Build the system for this programme, using the examples from mission 4.
Context: one programme, at least eight weeks of routine data. Use aggregated figures and anonymised notes only.
Resources: your routine data exports, visit reports, complaint themes, third-party forms if you use them, Excel or Python (pandas), Power Automate, and an approved AI assistant.
Step 2 of 6Deliverables, one by one
Deliverables checklist0 / 8 done
8 steps to a portfolio piece
The mission and the certificate unlock only when all 8 deliverables are ticked.
Step 3 of 6Templates to start from
Four ready-made files for your deliverables. Download, adapt and keep them.
Before you share it: what should a person check in your early warning system?
Think about it for a moment, then reveal.
The script, on past weeks with known problems.
Every alert, before it was sent.
Every AI-grouped issue, against the reports.
Any AI-generated text, reviewed and marked as AI-assisted.
No names or identifying details in any AI step or shared file.
Step 5 of 6How it's judged
Criterion
What good looks like
Tested rules
The script flags known problems in past weeks, and nothing else.
Reviewed alerts
No alert reached a programme manager without your review.
Confirmed issues
Every issue in the brief traces to field evidence.
Safe feedback handling
Messages anonymised; sensitive findings through the agreed channel.
Proportionate follow-up
Each response matches what was found, with an owner and a date.
Learning from accuracy
You know how often alerts were right, and changes are logged.
Step 6 of 6Check
Mission check+20 XP
Which deliverable shows whether your alerts were worth sending?
Show answer
The threshold review.
End of Mission 08
MISSION09
Responsible AI: what you must never delegate blindly
Guardrails · Badge: Responsible AI Guardian
10 challenges
Mission goal: know exactly which judgements stay with you, and the control for each AI risk.
Step 1 of 6Would you approve this report?
Click every sentence you would not send. There are five problems: an invented figure, an unsupported causal claim, an incorrect percentage, a missing indicator and a misleading interpretation.
Case study · Riverbend Food and Health Response Illustrative
Claude drafted May's monitoring brief from the alert log and the action tracker. Programme managers meet tomorrow.
Source (May)
Result
Alerts reviewed
6 (4 real problems, 2 seasonal)
Attendance, Site 3
290 against a four-week average of 408 (−29%)
Clinics 2 and 6
Clinic 2 stock-out confirmed; Clinic 6 a reporting error
Hotline
28 short-ration complaints from Sites 4, 9 and 16
Third-party forms
150 received, 16 under investigation
Late payments, Sites 2 and 5
Action open since 2 May
Problems found: 0 of 5+40 XP
Show answer
Five problems: the 45% (it is 29%), the invented halving of response time, the unsupported claim that staff are diverting food, the misleading reading of the clinic stock-outs (Clinic 6 was a reporting error), and the claim that all sources are covered (third-party monitoring is missing).
Step 2 of 6Your decision
Your decision+20 XP
You found the problems. Can the brief go to programme managers?
Show answer
Correct it first. Fix each problem, recheck every issue, then send it.
Step 3 of 6Five questions before you send it
The brief is corrected. Before you send it, answer the five questions. Tick each one you can honestly answer yes to.
Step 4 of 6Mini-game: delegate or never?
AI can scan, group, tag and draft. It cannot be accountable. For each task: can AI help (with checks), or must you never delegate it?
Delegate?+10 XP
Running weekly threshold checks on routine data
Show answer
AI can help. Automation does it well, with you reviewing each alert.
Delegate?+10 XP
Deciding whether an alert is a real problem or a seasonal effect
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Grouping issues from monitoring visit reports
Show answer
AI can help. With every group confirmed against the reports.
Delegate?+10 XP
Choosing thresholds that balance catching problems and false alarms
Show answer
Never delegate. Listed in the profile as human judgement.
Delegate?+10 XP
Tagging anonymised hotline messages by site and theme
Show answer
AI can help. With a sample of tags checked by you.
Delegate?+10 XP
Raising possible diversion or exclusion
Show answer
Never delegate. Listed in the profile as human judgement: it goes through the right channels.
Delegate?+10 XP
Checking GPS and timestamps on third-party forms
Show answer
AI can help. With every flagged form investigated by you.
Delegate?+10 XP
Judging what follow-up is proportionate for each issue
Show answer
Never delegate. Listed in the profile as human judgement.
Step 5 of 6Six risks, six controls
What can go wrong when AI raises your alerts? Name three risks before you look.
Think about it for a moment, then reveal.
Risk
What it looks like
Your control
Alert fatigue
So many false alarms that teams stop reading
Seasonal rules tested on past weeks; accuracy reviewed each quarter
Missed problems
A threshold raised until real problems slip through
Count missed problems before any change; log every change
Wrong calculations
A script that averages the wrong weeks
Test every script on weeks you know
Exposed callers
A complaint detail that identifies who called
Anonymise before AI; sensitive findings through the agreed channel
Accusation from a flag
A third-party monitor called a fraud before any check
A flag is a question: investigate first
Alerts without action
Alerts sent, nothing changes
Every alert and issue in the action tracker until closed
Step 6 of 6Tell readers how AI was used
Transparency is a control too. Add a short note to every monitoring brief, and adapt it to what was actually done.
Alerts in this brief come from an automated weekly check (a Python script on ActivityInfo exports). An AI assistant grouped issues from visit reports and drafted this brief. Every alert and issue was reviewed by [name], Monitoring Specialist, against field evidence. Hotline messages were anonymised before any AI step.
The AI-Enabled M&E Manager is covered in Tutorial 01, the AI-Enabled M&E Specialist in Tutorial 02, the AI-Enabled Senior M&E Specialist in Tutorial 03, the AI-Driven Programme Performance Specialist in Tutorial 10 and the AI-Powered Community Feedback Analyst in Tutorial 18.
Step 2 of 5What you need to show
Strengths you build on
Process and output monitoring in the field
Spot checks and post-distribution monitoring
Working with programme teams on follow-up
Designing checklists and observation tools
What you need to show
An alert script tested on past weeks
Alerts reviewed before they were sent
AI-grouped issues confirmed against reports
A threshold change tested and logged
Sensitive findings handled through the right channel
Step 3 of 5Tool: build your LinkedIn headline
Pick the pieces that are true for you. Only claim what you have actually done.
Your headline
Step 4 of 5CV bullets and qualifications
CV bullet templates
Set up weekly early warning alerts on ActivityInfo data with Python and Power Automate, reviewing each alert before it reached programme managers.
Grouped issues from [40] monitoring visit reports per quarter with Claude, confirming each against the reports before it entered the action tracker.
Combined hotline complaint themes with distribution data to detect [problem], raised through the agreed confidential channel.
Cut false alarms from [X] to [Y] per quarter by tuning thresholds tested on past data, with every change logged.
Replace the brackets with what you have actually done.
Qualifications in the profile
Bachelor's or master's degree in development studies, statistics, social sciences or a related field.
3–6 years in programme monitoring, including field monitoring and routine data analysis.
Useful certification: MEAL DPro (PM4NGOs).
Step 5 of 5Check
Mission check+20 XP
What's the best way to use "AI-Powered Monitoring Specialist" on your CV?
Show answer
Use it to describe your capabilities, backed by evidence, unless your employer actually uses the title.
End of Mission 10
MISSION11
Your next steps
Commit and celebrate · Badge: Transition Ready
1 challenge
Mission goal: turn what you learned into a two-week plan and unlock your certificate.
Step 1 of 2Commit to your next two weeks
Pick the actions you commit to for the next two weeks. Your plan builds itself below.
Start learning
Do this
Show this
My plan
Recommended first project: the early warning system from mission 8, starting with the 25% rule tested on past weeks.
Step 2 of 2Your completion certificate
Complete all 11 missions to unlock it. Add your name, take a screenshot and share your progress.
AI & M&E Career Transition Series · Tutorial 07
Your name
has completed the learning path
From Monitoring Specialist to AI-Powered Monitoring Specialist
This is a learning-progress certificate. It records what you completed in this free tool, in your own browser; it is not independently verified and not a formal qualification. Your mission 8 project is the evidence to show an employer.
End of Mission 11
Questions and answers
What does an AI-Powered Monitoring Specialist do?
Focuses on the monitoring side of M&E: tracking whether activities are being implemented as planned, reaching the right people and running into problems. Uses AI to analyse monitoring data, identify implementation problems, detect trends and generate early warnings so programme teams can respond during implementation rather than after it.
How is AI changing the Monitoring Specialist role?
Monitoring used to depend on someone noticing problems in spreadsheets and visit reports weeks after they occurred. AI-assisted analysis can now scan routine data and field notes continuously and raise alerts, so the specialist spends more time setting sensible thresholds, separating real problems from noise and making sure alerts lead to action.
Which judgements should an AI-Powered Monitoring Specialist never hand over to AI?
The career profile lists these as human judgement:
Deciding whether an alert reflects a real implementation problem or a data or seasonal effect.
Choosing thresholds that catch problems without overwhelming teams with false alarms.
Raising sensitive findings, such as diversion or exclusion, through the right channels.
Judging what follow-up is proportionate for each issue.
Which skills should I learn first?
The learning priorities in the career profile, in order:
Predictive modelling and forecasting. Basic anomaly detection and trend analysis underpin useful early warning.
Python for data. Scripts make it possible to run routine checks on every new batch of data.
Text classification. Visit notes and feedback contain early signs of problems that need consistent categorising.
Adaptive management. Early warnings are only useful if they connect to clear decision points.
Power Automate. Alerts and follow-up reminders can be sent automatically to the right people.
The profile lists 12 skills in all: M&E planning, Indicator design, Data quality assessment, Predictive modelling and forecasting, Data visualization, Workflow automation, Text classification, Validating AI outputs, Accountability and feedback mechanisms, Adaptive management, Mobile data collection and Analytical report writing.
Which tools does this role use?
The career profile names: KoboToolbox, SurveyCTO, ActivityInfo, Excel, Python (pandas), Power BI, Power Automate, Claude, ChatGPT and LLM APIs (OpenAI, Anthropic, Google). The tutorial shows how to check every AI output, whichever tool you use.
What qualifications and experience does the profile list?
Bachelor's or master's degree in development studies, statistics, social sciences or a related field.
3–6 years in programme monitoring, including field monitoring and routine data analysis.
Useful certification: MEAL DPro (PM4NGOs).
How long does the transition take?
This tutorial suggests a 12-week plan of about 3 hours a week, in four stages: understand AI (weeks 1–2), apply AI (weeks 3–4), integrate AI (weeks 5–8) and demonstrate competence (weeks 9–12). It is a planning suggestion, not a requirement.
Is "AI-Powered Monitoring Specialist" an official job title?
No. It is a proposed career title from the EvalCommunity framework, not a standardised occupational classification or a guaranteed vacancy. Use it to describe your capabilities, backed by evidence, unless your employer actually uses the title.
