Back
EvalCommunity Academy · Module 1 Application Lab

AI Agent Opportunity & Suitability Assessor

Work through a real M&E task and decide whether it needs human work, AI assistance, deterministic automation, an LLM-enabled workflow, or a genuine AI agent.

M&E first Evidence-aware Human oversight No unnecessary AI
Core principle
The evaluation problem comes before the technology.

Do not choose an agent because the task sounds complicated. First understand the methodological task, evidence requirements, risks, verification possibilities and human responsibilities.

Your laboratory pathway
1
Define problem
→
2
Assess task
→
3
Check context
→
4
Challenge AI
→
5
Decide
STEP 1 · DEFINE

Define the M&E problem

Describe the methodological task before deciding what technology to use.

REMEMBER — How to do this
Describe the task, inputs, process and intended output. Do not start with “I want an AI agent.”

Tool: You can draft this in Word, Google Docs, ChatGPT or Claude.
M&E example: “Compare monthly indicator values with targets, identify unusual changes and prepare evidence for evaluator review” is better than “Use AI to monitor indicators.”
Check: Could another evaluator understand exactly what work you want to improve?
STEP 2 · ASSESS

Assess the characteristics of the task

Rate each characteristic from 1–5. These ratings are a structured reflection tool, not a validated diagnostic score.

1. Frequency
1 = rare · 5 = very frequent
2. Information volume
1 = small · 5 = very large
3. Variability
1 = fixed · 5 = highly variable
4. Verifiability
1 = difficult · 5 = readily verifiable
5. Tools / retrieval
1 = unnecessary · 5 = essential
6. Multiple steps / iteration
1 = one step · 5 = iterative
7. Consequence of error
1 = low · 5 = very high
8. Professional judgement
1 = minimal · 5 = very high
REMEMBER — How to rate this
Do not rate based on whether you personally like AI. Rate the task itself.

Example: A monthly monitoring task may score high for frequency and information volume, but high risk and high methodological judgement do not mean “use no AI.” They mean that AI may require stronger human control.

Key lesson: task complexity may justify exploring agentic support; risk and professional judgement determine the degree of delegation and review required.
STEP 3 · CONTEXT

Check the methodological context

An agent cannot produce a defensible M&E output if it lacks the context required to interpret the evidence.

REMEMBER — How to prepare context
In ChatGPT or Claude: provide the methodological instructions and approved reference material before testing the task.

For an automated workflow: the system must explicitly pass the required context and evidence to the model. Do not assume that context from a previous ChatGPT conversation automatically exists in an API or Make.com workflow.

Check: Can the agent distinguish an indicator definition, an observed value, an interpretation and a judgement?
STEP 4 · CHALLENGE

Challenge the agent assumption

Before selecting an agent, identify the least complex solution that could perform the task adequately.

REMEMBER — Do not confuse complexity with agentic need
A task can be complex but still be handled by a deterministic workflow plus human review.

An agent becomes more defensible when the task genuinely requires things such as multiple steps, conditional decisions, retrieval, tool use, iteration, context/state or escalation.

M&E example: “Calculate whether an indicator is above or below target” may only need a formula. “Investigate why an indicator changed, retrieve contextual evidence, compare explanations and escalate uncertainty” may justify testing an agent.
STEP 5 · HUMAN BOUNDARY

Define what remains human

AI CAN
Retrieve, organise, classify, compare, calculate with appropriate tools and draft structured outputs.
AI CAN ASSIST WITH
Coding, synthesis, interpretation, hypothesis generation and prioritisation when reviewed.
HUMAN MUST DECIDE
Evaluative judgement, contextual interpretation, competing explanations and consequential recommendations.
AI SHOULD NOT DECIDE
High-stakes methodological or accountability decisions that cannot be adequately verified.
REMEMBER — How to implement human oversight
Do not write only “human review required.” Identify the exact control point.

Example: AI detects an unusual indicator change → retrieves evidence → drafts possible explanations → evaluator checks evidence and decides whether the issue is substantively important.

If you later automate the process in Make.com, this control point should become a deliberate approval or escalation step.
WORKED M&E EXAMPLE

Monitoring and indicator investigation

A monitoring team receives monthly indicator data. The proposed system checks changes, retrieves relevant reports and prepares issues for evaluator review.

Data → Quality check → Compare → Detect unusual change → Retrieve context → Compare explanations → Escalate → Human judgement
Why this may justify an agent: the task involves repeated monitoring, retrieval, conditional investigation and escalation. However, the agent should not decide that the programme succeeded or failed.
What would not justify an agent: simply calculating whether a value is above or below a target. A spreadsheet formula or deterministic workflow may be sufficient.

Make your decision

Complete the assessment before interpreting the recommendation.

FINAL REFLECTION

What have you learned from the assessment?

PROBLEM
Did I define the evaluation problem before selecting technology?
FIT
Does AI actually add value?
AGENCY
Does the task really require an agent?
CONTROL
What must remain human-led?
EVIDENCE
How could I verify that the system works?
Important: A decision not to build an AI agent is a successful outcome when a simpler solution is more appropriate.
"+ "" ); popup.document.close(); setTimeout(function(){ popup.focus(); popup.print(); },400); }); showStatus("Assessment complete. Review the recommendation rather than accepting it automatically.",true); get("m1result").scrollIntoView({ behavior:"smooth", block:"start" }); } function fallbackCopy(text){ var area=document.createElement("textarea"); area.value=text; area.style.position="fixed"; area.style.left="-9999px"; document.body.appendChild(area); area.focus(); area.select(); try{ document.execCommand("copy"); showStatus("Copied. Your decision record is on the clipboard.",true); }catch(e){ showStatus("Copy was not available in this browser. You can select the assessment text manually.",false); } document.body.removeChild(area); } function loadExample(){ get("m1task").value= "Review monthly monitoring information, compare indicator performance with approved targets, identify unusual changes, retrieve relevant contextual evidence and prepare issues for evaluator review."; var ratings={ m1frequency:5, m1volume:4, m1variability:4, m1verify:4, m1tools:4, m1steps:4, m1risk:3, m1judgement:4 }; Object.keys(ratings).forEach(function(name){ var el=root.querySelector( 'input[name="'+name+'"][value="'+ratings[name]+'"]' ); if(el){ el.checked=true; } }); get("m1input").value="Mixed evidence"; get("m1use").value="Informs a decision but is reviewed"; get("m1context").value= "Indicator definitions, approved targets, reporting period, programme context, theory of change assumptions, evaluation questions and rules for interpreting material deviations."; get("m1evidence").value= "Monitoring database, monthly reports, programme documents, contextual reports and relevant evaluation evidence."; get("m1simple").value="AI agent"; get("m1reason").value= "A deterministic workflow could calculate deviations, but the broader task requires retrieval, conditional investigation, comparison of evidence and preparation for evaluator review. An agent may therefore be worth testing while consequential judgement remains with the evaluator."; get("m1human").value= "The evaluator verifies the evidence, considers alternative explanations and decides whether an observed change constitutes an evaluative finding."; showStatus( "Worked example loaded. Review the fields and click “Assess my task”.", true ); } function saveAssessment(){ try{ var data={ fields:{}, radios:{} }; root.querySelectorAll("textarea,select").forEach(function(el){ data.fields[el.id]=el.value; }); root.querySelectorAll('input[type="radio"]:checked').forEach(function(el){ data.radios[el.name]=el.value; }); localStorage.setItem( "ecm1lab_saved", JSON.stringify(data) ); showStatus( "Saved locally. No information has been sent to an external service.", true ); }catch(e){ showStatus( "Your browser did not allow local storage. You can still use the assessment normally.", false ); } } function restoreAssessment(){ try{ var raw=localStorage.getItem("ecm1lab_saved"); if(!raw){ showStatus( "No saved assessment was found in this browser.", false ); return; } var data=JSON.parse(raw); if(data.fields){ Object.keys(data.fields).forEach(function(id){ var el=get(id); if(el){ el.value=data.fields[id]; } }); } if(data.radios){ Object.keys(data.radios).forEach(function(name){ var el=root.querySelector( 'input[name="'+name+'"][value="'+data.radios[name]+'"]' ); if(el){ el.checked=true; } }); } showStatus( "Restored. Your saved assessment has been loaded.", true ); }catch(e){ showStatus( "The saved assessment could not be restored.", false ); } } function resetLab(){ root.querySelectorAll("textarea").forEach(function(el){ el.value=""; }); root.querySelectorAll("select").forEach(function(el){ el.selectedIndex=0; }); root.querySelectorAll( 'input[type="radio"],input[type="checkbox"]' ).forEach(function(el){ el.checked=false; }); get("m1result").style.display="none"; get("m1result").innerHTML=""; get("m1status").style.display="none"; window.ecm1Decision=""; } get("m1assess").addEventListener("click",assess); get("m1example").addEventListener("click",loadExample); get("m1save").addEventListener("click",saveAssessment); get("m1restore").addEventListener("click",restoreAssessment); get("m1reset").addEventListener("click",resetLab); root.querySelectorAll(".m1complete").forEach(function(el){ el.addEventListener("change",function(){ if(this.checked){ showStatus( "Step confirmed. Continue with the laboratory.", true ); } }); }); })();