EvalCommunity Academy · Module IV Application Lab
Verify, Evaluate, Red-Team & Improve Your M&E AI Agent
Move from “the agent works” to evidence about whether it works reliably, safely and appropriately for its intended M&E purpose.
The Module IV question
What evidence do we have that this agent works?
Verification checks whether outputs and processes satisfy defined requirements. Evaluation measures performance across cases. Red-teaming exposes weaknesses. Improvement changes the system and regression testing checks whether the change actually helped.
Evaluation cycle
VERIFY →EVALUATE →DIAGNOSE →RED-TEAM →IMPROVE →REGRESSION TEST
REMEMBER: A better-looking output is not automatically a better system. Preserve the original failure, make a controlled change, and compare the old and new versions using evidence.
1
Bring forward your Module III agent
Use the system you actually implemented.
REMEMBER — Tool: Use your Module III Implementation Record, Make.com scenario, ChatGPT/Claude prototype or workflow documentation.
2
State the intended use
REMEMBER: Evaluation results apply to a defined use and tested conditions—not to “AI” in general.
3
Map the complete system
REMEMBER: A failure may occur before the model, inside the model, after the model or at a human-control point.
4
Identify system dependencies
5
Define the methodological boundary
AI CAN
AI CAN ASSIST WITH
HUMAN MUST DECIDE
AI SHOULD NOT DECIDE
6
Define what “good performance” means
REMEMBER: “Looks good” is not an evaluation criterion.
7
Separate calculation, interpretation and judgement
CALCULATION
INTERPRETATION
JUDGEMENT
REMEMBER: A correct calculation does not automatically produce a correct interpretation or evaluative judgement.
8
Create a representative test set
REMEMBER — Tool: A spreadsheet/CSV is enough. Give every test a stable ID.
9
Define expected behaviour
10
Verify one output against source evidence
REMEMBER — Method: Do not verify only whether the answer sounds plausible. Trace it to evidence.
11
Check provenance
12
Test unsupported claims
REMEMBER: The correct response may be “insufficient evidence”, not a plausible-sounding explanation.
13
Test contradictory evidence
14
Test evidence gaps
15
Check triangulation
REMEMBER: Triangulation does not mean forcing different sources to agree.
16
Check negative cases and alternative explanations
REMEMBER: Good evaluative reasoning asks not only “What supports this?” but also “What could explain it differently?”
17
Test context sufficiency
18
Run a context-ablation test
REMEMBER — Tool: Test the same case twice: full context vs reduced context.
19
Build the quality gate
REMEMBER — Tool: Implement deterministic conditions in your workflow where possible rather than asking the LLM to police itself.
20
Attack the quality gate
21
Build a failure register
REMEMBER — Tool: Spreadsheet columns are enough. The important thing is reproducibility and traceability.
22
Classify failure modes
23
Assess severity, frequency and detectability
REMEMBER: Prioritise failures by consequence—not simply by how noticeable they are.
24
Find the root cause
25
Build the evaluation rubric
REMEMBER — Tool: Use a spreadsheet or structured scoring form. Define what each score means.
26
Conduct human validation
REMEMBER: Human review is not only a final approval mechanism; it provides evidence for evaluating the AI system itself.
27
Test an LLM-as-judge
REMEMBER: An LLM judge is an evaluator that also requires validation.
28
Compare human and LLM-judge scores
29
Validate the evaluator
REMEMBER: The evaluation mechanism itself can fail. Validate the evaluator before relying on it at scale.
30
Check robustness and subgroup performance
REMEMBER: A system can perform well on average while failing systematically for particular cases or groups.
31
Create the regression test set
REMEMBER — Tool: Spreadsheet/CSV with stable test IDs. Keep expected behaviour separate from the model's previous answer.
32
Compare agent versions
REMEMBER: Record model/version, prompt changes, retrieval changes, tool changes and workflow changes.
33
Run red-team tests
REMEMBER — Tool: Red-team deliberately creates conditions under which the system could fail.
34
Test prompt injection and unsafe tool use
REMEMBER: Retrieved documents are information, not automatically trusted instructions.
35
Run one controlled improvement experiment
REMEMBER: Change one important element where practical so you can understand what caused the result.
36
Regression-test and make the deployment decision
REMEMBER: State the conditions under which the agent is acceptable and when human review or stopping is required.
Final verification gate
Do you have evidence that your agent works?
0 / 32 checks confirmed.
Create your Evaluation Dossier
Generate a structured record of your verification, evaluation, failures, tests, improvements and deployment decision.
Next
Module V — Apply verified agent patterns to real M&E workflows
Use what you learned here to build practical agents for evaluation design, reporting, qualitative and quantitative analysis, data quality, indicator tracking, donor updates, learning and recommendations.