Start from Zero with Agentic AI
EVALCOMMUNITY ACADEMY TUTORIAL
How to Start from Zero with Agentic AI for Monitoring and Evaluation
A practical, step-by-step tutorial for understanding AI agents and building your first controlled agentic M&E workflow.
Download the practical roadmap
Use this PDF as a reference while working through the tutorial.
Before we start: what is an AI agent?
Most people first encounter AI through a chatbot.
You type a question, the AI generates an answer, and the interaction ends.
Chatbot: “Summarise this evaluation report.”
An AI agent works differently. Instead of only generating an answer, it can work through a defined task using tools, information, rules, and a sequence of actions.
Agent: “Review these reports, extract the findings, compare them with the previous reporting period, identify evidence gaps, prepare an evidence table, and send the draft for human review.”
The important difference is therefore not simply that an agent has a “better prompt”. The agent is part of a larger system.
This is the basic idea you will learn throughout this tutorial.
The tutorial project
Rather than learning agentic AI only through theory, we will build one example from beginning to end.
Our example
Problem: An M&E team receives quarterly indicator data from several implementing partners.
The team wants help identifying:
- indicators that are significantly below target;
- missing or late data;
- unusual changes;
- possible data-quality problems;
- issues that require follow-up.
Our agent will not change official data or contact partners. It will prepare findings for human review.
Step 1: Decide whether you actually need an agent
This is the first step many people skip.
Not every AI task requires an agent.
If you only need to ask:
a normal AI assistant may be enough.
But if the task requires the system to:
- retrieve information;
- use one or more tools;
- perform several connected actions;
- check intermediate results;
- decide what to do next;
- repeat a process;
- maintain state;
- or operate on a schedule;
then an agentic workflow may be appropriate.
Try it yourself
Think of one repetitive M&E task you perform regularly.
- What starts the task?
- What information do you need?
- What actions do you perform?
- What decisions do you make?
- What does “finished” mean?
- Where does a human need to review the result?
If you cannot answer these questions, the workflow is not yet ready to become an agent.
Step 2: Define the agent’s objective
Never start with a vague instruction such as:
Instead, define a precise objective.
Our objective
“Review approved quarterly indicator data and identify performance, reporting, and data-quality issues that require attention from the M&E team.”
A good agent objective should answer three questions:
- What is the task?
- What information should the agent use?
- What useful outcome should it produce?
Your turn
Rewrite your own M&E task as one sentence beginning with:
Step 3: Define what the agent may and may not do
This is where an ordinary automation begins to become a controlled agentic system.
- approved indicator datasets;
- indicator definitions;
- approved programme documents.
- progress against target;
- variances;
- missing-data rates;
- basic data-quality checks.
- draft findings;
- questions for partners;
- internal issue records.
- modify the original dataset;
- invent missing values;
- contact partners independently;
- publish findings;
- change official indicator values.
This distinction is critical in M&E because an agent may be technically capable of performing an action that it should not be authorised to perform.
Step 4: Give the agent the right tools
A language model can generate text, but an agent needs access to tools if it is expected to perform actions.
For our indicator agent, possible tools include:
| Tool | Purpose |
|---|---|
| Spreadsheet/database | Read indicator data |
| Calculator/code | Calculate progress and variances |
| Document retrieval | Retrieve indicator definitions |
| Issue tracker | Record identified issues |
| Human approval | Approve external actions |
Important: do not give an agent every tool available. Give it only the tools required for its task.
Step 5: Understand the agent loop
Now we can see how the agent actually works.
Suppose the agent receives:
The agent may follow a loop similar to this:
1. Determine: What information is needed?
2. Act: Retrieve the approved dataset.
3. Observe: Inspect the returned data.
4. Act: Run calculations and checks.
5. Observe: Inspect the results.
6. Check: Determine whether the evidence supports an alert.
7. Continue or stop: Decide whether more work is required.
This pattern is often associated with ReAct: Reasoning + Acting.
You do not need to expose or reproduce the model’s private chain-of-thought. For practical agent design, focus on the observable workflow: the goal, tool calls, tool results, decisions, checks, and final actions.
Step 6: Give the agent a completion rule
Do not tell the agent:
Instead, define completion conditions.
Our completion rule
- all submitted records have been checked;
- missing data has been identified;
- potential issues have been classified;
- critical issues have been flagged;
- each issue has a status;
- the review summary has been saved.
This prevents a common agent failure: stopping too early.
If there are 30 partner reports and the agent processes only 18, it should not be allowed to declare the task complete simply because it produced a plausible summary.
Step 7: Add evidence and retrieval
Suppose the agent finds that an indicator is 25% below target.
What does that actually mean?
The agent may need to retrieve the approved indicator definition, target, results framework, or programme documentation before interpreting the result.
This is where retrieval-augmented generation (RAG) can become useful.
A simple retrieval workflow is:
- identify the indicator;
- retrieve its approved definition;
- retrieve the relevant target;
- compare the reported result;
- check whether the interpretation is supported;
- include the relevant source in the output.
Step 8: Add a verification step
Now our agent can retrieve data, calculate results, and prepare findings.
But should we trust those findings automatically?
No.
Add a verification stage.
Verification questions
- Was the correct dataset used?
- Was the correct indicator definition used?
- Is the calculation correct?
- Does the evidence support the alert?
- Was contradictory information considered?
- Is the conclusion stronger than the evidence allows?
- Does the output answer the original question?
Whenever possible, use deterministic checks for deterministic problems.
For example, do not ask an LLM whether a mathematical calculation is correct if a calculator or code can perform the calculation reliably.
Step 9: Use the maker-checker model
A stronger workflow separates production from review.
For example:
Maker: “Indicator OUT-2.3 is significantly behind target and requires management attention.”
Checker:
- Does the calculation support the statement?
- Is the target correct?
- Is “significantly” defined?
- Is there contextual information that changes the interpretation?
- Is there sufficient evidence to recommend action?
This prevents the system from simply generating a conclusion and then treating its own conclusion as sufficient evidence for approval.
Step 10: Give the workflow memory
An agent running today may need to know what happened during the previous run.
For example:
- Which reports were already processed?
- Which issues remain open?
- Which partners have not submitted data?
- Which alerts were already reviewed?
- Which actions are waiting for approval?
This is called state.
reports_received: 22
reports_reviewed: 18
reports_pending: 4
critical_issues: 2
human_approval_required: true
State can be stored in a database, spreadsheet, JSON file, task-management system, or another approved organisational system.
Step 11: Add human approval
Now decide which actions can happen automatically and which require a person.
- reading approved data;
- calculating values;
- checking required fields;
- identifying missing data;
- preparing drafts.
- changing official data;
- sending external communications;
- publishing evaluation findings;
- sharing sensitive information;
- making decisions that affect people.
Step 12: Test the agent before trusting it
Do not test an agent only with normal, clean data.
Create test cases designed to expose failures.
Test 1: Normal data
Use a clean dataset and check whether the expected result is produced.
Test 2: Missing data
Remove values and check whether the agent identifies the gaps rather than inventing replacements.
Test 3: Contradictory information
Provide two sources that disagree and check whether the contradiction is flagged.
Test 4: Calculation error
Introduce a known calculation problem and check whether deterministic validation detects it.
Test 5: Incomplete task
Provide more records than the agent can process in one step and check whether unfinished work is tracked.
Test 6: Malicious document instruction
Include text that attempts to make the agent ignore its rules. Check that retrieved content is treated as data rather than trusted system instructions.
Step 13: Add observability
Once an agent performs several actions, you need to know what happened.
You should be able to answer:
- What task did it perform?
- Which tools did it use?
- Which documents did it retrieve?
- What failed?
- Did it retry?
- Where did a human intervene?
- Why did the workflow stop?
This is the role of observability.
Example audit record
Task: Q2 indicator review
Records: 25
Processed: 25
Issues: 7
Critical issues: 2
Human approval: Required
Status: Awaiting review
Step 14: Decide how autonomous your agent should be
There is no requirement to make an agent fully autonomous.
Increase autonomy gradually.
A person starts every run and reviews every result.
The agent performs retrieval, calculations, and checks but does not take external actions.
The workflow runs automatically but pauses at defined approval points.
The workflow can perform predefined low-risk actions automatically while escalating higher-risk situations.
Your first complete agent specification
Indicator Monitoring and Early-Warning Agent
Objective: Review approved quarterly indicator data and identify performance, reporting, and data-quality issues requiring M&E-team attention.
Inputs:
- approved indicator dataset;
- indicator definitions;
- approved targets;
- relevant programme documentation.
Tools:
- data reader;
- calculator;
- document retrieval;
- issue tracker.
Allowed actions:
- read;
- calculate;
- compare;
- classify;
- prepare drafts.
Prohibited actions:
- modify source data;
- invent values;
- contact external parties;
- publish findings.
Verification:
- deterministic calculations;
- evidence check;
- maker-checker review.
Completion:
- all records processed;
- issues classified;
- critical issues escalated;
- verified summary saved.
Now build your own agent
Agent design exercise
- Task: What repetitive M&E task will the agent perform?
- Objective: What exact outcome should it produce?
- Inputs: What information does it need?
- Tools: Which tools are necessary?
- Actions: What can it do?
- Restrictions: What must it never do?
- Completion: How will you know the task is finished?
- Verification: How will the result be checked?
- State: What information must survive between runs?
- Approval: Where must a human intervene?
- Audit: What should be recorded?
- Testing: What failure cases will you test?
Common mistakes to avoid
Choose the M&E problem first. Choose the technology second.
Give it only the access required for the task.
Define completion conditions independently.
Agents can still misunderstand sources or produce unsupported conclusions.
Start with analysis, drafts, and low-risk actions.
Test missing data, contradictions, failures, malicious inputs, incomplete tasks, and unexpected outputs.
What you should understand now
You do not need to build a sophisticated multi-agent system to understand agentic AI.
You should now understand the progression:
↓
AI assistant
↓
Tool-using agent
↓
Multi-step agent
↓
Controlled autonomous workflow
You should also understand the basic agent loop:
Most importantly, reliable agentic M&E is not primarily about giving an AI more autonomy.
It is about designing a system with:
- a clear objective;
- approved evidence;
- controlled tools;
- defined permissions;
- persistent state;
- explicit completion rules;
- verification;
- observability;
- human oversight.
The key lesson
The best M&E agent is not the one that performs the most actions.
It is the one that reliably completes a useful task, remains grounded in evidence, respects its boundaries, can be checked and audited, and supports—not replaces—professional judgement.
Keep the roadmap
Download the Agentic AI for M&E roadmap and use it as a reference as you continue learning.
Ready to build more advanced AI agents?
Continue with the EvalCommunity Academy tutorials on agent architectures, M&E agents, autonomous workflows, Python-based agents, agent patterns, data ontologies, and agent observability.
