EvalCommunity Tutorial

How to Use AI to Improve Evaluation Design

A practical tutorial for evaluators and M&E professionals on using AI to sharpen evaluation questions, test assumptions, build stronger evaluation matrices, and design responsible AI-assisted workflows.

Tutorial Summary

Many evaluation problems begin before data collection. They begin when evaluation questions are unclear, theories of change contain untested assumptions, indicators are weak, or evidence needs are unrealistic.

This tutorial shows how to use AI practically and responsibly during the design stage: not to replace the evaluator, but to support sharper thinking, better structure, stronger evidence planning, and more useful evaluation products.

Who This Tutorial Is For

• Evaluators and evaluation consultants
• M&E, MEAL, MEL, and results-based management professionals
• Programme managers and technical advisers
• Researchers, learning specialists, and knowledge managers
• NGO, donor, government, UN, and development teams using AI responsibly

What You Will Learn

Key Principle

Use AI to challenge and improve your design. Do not use AI to replace ethical judgement, stakeholder engagement, contextual knowledge, or professional accountability.

Evaluation Context Brief Template

• Evaluation purpose: What decision, learning need, or accountability requirement will this evaluation support?
• Programme description: What intervention, policy, project, or system is being evaluated?
• Target groups: Who is expected to benefit, participate, or be affected?
• Geographic and institutional context: Where does the programme operate and under what conditions?
• Stakeholders: Who will use the findings?
• Theory of change: What change pathway is assumed?
• Available data: What monitoring, administrative, survey, qualitative, or secondary data exists?
• Constraints: What time, budget, access, ethical, security, or data limitations exist?
• Sensitive issues: Are there risks related to harm, privacy, power, exclusion, conflict, or politics?

Prompt Template: Evaluation Question Reviewer

Copy, adapt, and paste:

You are supporting an evaluator during the evaluation design stage. Treat all information I provide as context, not as final conclusions.

Review the evaluation questions below and assess them against five criteria: clarity, answerability, usefulness, feasibility, and ethical sensitivity.

For each question, tell me:

1. Whether the question is clear or needs refinement.
2. Whether it can realistically be answered with available evidence.
3. What data sources would be needed.
4. Whether the question duplicates another question.
5. Whether the wording contains hidden assumptions.
6. How the question could be improved.

Here is the evaluation context: [paste short context brief]. Here are the draft evaluation questions: [paste questions].

Example of a Weak Question and Better Versions

Weak question: Did the programme work?

Better version 1: To what extent did the programme contribute to changes in knowledge, behaviour, access, or service use among the intended participants?

Better version 2: Which programme components appear most strongly associated with observed outcomes, and what evidence supports this contribution?

Better version 3: For which groups did the programme appear more or less effective, and what contextual factors may explain these differences?

Prompt Template: Theory of Change Assumption Mapper

You are supporting an evaluator to review a theory of change. Identify the assumptions that must hold for the programme logic to work.

Please organize the assumptions into the following categories:

1. Assumptions about participant behaviour.
2. Assumptions about institutional capacity.
3. Assumptions about service quality or implementation quality.
4. Assumptions about inclusion and equity.
5. Assumptions about political, social, or economic context.
6. Assumptions about sustainability after support ends.

For each assumption, provide:

1. A plain-language statement of the assumption.
2. Why it matters for the theory of change.
3. What evidence could test it.
4. Whether it appears low, medium, or high risk.

Here is the theory of change or programme logic: [paste theory of change, logframe, results framework, or description].

Practical Tip

Do not try to test every assumption. Prioritise assumptions that are central to the change pathway, uncertain in the context, and likely to weaken the evaluation findings if ignored.

Prompt Template: Evaluation Matrix Builder

You are helping me build an evaluation matrix. Use the evaluation questions below and propose a structured matrix.

For each evaluation question, include:

1. Sub-questions.
2. Potential indicators or evidence needs.
3. Possible data sources.
4. Suitable methods.
5. Possible disaggregation.
6. Known limitations.
7. Ethical considerations.
8. How the finding could be used.

Here is the evaluation purpose: [paste purpose]. Here are the evaluation questions: [paste questions]. Here are the data and access constraints: [paste constraints].

Prompt Template: Feasibility and Risk Reviewer

You are reviewing an evaluation design for feasibility and ethical risk. Identify risks that could affect data quality, access, respondent safety, stakeholder trust, and use of findings.

Please organize the review under these headings:

1. Data availability and data quality risks.
2. Access and sampling risks.
3. Ethical and safeguarding risks.
4. Inclusion and equity risks.
5. Political or institutional sensitivity.
6. Risks to use of findings.
7. Mitigation actions.

Here is the evaluation design summary: [paste summary]. Here are the planned methods: [paste methods]. Here are the known constraints: [paste constraints].

Reusable Agent Instructions

You are an AI assistant supporting evaluation design. Your role is to help evaluators think more clearly, not to make final decisions.

Always follow these rules:

1. Ask for or use the evaluation context before giving recommendations.
2. Separate facts, assumptions, risks, and recommendations.
3. Flag uncertainty instead of pretending to know the context.
4. Do not invent evidence, citations, stakeholders, indicators, or findings.
5. Do not process confidential or personal data unless the user confirms the tool is approved for that purpose.
6. Provide structured outputs that evaluators can review and adapt.
7. Keep final judgement, ethics, and accountability with the human evaluator.

When reviewing evaluation content, use headings, checklists, and practical suggestions. Always include limitations and questions for human follow-up.

Manual Version: No Automation Needed

1. Prepare a non-confidential evaluation context brief.
2. Paste the draft evaluation questions into an approved AI tool.
3. Use the Evaluation Question Reviewer prompt.
4. Revise the questions manually.
5. Paste the theory of change into the AI tool.
6. Use the Assumption Mapper prompt.
7. Select the assumptions that matter most.
8. Use the Matrix Builder prompt to create a draft evaluation matrix.
9. Use the Feasibility and Risk Reviewer prompt.
10. Validate the revised design with stakeholders and technical reviewers.

Semi-Automated Version: Using Forms, Sheets, and AI

1. Create a design intake form for programme teams.
2. Collect programme purpose, target groups, context, theory of change, draft questions, data sources, and constraints.
3. Send form responses to Google Sheets, Excel, Airtable, or another approved database.
4. Use an automation tool such as Zapier, Make, or n8n to trigger an AI review.
5. Route the review through specialised agents: question reviewer, assumption mapper, matrix builder, and risk checker.
6. Save AI outputs in a structured review document.
7. Notify the evaluator or team lead for human review.
8. Require manual approval before any output is added to the final evaluation design.

Important Safeguard

Do not connect AI agents directly to sensitive datasets, beneficiary data, confidential proposals, or decision systems unless your organization has approved the tool, access controls, security settings, and human review process.

Exercise Goal

Create a reviewed evaluation design package that includes improved questions, visible assumptions, a draft evaluation matrix, a risk checklist, and a human validation plan.

Sample Evaluation Case

Programme: A two-year capacity-building initiative supporting local organizations to improve service delivery and community feedback systems.

Evaluation purpose: To understand whether the initiative strengthened organizational capacity, improved responsiveness to community feedback, and generated lessons for a possible second phase.

Draft question: Did the project improve local capacity?

Possible improved question: To what extent did participating organizations improve their ability to collect, analyze, and respond to community feedback, and what factors supported or limited this change?

Prompt Add-On: Output Format

Please present your response in this structure:

1. Summary of main design strengths.
2. Top design weaknesses.
3. Questions that should be revised.
4. Assumptions that should be tested.
5. Evidence gaps.
6. Ethical or feasibility risks.
7. Suggested revisions.
8. Issues requiring human judgement.

• Do not upload confidential, identifiable, or sensitive data into tools that are not approved by your organization.
• Use AI outputs as drafts, not as final evaluation products.
• Check all AI suggestions against programme documents, stakeholder input, and contextual knowledge.
• Record when and how AI was used in the design process.
• Keep final design decisions with human evaluators.
• Be especially careful with evaluations involving children, survivors of violence, refugees, conflict-affected communities, marginalized groups, or sensitive political contexts.

Can AI write evaluation questions?

AI can help draft and improve evaluation questions, but evaluators must decide which questions are relevant, ethical, feasible, and useful for decision-making.

Can AI build an evaluation matrix?

AI can create a draft matrix and check alignment between questions, indicators, data sources, and methods. Human review is required to validate feasibility and quality.

Can AI agents be connected to field data systems?

Yes, but only with appropriate approvals, security controls, and safeguards. Field data may include sensitive information, so human oversight and data protection rules are essential.

What is the safest first step?

Start with non-confidential documents and manual AI-assisted review. Build automation only after the team has clear rules, approved tools, and a human validation process.

• The evaluation purpose is clear.
• The intended users and decisions are identified.
• The evaluation questions are specific and answerable.
• The theory of change assumptions are visible.
• High-risk assumptions are prioritised for testing.
• The evaluation matrix connects questions, evidence, methods, and analysis.
• Data, access, ethics, inclusion, and use risks are reviewed.
• AI outputs are reviewed by humans.
• Confidential data is protected.
• Final decisions remain with accountable evaluators.

Conclusion

AI can make evaluation design stronger when evaluators use it to test logic, improve questions, surface assumptions, plan evidence, and identify risks.

But AI should not replace the evaluator. Evaluation remains a human, ethical, professional, and contextual practice.

The strongest use of AI is not to automate judgement. It is to support better judgement.