When AI Enters the Evidence Process

What WHO’s 2026 guidance means for Monitoring & Evaluation

EvalCommunity Case Study Simulation | AI + M&E | Ethics, Evidence & Oversight

The WHO Case

WHO identifies three broad categories of AI-related health research:

01 — Research using data + AI
AI is used to collect or analyse large datasets to generate new knowledge.

02 — Research using AI tools
AI becomes part of the research process itself, for example helping formulate hypotheses, design studies, identify participants, monitor activities or produce research outputs.

03 — Research on AI
Research examines the utility, safety, efficacy or implementation of AI-based tools and technologies.

Now translate this to M&E.

Collect → Process → Analyse → Interpret → Synthesise → Communicate → Support Decisions

AI may influence several stages of the evidence chain simultaneously.

Why AI Changes the Evidence Process

1. The research-to-product cycle can be compressed

AI research can move rapidly from research to widely deployed products or applications.

M&E question: What happens if an AI-supported evaluation method is adopted across many programmes before its limitations are properly understood?

2. Results may not transfer across contexts

An AI system developed using one population or context may not perform appropriately elsewhere.

M&E question: Can an AI model trained on evaluation data from one country automatically be trusted in another?

3. Public or anonymized data does not eliminate ethical questions

Even research using anonymized or publicly available data can raise ethical and human-rights concerns.

M&E question: Does publicly available programme data automatically mean it is appropriate to put that data into an AI system?

4. The barrier to entry is lower

AI and data-science tools can make sophisticated analysis accessible to people who may not have traditional research training.

M&E question: Does access to an AI tool equal competence to conduct an evaluation?

5. AI can amplify existing risks

WHO identifies concerns including privacy, bias and discrimination, linked to how AI is designed, the data used to train it and how it is deployed.

M&E question: What happens when biased evaluation data becomes the training material for an AI system?

AI in the Evidence Workflow

AI Use Case 1 — Data Analysis

AI identifies patterns across thousands of observations.

Potential value: Faster exploration of complex datasets.

Evaluator questions:

• Is the underlying dataset appropriate?
• Could important groups be underrepresented?
• Can findings be independently reproduced?
• Is the model detecting meaningful relationships or merely correlations?

AI Use Case 2 — Qualitative Coding

An LLM processes thousands of interview transcripts and proposes themes.

Potential value: Large-scale qualitative analysis.

Evaluator questions:

• Were the transcripts appropriate to process through the AI system?
• How accurate is the coding?
• Are minority perspectives being overlooked?
• Who validates the themes?
• Are prompts and model versions documented?

AI Use Case 3 — Predictive Analysis

An AI model predicts which programme participants are most likely to experience a particular outcome.

Potential value: Early identification of patterns or groups requiring attention.

Evaluator questions:

• What data produced the prediction?
• How was the model validated?
• Does performance differ between groups?
• Is prediction being confused with causation?
• Who is responsible when the prediction is wrong?

AI Use Case 4 — Evidence Synthesis

AI summarizes hundreds of evaluation reports.

Potential value: Faster evidence synthesis.

Evaluator questions:

• Did the system omit important evidence?
• Can every conclusion be traced to its source?
• Did the AI introduce unsupported claims?
• Was contradictory evidence preserved?

AI Use Case 5 — Automated Reporting

AI produces the first draft of an evaluation report.

Potential value: Reduced administrative workload.

Evaluator questions:

• Who verifies every substantive finding?
• Are limitations accurately represented?
• Is uncertainty preserved?
• Is AI use disclosed?

The Oversight Gap

WHO identifies challenges including unclear regulatory boundaries, standards that may not address all AI-specific concerns, and limitations in the expertise and capacity of existing research ethics committees.

WHO also emphasizes that responsibility is shared across researchers, funders, data-access mechanisms, journals, regulators and other actors.

The M&E parallel: Evaluation has its own quality-assurance and oversight mechanisms. But when AI enters the evidence process, those mechanisms may need to address new questions about data, models, context, transparency, validation and accountability.

You Are the M&E Manager

Your organization wants to introduce an AI system that will analyse 5,000 evaluation interviews and 10 years of programme data.

Leadership says:

“The AI can complete the first analysis in two days. Let’s start next week.”

Your team has concerns about:

• data confidentiality;
• bias;
• model accuracy;
• qualitative interpretation;
• contextual differences;
• transparency; and
• human oversight.

What do you do first?

A. Start using the AI immediately because the evaluation team will review the final report.

B. Reject AI entirely because evaluation must remain human-led.

C. Conduct an AI-specific risk and governance assessment before deployment.

D. Let the AI system analyse the data and assess its own reliability.

A Human-First AI Workflow

Step 1 — Define the AI Use

Document exactly what AI will do.

Instead of: “We will use AI for the evaluation.”

Specify: “AI will classify interview excerpts into preliminary thematic categories. A qualified evaluator will validate, revise and document the final coding.”

Step 2 — Assess the Data

• What data will enter the system?
• Who owns it?
• Does it contain personal or sensitive information?
• Was consent obtained for this type of use?
• Can the data be processed safely?

Step 3 — Identify AI-Specific Risks

Bias | Privacy | Hallucination | Misclassification | Context Failure | Transparency | Automation Bias

Step 4 — Test Before Deployment

Test accuracy, consistency, subgroup performance, contextual validity, reproducibility and error rates before relying on AI outputs.

Step 5 — Maintain Human Oversight

AI Output → Human Review → Interpretation → Contextualization → Evaluation Judgment

AI produces an input into the evaluation process. It does not automatically become the evaluator.

Step 6 — Document AI Use

Can AI Evaluate the Evaluator?

WHO examines the potential use of generative AI and automated text analysis to assist research ethics committees.

This leads to a parallel M&E question:

Could AI be used to review the quality of an evaluation?

An AI system could potentially review an evaluation protocol and flag:

• missing ethical considerations;
• potential sampling problems;
• inconsistent indicators;
• possible bias;
• weak data-quality procedures;
• insufficient stakeholder engagement;
• unexplained methodological assumptions; and
• inadequate disclosure of AI use.

That could be useful.

But a second question immediately follows:

Who validates the AI’s assessment?

Human-First principle: AI can support quality assurance. AI does not automatically become the authority on quality.

The Governance Ecosystem

WHO emphasizes that responsibility cannot rest solely with research ethics committees. Different actors may contribute to oversight, including researchers, funders, data hubs, data-access committees, scientific journals and regulatory agencies.

For M&E, this suggests an analogous ecosystem:

Evaluator
↓
M&E / Evaluation Manager
↓
Data & Information Governance
↓
Organization / Ethics / QA
↓
Funder / Commissioner
↓
Stakeholders & Communities
↓
Decision-Makers

AI may operate across the entire chain.

Accountability must therefore operate across the entire chain as well.

The Global South Question

WHO gives particular attention to AI-related research in low- and middle-income countries, including fairness and inclusion, equitable benefit sharing, data colonialism, ethics dumping, power inequities and capacity-building.

Ask these questions:

• Who designed the system?
• Whose data trained it?
• Whose assumptions are embedded in it?
• Who validates it in the local context?
• Who benefits from its use?
• Who carries the risks when it is wrong?

The EvalCommunity AI Governance Checklist

What M&E Professionals Should Take Away

AI can change how quickly evidence is produced.

But speed does not automatically produce validity, quality, ethical integrity, contextual relevance, transparency or accountability.

Do not evaluate only the AI output. Evaluate the entire evidence process that produced it.

That means examining the data, model, method, context, assumptions, validation, human judgment and decision pathway.

Reflection Questions

1. Where exactly does AI enter my evidence process?
2. What could AI get wrong?
3. Which populations could be disproportionately affected?
4. What evidence do I have that the AI performs adequately in my context?
5. Who reviews the AI’s work?
6. Can I explain how AI influenced the final findings?
7. Would stakeholders understand the way AI was used?
8. If the AI is wrong, who is accountable?

Source

World Health Organization (2026). Artificial intelligence-related health research: Ethics review and oversight. Geneva: World Health Organization.

Read the original WHO report →

This EvalCommunity case study is an educational adaptation of the WHO report. The original WHO document addresses AI-related health research and research ethics oversight. The M&E scenarios, questions and applications presented here are an EvalCommunity adaptation of the issues discussed by WHO and are not WHO recommendations specifically for Monitoring & Evaluation.

Confidentiality & educational use: This case study is intended for professional learning, discussion and educational purposes. The hypothetical M&E scenarios are illustrative and do not represent confidential information or a specific organization’s evaluation.