How can AI expand the evidence base of an evaluation?

Large-scale evaluations can involve thousands of documents, multiple languages
and many different sources of evidence. Processing this information manually can
be time-consuming and may limit how much evidence an evaluation team can review.

The challenge is not simply to process more information. Evaluation teams also
need to maintain data integrity, traceability, human judgement and methodological
quality.

The United Nations Population Fund (UNFPA) provides a useful
real-world example. Its independent evaluation of support to the
2020 round of population and housing censuses used AI to
support document review and analysis across a large multilingual evidence base.

The evaluation covered UNFPA’s census support from 2015 to 2024 and drew on
nearly 1,300 documents, key informant interviews, 16 country
case studies, a country office survey and a regional study of Latin America
and the Caribbean.

Read the original UNFPA case

1. The evaluation challenge

The evaluation had to bring together evidence from multiple sources while
examining UNFPA’s support to the 2020 census round across a very large
geographic and documentary landscape.

The evidence base included:

• Nearly 1,300 documents
• Key informant interviews
• 16 country case studies
• A country office survey
• A regional study of Latin America and the Caribbean

AI was introduced to help the evaluation team process documentary evidence
at a scale that would have been difficult to manage through manual review alone.

2. A multi-tool AI workflow

UNFPA did not use one AI system to perform the entire evaluation. Instead,
three specialized tools supported different stages of the document-review
workflow.

The workflow illustrates an important principle for M&E:
AI can be assigned specific tasks rather than being given unrestricted
responsibility for the entire evaluation process.

3. AI-assisted translation expanded the evidence base

DeepL Pro was used to translate Spanish and French documents from three
large institutional datasets:

• Annual narrative reports
• Country programme documents
• Country programme evaluations

This enabled the analysis to draw on the multilingual evidence base rather
than limiting the documentary review to English-language material.

4. AI-assisted document filtering and extraction

InsightWise was then used to filter documents for census-relevant content.
UNFPA selected the three datasets partly because of their scale and relatively
standardized formats, which suited AI-augmented extraction.

The process produced three structured databases of extracted text. Excerpts
were tagged using analytical variables such as country and
region.

This transformed a large collection of documents into structured evidence
that could be queried and analysed more efficiently.

5. Claude was used in two stages

Claude AI was used in two distinct phases of the evaluation.

Before fieldwork

The first analysis generated preliminary syntheses linked to the evaluation
questions. These outputs helped inform data collection for the country
case studies.

During reporting

The second analysis was deliberately narrower. Claude was used for targeted
frequency analyses, including counting references to Chief Technical Advisers,
training topics and census technologies.

The team deliberately prompted Claude for counts and lists rather
than interpretive analysis.

6. Human verification remained essential

AI outputs were validated against the original source documents
before being integrated into the final analysis.

The workflow can therefore be represented as:

This is particularly important in evaluation because an error in extraction
or retrieval can influence later analysis if it is not detected.

7. AI expanded triangulation

The expanded document review allowed the evaluation team to triangulate
findings from the country case studies and regional analysis against
UNFPA’s census support worldwide.

According to UNFPA, this broader evidence base helped address evidence gaps
and enabled findings to be generalized across the 160 countries
and territories supported during the 2020 census round, rather
than being limited to the 16 country case-study contexts.

8. Prompt engineering mattered

The UNFPA experience also highlighted the importance of precise prompt
engineering.

The source reports that reliable outputs depended on framing prompts around
verifiable evidence, lists and frequency counts, rather than
broad narrative synthesis or interpretation.

This suggests a useful principle for M&E professionals:

9. AI governance came before processing

A dedicated governance framework guided AI use throughout the evaluation.
It was aligned with:

• UNFPA’s strategy for generative AI in evaluation
• The United Nations Principles for the Ethical Use of Artificial Intelligence
• UNFPA’s information security policy

Enterprise-grade tools were deployed under strict
zero-data-retention protocols. UNFPA states that this ensured
its data was not stored or used to train external models.

For evaluation teams working with confidential information, this illustrates
why AI governance and data protection need to be considered before
uploading evaluation material to an AI system.

10. AI was part of a broader evaluation strategy

The census evaluation was one of several exercises through which UNFPA’s
Independent Evaluation Office is scaling AI across the evaluation function.

UNFPA’s 2024 strategy on generative AI in evaluation established six principles
and a phased plan for responsible AI use.

The safeguards described by UNFPA include:

• Ethical AI clauses in consultant contracts
• AI disclaimers in reports
• Protocols for human verification
• Internal tracking of evaluation quality
• Internal tracking of efficiency gains

The strategy also shaped UNFPA’s internal
Evaluation Assistant, an AI platform intended to make
evaluation evidence more accessible for timely decision-making.

11. What benefits did UNFPA report?

UNFPA reports that integrating AI produced notable gains in:

• Cost efficiency
• Time savings
• Overall quality
• Analytical reach

However, the source article does not provide numerical estimates for these
gains. They should therefore be treated as reported qualitative
benefits, not quantified effect sizes.

12. The complete AI-supported evaluation workflow

13. What can M&E professionals learn from this case?

1. Start with governance. Define data-protection and AI-use requirements before processing evidence.
2. Use AI for defined tasks. Translation, filtering, extraction and counting can be separated into controlled workflows.
3. Keep important outputs verifiable. Link AI-generated results back to the original evidence.
4. Keep interpretation with evaluators. AI-supported retrieval is not the same as evaluation judgement.
5. Use AI to expand triangulation. A larger usable evidence base can strengthen comparison across contexts.
6. Track quality as well as efficiency. Faster processing is useful only if evidence quality remains acceptable.

14. Practical checklist for AI-supported evaluation

Continue learning with EvalCommunity

AI Tool Evaluation Protocol for M&E

— Learn how to test, compare and document AI tools used in M&E.

AI Evidence Synthesis for M&E Literature Reviews

— Learn how AI can support structured evidence-synthesis workflows.

Using AI Safely with Sensitive M&E Data

— Practical guidance for responsible AI use with sensitive evaluation information.