Introduction

UNFPA’s use of AI in the independent evaluation of its humanitarian action provides a practical example of how artificial intelligence can support large-scale evaluation work without replacing professional evaluation judgment. The evaluation of UNFPA’s capacity in humanitarian action from 2019 to 2025 covered 15 countries, six humanitarian contexts, more than 1,500 documents and publications, and over 600 interviews and focus group discussions.

The scale of the evidence base created a methodological challenge: processing the material manually at the required pace and depth would have limited both the breadth and rigour of the evaluation. The evaluation team therefore explored whether AI could support experienced evaluators responsibly, particularly in evidence processing, multilingual analysis, synthesis, triangulation, and cross-country comparison.

This case study is relevant for evaluators because it shows a carefully governed, ethics-first approach to using AI in a sensitive humanitarian evaluation. It demonstrates how AI can increase efficiency and improve evidence coverage while keeping human validation, evaluator expertise, transparency, and responsible data protection at the centre of the process.

Case Background

Humanitarian evaluations often require teams to synthesize large volumes of documentary, interview, and focus group evidence across multiple countries and crisis contexts. In the UNFPA case, the independent evaluation of humanitarian capacity from 2019 to 2025 included extensive multilingual evidence, with a significant proportion of source material in French and Spanish.

The evaluation team saw an opportunity to test whether AI could improve the breadth, pace, and rigour of evidence processing. However, the team did not begin with technology. It first developed a deliberate ethics-first strategy, then selected tools that could meet evaluation needs while respecting security, privacy, multilingual inclusion, and human oversight requirements.

The case is also part of a broader institutional commitment. UNFPA’s Independent Evaluation Office had already developed a strategy for a GenAI-powered evaluation function, including principles and a phased roadmap for responsible use. The humanitarian evaluation became one of the first centralized evaluations to put those principles into practice at scale.

The Evaluation Problem

The core problem was scale. The evaluation team needed to process more than 1,500 documents and publications and over 600 interviews and focus group discussions across 15 countries. A purely manual process would have required weeks of sequential review and could have constrained the depth and consistency of analysis.

The second challenge was language inclusion. Four of the sampled countries were francophone and three were Spanish-speaking. If multilingual evidence were more difficult to process, there was a risk that evidence from these contexts could be underused or deprioritized in the synthesis.

The AI Strategy

The evaluation team tested two tools: InsightWise and Google NotebookLM. NotebookLM was selected for most of the analytical work because it was easy to use, integrated with Google Translate, and supported processing of French and Spanish materials alongside English-language documents.

The selection also reflected UNFPA IEO’s institutional preference for tools within the Google ecosystem, which met information security and data privacy requirements. This was especially important because the evaluation handled sensitive humanitarian evidence, including anonymized interview and focus group material.

How AI Was Used

1. Data analysis

Before any interview or focus group data were ingested into the platform, the team removed all personal identifiers. AI then helped generate concise summaries and identify recurring themes, patterns, and emergent evidence across a very large transcript dataset. This did not replace manual coding; it complemented the evaluation team’s coding against a pre-agreed evaluation matrix.

2. Secondary data review

AI supported structured scanning and extraction across the 1,500-document evidence base. It helped draw out relevant passages and organize them by country, evaluation assumption, and timeframe.

3. Synthesis

AI assisted with cross-country comparisons and flagged points of convergence and divergence between data sources. These signals helped evaluators identify areas of agreement, contradiction, or tension that required deeper professional scrutiny.

Governance and Safeguards

The evaluation team embedded human oversight at every stage. AI-generated outputs were not accepted automatically. Evaluators reviewed outputs, checked them against primary and secondary sources, and corrected or discarded results that were inaccurate, superficial, or reflected bias.

Two workshops strengthened external quality control. An analysis workshop with evaluation managers took place immediately after data collection, and a co-creation workshop enabled the full Evaluation Reference Group to critically review AI-supported findings.

The process was governed by UNFPA IEO’s GenAI-powered evaluation strategy and the UNEG Ethical Principles for Harnessing AI in United Nations Evaluations. Data protection was also supported by UNFPA’s enterprise agreement with Google, which helped ensure that sensitive evaluation data, even in anonymized form, was not retained or used beyond the immediate analytical task.

Transparency was built into the evaluation report through an annex documenting how AI was applied, which tools were used, and how ethical and responsible use was upheld.

Results and Limitations

Tangible efficiency gains

AI reduced the time needed to process and structure a dataset that would otherwise have required weeks of sequential manual review. The process delivered a verified cost saving of $12,000 and gave evaluators more time to focus on interpretation, validation, and professional judgement.

Stronger multilingual inclusion

The multilingual processing capability helped ensure that evidence from francophone and Spanish-speaking contexts contributed to the synthesis on equal terms. This strengthened evidence coverage and reduced the risk that language constraints would shape what counted in the final analysis.

Clear limits of AI-generated analysis

The experience showed that AI tools can be highly efficient at extracting and summarizing information, but the resulting analysis can remain superficial. Complex interpretation, contextual judgement, and meaningful insight still depend on human evaluators with institutional and methodological expertise.

Evaluation Framework for Responsible AI Use in Humanitarian Evaluations

EvalCommunity Academy users can adapt the following framework when considering AI support for complex evaluations.

Practical Lessons for M&E and Development Professionals

First, use AI only where it improves evaluation quality and efficiency. UNFPA’s example shows a demand-driven approach: AI was used because the evidence base was large, multilingual, and complex, not because AI adoption was an end in itself.

Second, protect sensitive data before using AI tools. Removing personal identifiers before ingestion is essential, especially in humanitarian evaluations involving staff, partners, and community members.

Third, treat multilingual processing as an inclusion issue. AI can help prevent evidence from non-English contexts from being deprioritized, but evaluators must still verify whether translation and synthesis preserve meaning.

Fourth, document AI use transparently. An annex describing tools, stages of use, and safeguards strengthens credibility and creates a learning resource for other evaluation teams.

Finally, keep humans in the lead. AI can accelerate extraction, summarization, and pattern identification, but evaluators remain responsible for interpretation, triangulation, ethical judgement, and final findings.

FAQ

Conclusion

UNFPA’s humanitarian evaluation shows how AI can support complex evaluation work when it is introduced with clear purpose, ethical safeguards, human oversight, multilingual inclusion, and transparent documentation. The case is not a story of replacing evaluators. It is a story of helping evaluators handle scale while preserving the professional judgement required for credible findings.

For EvalCommunity Academy users, the main lesson is practical: responsible AI in evaluation requires more than choosing a tool. It requires a governance strategy, secure data handling, validation routines, quality control workshops, and a clear commitment to keeping human evaluators in the lead.