The challenge

Generative AI was rapidly changing knowledge work, including evaluation. The UNFPA Independent Evaluation Office (IEO) recognized that GenAI could potentially improve the efficiency, effectiveness and timeliness of evaluations, while also creating risks related to accuracy, bias, privacy, transparency, accountability and over-reliance on AI.

The response was not simply to select a collection of AI tools. The IEO developed a strategy for determining where, when and how GenAI could be integrated into the evaluation function.

The strategy was aligned with UNFPA’s existing priorities around evidence-informed decision-making, learning, adaptation and accountability.

A needs-driven approach

A central feature of the strategy was its demand-driven approach. The IEO conducted a needs assessment involving IEO staff, regional M&E advisers and country M&E staff or focal points, with additional input from policy, technical, innovation, legal, ethics and information technology functions.

The purpose was to identify needs and bottlenecks across the evaluation lifecycle before considering potential GenAI solutions.

The strategy explicitly states that when a non-GenAI solution can meet an evaluation need better than a GenAI solution, the non-GenAI solution should be pursued.

The needs assessment identified 30 unique user needs that the IEO considered possible candidates for GenAI-supported optimization.

Six principles for GenAI-powered evaluation

The strategy sets out six principles to guide the use of GenAI in evaluation:

1. Pursue a demand-driven approach.
   GenAI should address identified evaluation needs rather than being adopted simply because the technology is available.
2. Promote diversification and innovation.
   The evaluation function should continue exploring new GenAI approaches that can support data collection and analysis and reduce technical barriers.
3. Uphold quality and credibility.
   GenAI-assisted analysis should remain rigorous and transparent and should contribute to credible findings and recommendations.
4. Maximize the use and utility of evaluations.
   GenAI can support the accessibility, timeliness and relevance of evaluation outputs for decision-making and learning.
5. Adhere to a human rights-based approach.
   GenAI use should reflect principles including transparency, fairness, participation, privacy, accuracy and a human-centered approach.
6. Foster evaluation capacity development.
   Evaluation stakeholders should develop the capacity to use, interpret and critique GenAI-powered evaluation, particularly in the Global South.

Where could GenAI support evaluation?

The strategy identified potential applications across several stages of the evaluation lifecycle.

Evidence synthesis and information management

• Evidence repositories and information synthesis
• Search and retrieval of conclusions, recommendations and citations
• Summarization of documents for evaluation and reporting purposes

Data analysis

• Qualitative categorization and coding
• Pattern recognition
• Sentiment analysis
• Automatic transcription
• Quantitative analysis of extensive secondary datasets to identify patterns, correlations and trends

Reporting

• Initial report designs and drafts
• Background and context
• Findings and recommendations
• Editing and readability
• Visualizations

Dissemination and use

• Translation
• Audience segmentation
• Slide-deck generation
• Accessible communication materials

Evaluation quality assurance

• Making quality assurance processes faster and more iterative
• Improving adherence to existing evaluation policy
• Supporting assessment of evaluation designs
• Supporting assessment of evaluative reasoning
• Supporting triangulation

What risks did UNFPA identify?

The strategy examines potential harms at several levels rather than treating AI risk as a single issue.

How did the strategy propose mitigating these risks?

• Align GenAI tools with relevant information-security policies and requirements.
• Use strong security measures, access controls and appropriate compliance processes.
• Apply human judgment to assess the integrity of GenAI outputs.
• Include GenAI use in evaluation quality assurance.
• Disclose GenAI use in evaluation materials.
• Document how GenAI tools are used.
• Maintain human involvement at key stages.
• Define escalation processes for unclear cases.
• Monitor potential harms and address identified issues.
• Consider environmental impacts and model documentation.

Human oversight and accountability

Human responsibility is a central element of the strategy. The IEO’s risk-mitigation framework states that human judgment is paramount for evaluation quality assurance and assessment.

The strategy also establishes AI-use requirements for consultants. AI tools cannot be used under the relevant IEO contract without prior written agreement. Where AI use is approved, consultants must disclose its use, use vetted tools and verify the accuracy of AI-generated results.

The strategy also places responsibility on companies conducting UNFPA evaluation work to establish verification processes, monitor AI analyses and address identified biases, errors or shortcomings.

The implementation roadmap

UNFPA proposed a phased approach to introducing GenAI into evaluation.

1. Feasibility and needs assessment

Identify evaluation-process user needs and map existing GenAI solutions to prioritized use cases.

2. GenAI capacity building

Develop training on GenAI applications in evaluation, evaluation principles and responsible and ethical use. The roadmap also includes consultation, troubleshooting and networking among GenAI and evaluation professionals.

3. Piloting

Pilot GenAI-powered evaluation methods, including secondary-data analysis and synthesis for centralized evaluation and data extraction and analysis for a meta-synthesis exercise.

4. Scaling and sustainability

Take stock of lessons from centralized evaluation, conduct internal quality assurance, progressively roll out GenAI solutions to decentralized evaluations and continue developing the tools, clauses and disclaimers needed for responsible use.

The strategy also calls for an iterative and adaptive approach because user needs, available tools and the GenAI environment continue to change.

How did UNFPA plan to measure implementation?

An important feature of the strategy is that UNFPA proposed an M&E framework for monitoring implementation of the GenAI strategy itself.

The framework distinguishes between the responsible integration of GenAI into evaluation processes and evidence that those evaluations demonstrate optimized processes and products.

Important: These are targets defined in the strategy’s M&E framework. They should not be interpreted as reported achievement results.

What can M&E professionals learn from the UNFPA strategy?

1. Start with evaluation needs.
   Identify the problem or bottleneck before selecting an AI solution.
2. Do not treat efficiency as the only outcome.
   The strategy explicitly recognizes the need to balance timeliness with accuracy, propriety, utility and credibility.
3. Build risk assessment into AI adoption.
   Consider privacy, bias, transparency, accountability, human oversight and environmental impacts.
4. Keep human judgment central.
   AI use does not remove responsibility for evaluation quality.
5. Invest in evaluator capacity.
   Evaluators need the ability to use, interpret and critically assess GenAI outputs.
6. Measure whether AI actually improves evaluation.
   The UNFPA strategy includes indicators for both responsible adoption and optimization of evaluation processes and products.
7. Use an adaptive approach.
   The strategy recognizes that GenAI tools, risks and user needs will continue to change.

Frequently asked questions

Related EvalCommunity resources

• AI Tool Evaluation Protocol for M&E
  — a practical approach to testing and documenting AI tools for evaluation tasks.
• AI Evidence Synthesis for M&E Literature Reviews
  — guidance on using AI in evidence synthesis.
• How to Use AI Safely with Sensitive M&E Data
  — guidance on responsible use of AI with sensitive evaluation information.
• Principles for AI Use in Monitoring & Evaluation
  — principles for responsible AI use in evaluation.
• AI in Monitoring & Evaluation Certificate
  — practical training on applying AI across the M&E lifecycle.