Introduction

The i-eval AI Assistant is an AI-powered chatbot launched by the International Labour Organization’s Evaluation Office to help staff access, analyse, and apply evaluative knowledge more quickly. Embedded in i-eval Discovery, the ILO’s public evaluation dashboard, the assistant allows users to ask natural-language questions and receive clear, source-cited responses drawn from ILO evaluation evidence.

The tool was developed through a co-creation process between EVAL and INFOTEC, building on a rich repository of more than 1,700 evaluation reports, 3,000 lessons learned, 1,600 good practices, and 3,000 recommendations. It uses Retrieval-Augmented Generation (RAG) to produce tailored, source-cited summaries in under 20 seconds, drastically reducing the time needed to synthesize evidence across multiple reports.

For evaluators, this case is important because it shows how generative AI can be used not only for efficiency, but to improve the practical use of institutional evaluation knowledge. The central lesson: AI strengthens evidence use when it is embedded in a quality-assured evidence system, linked to original sources, and designed around real user needs.

Case Background

Evaluation offices often hold large volumes of valuable knowledge, but staff across an organization may struggle to find, interpret, and apply that knowledge at the right moment. Reports can be long, scattered across databases, or difficult to synthesize quickly. As a result, lessons learned may not fully influence project design, implementation, policy, or communication.

The ILO Evaluation Office addressed this challenge through the i-eval AI Assistant. The assistant builds on the existing i-eval Discovery platform, which already organizes a large body of evaluation evidence. By adding an AI assistant interface, ILO staff can ask targeted questions and receive concise, source-cited summaries that point back to original evaluation reports.

The initiative also fits into EVAL’s broader 2023–25 Evaluation Strategy, which emphasizes AI-driven digital solutions and improved use of evaluative knowledge for accountability, learning, and programming. Together with i-eval Discovery and the Automated Management Response System (AMRS), the assistant contributes to a wider digital evaluation ecosystem within the ILO.

The Evaluation Problem

The main problem was not the absence of evaluation evidence. The ILO already had a rich evidence base. The challenge was making that evidence usable for staff who needed timely insights for practical decisions. Without rapid synthesis, users might spend hours or days searching through reports, extracting relevant findings, and comparing lessons across projects.

This is a common challenge in evaluation knowledge management: organizations generate evidence, but evidence use depends on whether users can access it in a form that matches their decision needs, language, time constraints, and work context. The ILO Evaluation Office recognized that a well-structured repository does not automatically translate into effective use—evaluation knowledge requires synthesis.

The AI-Enabled Solution

The i-eval AI Assistant uses generative AI and Retrieval Augmented Generation (RAG) to search across evaluation evidence and generate tailored summaries. Because outputs are source-cited and linked to original reports, users can verify the evidence and explore the underlying documents when needed.

The assistant is designed to move staff beyond manually reading hundreds of pages toward receiving on-demand, needs-based, quality-oriented evaluation insights. In less than 20 seconds, users can receive coherent content that can inform project design, implementation, strategies, policies, decision-making, and communications.

A critical design feature is the “5-chunk rule”—limiting each report to a maximum of five content chunks. This was introduced to force the system to synthesize across multiple sources, avoiding the “dominant document” problem where a single report disproportionately shapes the answer. This reflects the evaluation principle that diversity of evidence is more valuable than depth from a single source.

Who Should Use It?

The assistant is intended for ILO staff who need to apply evaluation evidence in their work. The ILO infographic identifies four main user groups:

• Staff designing or improving new and existing projects.
• Project implementers who want to learn from similar projects and use evaluative evidence during implementation.
• Staff preparing briefs, speeches, or communication products on ILO field work and organizational learning.
• Policy and thematic teams that want to complement research with evaluation evidence.

Benefits for Evaluators and Decision-Makers

1. Faster access to evaluation insights

The assistant reduces the time needed to locate and synthesize relevant findings from large reports. This supports faster preparation of project documents, briefs, recommendations, and evidence summaries.

2. Better use of institutional knowledge

Evaluation evidence becomes more accessible to non-specialist users. Staff who may not have time to review full reports can still engage with lessons learned and good practices in a structured way.

3. Stronger project design and implementation

By allowing staff to ask questions about what has worked, what challenges appeared, and what lessons emerged in similar contexts, the assistant can help teams design stronger interventions and adapt implementation more responsively.

4. Improved accountability and learning

Source-cited outputs help preserve traceability. Users can see where information comes from and return to original reports for deeper analysis, which is essential for responsible AI use in evaluation settings.

5. Practical feedback for improvement

During rollout, users were asked to explore prompts, use the information to inform their work, and provide feedback. This creates a learning loop between users, developers, and evaluation knowledge managers.

Evaluation Framework for AI Assistants in Evaluation Offices

EvalCommunity Academy users can adapt the following framework when assessing AI assistants for evaluation knowledge management.

Practical Lessons for M&E and Development Professionals

1. Start with a trusted evidence base. The value of an AI assistant depends heavily on the quality, structure, and credibility of the documents it retrieves from. Evaluation offices should invest in clean repositories, metadata, quality assurance, and source traceability before expecting AI to produce reliable insights.

2. Design for specific user workflows. The i-eval AI Assistant is relevant because it supports real tasks: project design, implementation, policy work, speeches, briefs, and strategic communication. AI tools are more useful when they are integrated into existing decision points.

3. Keep citations visible. Source links are not an extra feature; they are central to trust. Evaluation users need to know where findings come from, especially when AI is synthesizing complex reports.

4. Use pilots to learn. The first rollout invited a defined group of staff to test the assistant and provide feedback before wider deployment. This staged approach helps teams identify user needs, prompt patterns, quality issues, and training requirements.

5. Embed diversity of evidence deliberately. The “5-chunk rule” was a critical design choice to force synthesis across multiple sources. Without such guardrails, AI systems tend to amplify a single document.

6. Position AI as a learning tool, not a replacement for evaluation expertise. The assistant can speed up discovery and synthesis, but users still need professional judgment to interpret evidence, understand context, and apply findings responsibly.

Frequently Asked Questions

Conclusion

The i-eval AI Assistant is a useful case study in AI-enabled evaluation knowledge management. It shows how a large organization can use AI to make evaluation evidence faster to access, easier to apply, and more visible in everyday decision-making.

For EvalCommunity Academy users, the main lesson is practical: successful AI in evaluation begins with a real evidence-use problem, a trusted knowledge base, clear citations, user-centered design, and careful governance. The goal is not simply to automate summaries. The goal is to strengthen evidence-informed action.

The ILO experience also demonstrates that meaningful innovation in evaluation depends not only on the size or budget of an Evaluation Office, but on sustained commitment to knowledge, clarity of purpose, systems thinking, and the ability to translate evaluation principles into practical, accessible, and usable tools.