Imagine being asked to find useful lessons across hundreds of evaluation reports, with very little time to read them all.

That was the situation faced by a team in the Independent Evaluation Group (IEG) of the World Bank Group.

The team worked with 578 project evaluations to investigate factors associated with project performance in contexts of migration. Rather than relying on a fully manual review, they combined text mining and machine learning with qualitative evaluation methods.

The result was a seven-step process that progressively reduced a very large evidence base to material that evaluators could examine in detail.

The case remains relevant in 2026. The technology has changed considerably since the paper was published, but the underlying problem has not: M&E teams often have more documentary evidence than they can realistically review in depth.

The case in brief

The original paper, published in 2021 as part of the BetterEvaluation Innovation Working Paper series, was written by Stephen Porter, Harsh Anuj, Yingjia Liu and Kristin Strohecker of the Independent Evaluation Group, World Bank Group. :contentReference[oaicite:2]{index=2}

1. Start with the evaluation question

The team did not begin with a machine-learning technique.

They began with an information gap.

The initial task was to identify IEG evaluations related to migration. The team then refined this into a more useful question about the factors associated with project performance in migration contexts.

That distinction matters. Searching for documents is not the same thing as answering an evaluation question.

A large collection of relevant documents can still produce a weak synthesis if the team has not decided what it is trying to learn from them.

2. Find the relevant evidence

The researchers developed search criteria based on keywords, document type and date range. They used a taxonomy of migration-related terms and refined the search when irrelevant results appeared.

That refinement was necessary. Some search terms produced material that had little to do with migration in the development sense. The team therefore adjusted the search strategy rather than accepting the first set of results.

The final evidence base contained 578 project evaluations. The search identified 123 evaluations with migration-related content and produced 1,197 sets of text extracts for further analysis.

This part of the case is easy to overlook, but it is one of its most useful lessons:

Retrieval is part of the method.

If relevant documents are missed, no machine-learning model or language model can recover evidence that was never included in the analysis.

3. Use machine learning to find patterns

The team used Latent Dirichlet Allocation (LDA) to identify topics within the extracted material.

LDA is an unsupervised machine-learning method. Rather than giving the model a fixed set of themes, the researchers allowed it to identify groups of words that tended to occur together.

The model produced 38 candidate topics.

That was not the end of the analysis.

The researchers examined the words and passages associated with the topics, removed material that was not useful, and refined the model. This process resulted in 10 relevant topics across 49 project evaluations.

4. Go back to the source material

Once the candidate topics had been identified, the team moved back toward detailed qualitative review.

They examined whole paragraphs rather than relying only on short extracts. This produced 259 paragraphs for further consideration and eventually narrowed the detailed review to 15 project evaluations.

The researchers also reviewed the relevant passages in the context of the full evaluation reports. This helped prevent isolated words or sentences from being interpreted without the surrounding evidence.

They also consulted colleagues with relevant subject-matter expertise, which contributed additional thematic evaluations to the final synthesis.

This back-and-forth is important. The process was not simply:

documents → algorithm → conclusions

It was closer to:

documents → computational screening → human review → revised analysis → synthesis

5. Turn topics into themes

A machine-generated topic is not automatically an evaluation finding.

The team compared the topics, reviewed the underlying evidence and looked for recurring issues across evaluations. They also considered positive and negative evidence rather than selecting only material that supported an emerging interpretation.

Four themes eventually emerged:

1. Performance improves by targeting specific subgroups, locations and time horizons.
2. Ongoing capacity development is needed to address migration.
3. Granular analysis provides a basis for a coherent response.
4. World Bank support to coordination works well when it is based on comparative advantage.

The authors also make an important qualification: the four themes were not intended to be comprehensive. They reflected what emerged from the evidence included in the synthesis.

6. Know when the evidence is not enough

The researchers tried to identify a broader explanation for project performance.

They could not establish one convincingly.

The evaluations did not provide enough information about how the different themes interacted. Rather than forcing the evidence into a single overarching explanation, the team retained a more limited synthesis of the factors supported by the available material.

7. Finish with something decision-makers can use

The final output was a six-page briefing for World Bank management and Executive Directors.

The purpose was not to showcase the machine-learning model. The purpose was to communicate the lessons that emerged from the evidence in a form that could be used.

That is an important distinction for M&E teams. A sophisticated analytical process has little value if the final product does not help someone understand an issue, make a decision or improve a programme.

What the World Bank team learned

One of the most useful observations in the paper comes from the authors’ reflection on their own process.

They initially expected there would be a clear point at which the data-science work ended and evaluation methods took over.

In practice, the two were intertwined.

Evaluation expertise influenced the search strategy and interpretation of the results, while data-science expertise helped with the large-scale processing of the text.

The result was neither a conventional manual synthesis nor a fully automated analysis. It was a combination of the two.

That is probably the most useful way to think about AI-assisted evidence synthesis today: not as automation of evaluation, but as a different way of organising the work.

How the same problem could be approached in 2026

The original case used keyword searches, text mining and LDA. An evaluation team working today has a much broader set of options.

• Keyword and Boolean search for transparent initial retrieval.
• Semantic search and embeddings for finding conceptually related passages.
• Clustering or topic modelling for exploring recurring patterns.
• LLM-assisted extraction for turning unstructured passages into structured evidence fields.
• AI-assisted qualitative coding for generating or testing candidate codes.
• RAG-based workflows for producing summaries grounded in a defined evidence collection.
• Human review for checking important classifications, interpretations and conclusions.

These approaches can make the workflow faster. They do not remove the need to decide what evidence belongs in the analysis or what the evidence means.

Recent examples show how the field is moving

World Bank — 2023

In 2023, the World Bank Independent Evaluation Group published Machine Learning in Evaluative Synthesis: Lessons from Private Sector Evaluation in the World Bank Group. The resource examines how machine learning can support the extraction, classification and synthesis of large volumes of evaluation text. :contentReference[oaicite:3]{index=3}

The connection with the 2021 case is useful: the organisation continued exploring computational methods for evaluative synthesis rather than treating the earlier experiment as a one-off.

UNICEF — 631 evaluation reports

A more recent UNICEF case study describes a semi-automated analysis of 631 evaluation reports aligned with 64 outcomes from UNICEF’s 2022–2025 Strategic Plan. The workflow used machine-readable text, AI-assisted extraction and filtering. UNICEF reports a 92% reduction in text volume requiring detailed review. :contentReference[oaicite:4]{index=4}

This is a particularly useful comparison for M&E professionals because the problem is very similar: a large evaluation evidence base that would be difficult to review efficiently by hand.

LLMs and systematic review — 2025

A 2025 case study compared a conventional systematic-review workflow with a ChatGPT-4-assisted process. The authors found meaningful time savings for several operational tasks, but reported more difficulty in areas requiring interpretation, risk-of-bias assessment and final drafting. :contentReference[oaicite:5]{index=5}

That is consistent with the lesson from the World Bank case: speeding up the mechanical parts of synthesis is one thing; deciding what the evidence actually means is another.

LLMs and evidence triangulation — 2025

A 2025 Nature Communications study examined the use of LLMs to extract and synthesise evidence across different study designs. Its two-step extraction approach performed better than a one-step approach for several tasks, including identifying the direction of effects and statistical significance. The study also reported limitations involving study-design classification, terminology and relation extraction. :contentReference[oaicite:6]{index=6}

The example is relevant because it moves the conversation beyond document search toward a more demanding question: can AI help organise evidence from different studies so that evaluators can compare and triangulate it?

From the 2021 case to today’s workflow

A practical workflow for an M&E team

Suppose your organisation has 2,000 evaluation reports, learning documents and programme reviews. You are asked to identify factors that appear repeatedly in stronger-performing programmes.

A workflow inspired by the World Bank case could be:

1. Define the information gap. What exactly do you need to learn?
2. Set the boundaries. Which documents, programmes, dates and evaluation types belong in the analysis?
3. Retrieve the evidence. Combine transparent search methods with semantic retrieval where useful.
4. Extract candidate passages. Identify the parts of the documents that may answer the question.
5. Explore patterns. Use topic modelling, clustering or an LLM to suggest recurring issues.
6. Check the source material. Read enough of the surrounding context to verify the pattern.
7. Develop themes. Compare evidence across documents rather than relying on individual examples.
8. Look for disconfirming evidence. Ask what does not fit the emerging interpretation.
9. Synthesise carefully. Separate what the evidence shows from what it merely suggests.
10. Keep an audit trail. Record the sources, searches, AI methods, prompts, coding decisions and human checks.
11. Produce the decision product. Make the result useful to the people who need to act on it.

Where should AI stop?

What can go wrong?

The World Bank case is valuable partly because the process was not treated as automatic.

• Search terms can miss relevant evidence.
• Search terms can retrieve irrelevant material.
• Machine-generated topics can be difficult to interpret.
• Short extracts can lose context.
• Different reports can use different language for similar ideas.
• The quality of a synthesis is constrained by the quality of the underlying evaluations.
• AI-generated interpretations can look convincing even when the supporting evidence is weak or incomplete.

For that reason, an AI-assisted synthesis should leave a clear path from the final finding back to the evidence that supports it.

A simple quality check

Before accepting an AI-assisted synthesis, ask:

Key lessons for M&E professionals

1. Start with the information need, not the AI tool.
2. Treat document retrieval as part of the methodology.
3. Use machine learning to surface patterns, not to decide what those patterns mean.
4. Return to the original evidence before developing conclusions.
5. Look for evidence that challenges the emerging story.
6. Do not force a synthesis beyond what the evidence can support.
7. Keep human judgement visible.
8. Document how AI was used so the process can be reviewed and repeated.

Questions for your team

1. What is the largest evidence-synthesis task your team currently performs manually?
2. How many reports or documents are normally involved?
3. Which part of the process consumes the most time?
4. Could AI help with retrieval or extraction without taking over interpretation?
5. How would you validate AI-generated topics or codes?
6. How would you preserve traceability from a finding back to the original evidence?

Try the approach yourself

Take a small collection of evaluation reports and reproduce a simplified version of the World Bank workflow.

1. Define one information gap.
2. Set inclusion and exclusion criteria.
3. Search and retrieve relevant passages.
4. Ask an AI tool to suggest candidate topics.
5. Review the source passages yourself.
6. Refine the topics.
7. Develop two or three themes.
8. Look specifically for contradictory evidence.
9. Write a short synthesis and document how you reached it.

You do not need hundreds of reports to learn the method. A small, well-defined dataset is enough to practise the workflow.

Main source

Porter, S., Anuj, H., Liu, Y., & Strohecker, K. (2021). Machine learning and meta-ethnography: Seven steps to synthesising 578 evaluations into four themes. BetterEvaluation Innovation Working Paper Series, Independent Evaluation Group, World Bank Group.

Read the original case study

Selected contemporary references

• World Bank IEG (2023). Machine Learning in Evaluative Synthesis: Lessons from Private Sector Evaluation in the World Bank Group.
  Read the resource
• UNICEF Evaluation Office. Accelerating evaluation synthesis analysis: A case study. Analysis of 631 UNICEF evaluation reports.
  Read the case study
• Tomassini, F. et al. (2025). Leveraging ChatGPT-4 for Evidence Synthesis: A Case Study on the Use of a Large Language Model in a Systematic Review.
  PubMed
• Shi, X. et al. (2025). Evidence triangulator: using large language models to extract and synthesize causal evidence across study designs. Nature Communications, 16, 7355.
  Read the article