1. The Case at a Glance

The Independent Evaluation Group (IEG) faced a familiar evaluation challenge at an unusually large scale:
how to classify evidence across thousands of pages of project documentation without sacrificing methodological rigor.

The evaluation examined more than 25 years of World Bank Group support for pandemic preparedness.
The team needed to identify and classify pandemic preparedness interventions across more than
1,300 lending operations spanning sectors including health, agriculture, environment,
water and private sector development.

Manual coding was considered too resource-intensive for the scale of the portfolio.
Supervised machine learning was another possibility, but it would have required a substantial
human-coded training dataset.

2. The Evaluation Question

The evaluation examined more than 25 years of World Bank Group support for pandemic preparedness.
The team wanted to understand:

• where investments were made;
• which preparedness capacities received the greatest attention;
• where important gaps remained; and
• whether support aligned with countries’ preparedness needs.

Answering these questions required identifying and classifying pandemic preparedness interventions
across a large and diverse portfolio and reviewing thousands of pages of project documentation.

3. Why AI?

IEG considered several approaches to the classification problem.

• Manual coding: familiar and potentially reliable, but requiring an enormous
  investment of time and resources at this scale.
• Supervised machine learning: an approach IEG had used previously,
  but one requiring a large human-coded training dataset.
• Generative AI: an opportunity to classify large volumes of text while
  applying a structured validation process.

The team chose to experiment with generative AI, while adapting concepts from supervised
machine learning such as validation, benchmarking, iterative refinement and quality assurance.

4. The IEG AI Roadmap

IEG called its workflow the AI Roadmap. It was designed as an iterative process
rather than a one-way automated pipeline. The team could return to earlier decisions,
refine the taxonomy, improve prompts, reassess performance or strengthen quality assurance.

5. Stage 1 — Building the Taxonomy

Before asking an AI model to classify anything, the evaluation team had to determine what
constituted a pandemic preparedness intervention.

IEG used subject-matter expertise, literature reviews and iterative discussions within the
evaluation team to develop a detailed taxonomy.

Each category contained a label and a precise operational definition.
Those definitions subsequently became part of the instructions guiding the AI’s classification decisions.

6. Stage 2 — Designing the Classification Prompt

Prompt design required several iterations and back-and-forth exchanges between analysts and evaluators.

One important decision was determining which parts of project documents best captured the intervention
and how much contextual information was enough without introducing unnecessary noise.

The final setup asked the model to classify each text-segment/intervention pair using a simple
Yes/No response together with a brief explanation.

The explanation provided an additional way for the team to examine whether the model was identifying
the intended signals.

7. Stage 3 — Benchmarking AI Against Human Judgment

Designing the prompt was only the beginning. IEG then tested the model against a human-coded
validation sample.

The team used four classification metrics:

Because the data were unbalanced, IEG considered accuracy alone insufficient.
The team prioritized recall because missing a relevant intervention could
systematically understate preparedness activities.

False positives, by contrast, could be identified and corrected through human review.

8. Stage 4 — Refining the Model

Benchmarking showed where the workflow needed improvement.
Depending on the results, the team could:

• clarify taxonomy definitions;
• adjust category labels;
• modify prompt instructions;
• add illustrative examples; or
• change the amount of input text provided to the model.

After each refinement, the classification was rerun and compared with the human-coded benchmark.
The process continued until further changes no longer improved performance.

9. Reported Performance

These figures are the performance results reported by IEG for this particular classification exercise.
They should not be interpreted as a general benchmark for AI in evaluation.

10. Stage 5 — Targeted Quality Assurance

Because the model was intentionally optimized toward recall, IEG expected false positives.
Rather than manually reviewing every prediction, the team focused quality assurance on classifications
most likely to contain errors.

Evaluator expertise was used to identify combinations that did not make operational sense.
For example, a health project classified as supporting veterinary workforce training,
or an agriculture project appearing to include investments in medical countermeasure policies,
could be flagged for review.

The team also observed that longer text passages tended to generate more overprediction,
so these received additional scrutiny.

According to IEG, roughly three-quarters of the suspicious predictions were indeed
incorrect and were successfully corrected through this final review.

11. The AI Evidence Quality Chain

The model is only one component of the evidence-quality process.
The credibility of AI-assisted classification depends on the workflow surrounding the model.

12. Why This Matters for M&E

Evaluators are increasingly using AI for document review, coding, evidence synthesis,
qualitative analysis and extraction.

The IEG case demonstrates how AI-assisted analysis can be connected to established principles
of validation and quality assurance.

• Define the construct before automating it.
• Create explicit classification rules.
• Benchmark AI against human-coded evidence.
• Select metrics according to the consequences of errors.
• Iteratively improve the workflow.
• Target human review where it adds the most value.
• Document the process so results remain auditable.

13. Accuracy Is Not Enough

One of the most useful lessons is that the choice of AI performance metric should reflect
the consequences of different types of errors.

In this case, IEG considered missed interventions particularly consequential and therefore
prioritized recall, while using human review to address false positives.

14. EvalCommunity Perspective: AI Classification QA

The following is an EvalCommunity interpretation inspired by the IEG case.
It is not a framework reported by IEG.

15. Suggested M&E Indicators

Building on the IEG case, an M&E team could consider monitoring:

• AI-human agreement on a validated sample
• False-negative rate
• False-positive rate
• Precision, recall and F1-score
• Performance by sector or document type
• Performance by language where relevant
• Percentage of classifications requiring human correction
• Percentage of targeted QA cases revealing an AI error
• Time required per 1,000 evidence units
• Traceability of classifications to source text

These are proposed EvalCommunity indicators, not indicators reported by IEG in the source case.

16. A Practical Human-in-the-Loop Workflow

1. Define the evaluation construct.
2. Develop the taxonomy and operational definitions.
3. Create a human-coded validation sample.
4. Design and test the AI prompt.
5. Benchmark AI against human judgment.
6. Refine the taxonomy, prompt or context.
7. QA classifications most likely to contain errors.
8. Document the methodology and results.

17. What This Means for the Evaluator

AI can take on more of the repetitive analytical workload, but evaluator expertise remains important
for defining constructs, designing validation strategies, interpreting unexpected results and determining
whether evidence is sufficient to support an evaluative conclusion.

In the IEG workflow, human expertise was embedded throughout the process rather than added only at the end.

18. What This Case Does Not Demonstrate

• It does not establish a general accuracy rate for generative AI in evaluation.
• It does not show that human coding is no longer necessary.
• It does not establish that the same workflow will perform equally well on other evaluations.
• It does not demonstrate that AI classification automatically produces valid evaluation findings.

The reported results relate to IEG’s specific classification task and validation process.

19. EvalCommunity Interpretation

The IEG case suggests that the key question is not simply whether AI can classify evaluation evidence.
The more important question is whether the process surrounding the AI is rigorous enough
for the intended evaluation purpose.

From an EvalCommunity perspective, four principles stand out:

• Define before automating.
• Benchmark before trusting.
• Measure the errors that matter.
• Keep human judgment accountable for the final evidence.

20. Questions for the Author

The published case also raises questions that could be valuable for the wider evaluation community.

1. How large was the human-coded validation sample, and how was it selected?
2. How many human coders were involved, and how was consistency between human coders established?
3. How well did the final model perform on documents different from those used during refinement?
4. Were false negatives and false positives distributed differently across sectors or intervention types?
5. How sensitive were results to changes in the prompt, model or amount of contextual text?
6. How much time did targeted quality assurance require compared with reviewing all classifications manually?
7. Could the approach eventually be extended to other stages of the evaluation evidence chain?

These are questions from EvalCommunity for further discussion with the author and evaluation community;
they are not claims about IEG beyond what is reported in the source.

21. Confidentiality

Use approved AI tools and do not enter confidential, personal or sensitive evaluation data
unless appropriate safeguards and authorization are in place.

22. Evidence Boundary

Original source:
Mercedes Vellez, Independent Evaluation Group, World Bank Group,
“Can We Trust AI to Classify Evaluation Evidence? An AI Roadmap with Human-in-the-Loop at Every Turn,”
August 24, 2026.

Read the original IEG article