UNICEF – AI for Evaluation Synthesis of Reports: A Case Study
UNICEF – AI for Evaluation Synthesis of Reports: A Case Study
What is AI-powered evaluation synthesis at UNICEF?
AI-powered evaluation synthesis refers to the use of computational methods—including text mining, neural networks, and automated filtering—to analyze large volumes of evaluation reports. In this case, UNICEF and Newcastle University researchers developed a semi-automated pipeline to extract, classify, and synthesize evidence from 631 evaluation reports, making a previously infeasible manual review possible while maintaining quality and reproducibility.
Why did UNICEF need AI for evaluation synthesis?
UNICEF publishes comprehensive evaluation reports to disseminate outcomes of interventions and accelerate results for children. Between 2018 and 2023, UNICEF published 875 such reports, often exceeding 150 pages each. Manual synthesis of hundreds of reports aligned with 64 strategic outcomes would have been too time-consuming and resource-intensive. The volume of unstructured text—combined with multiple languages (English, French, Spanish, Portuguese)—created an analytical challenge that required automation to achieve a broad, timely synthesis.
Which AI tools and methods were used?
📄 pypdf
Bulk PDF-to-text conversion of 631 reports
🧠 SPECTER
Transformer-based neural network for sentence classification (trained on 5 classes: Lessons, Recommendations, Enablers, Barriers, Background)
🔍 SWIFT-Review
Text-mining workbench for Boolean and adjacency searching, filtering, and highlighting relevant text
🌐 Google Translate API
Automatic translation of 162 non-English reports (French, Spanish, Portuguese)
What was the step-by-step AI workflow?
1. PDF-to-text conversion
Python script using pypdf converted 631 PDF reports into plain text, recording any processing errors.
2. Section identification
Rule-based regex extracted executive summaries (88% success rate). Lessons/recommendations sections were too variable for reliable extraction.
3. Sentence mining with neural network
A SPECTER transformer model was trained on manually labelled sentences to classify 5 categories. Applied to full texts, it retained top 30 sentences for Lessons and Recommendations per report.
4. Translation
162 non-English reports were automatically translated to English using Google Translate API.
5. SWIFT-Review filtering
Reduced text (executive summaries + mined sentences) was imported. Complex adjacency searches (e.g., terms within 5 words) filtered reports for each of 64 outcomes.
6. Human review
Researchers skimmed highlighted text from filtered reports (10-20 minutes each), consulting original PDFs when needed, and extracted final evidence.
How well did the AI system perform?
Text reduction and efficiency
| Total text volume reduction | 92% – from full reports to executive summaries + 143 mined sentences (avg. per report) |
| Reports screened per outcome | Average 21 reports (median 0.6 precision) |
| Time per report (pilot) | 10-20 minutes with AI highlighting, up to 30 minutes if original PDF consulted |
Sentence classifier performance (at 0.2 probability threshold)
| Lessons Learned | Recall: 0.93 | Precision: 0.29 | F1: 0.44 |
| Recommendations | Recall: 0.93 | Precision: 0.53 | F1: 0.68 |
| Enablers | Recall: 0.00 – insufficient training data, ambiguous concept |
| Barriers | Recall: 0.00 – insufficient training data, ambiguous concept |
Note: High recall (93%) for Lessons and Recommendations meant the model captured almost all relevant sentences, though precision was moderate—some irrelevant text was included, which human reviewers filtered out.
Filtering precision across outcomes
| Overall precision (all outcomes) | 0.52 (386 true positives / 730 screened) |
| Median precision (per outcome) | 0.6 |
| Range | 0.08 to 1.0 (depending on outcome specificity) |
What limitations were identified?
📉 Incomplete section extraction
22% of executive summaries could not be automatically identified due to PDF quality or structural variations; required manual addition.
🧩 Ambiguous concept detection
"Enablers" and "Barriers" were too poorly defined and overlapped with Lessons/Recommendations; the model could not learn them from limited data.
🌍 Translation edge cases
Some documents were missed by translated searches due to American/British spelling variations or ambiguous words (e.g., French "cours" meaning lesson or during).
⚖️ Sensitivity-precision trade-off
Broad, sensitive searches increased workload; narrow searches risked missing relevant reports. Balance varied per outcome.
How did the team mitigate these limitations?
- Hybrid approach: Combined automatically extracted executive summaries with mined sentences to ensure baseline coverage.
- Human-in-the-loop: All AI outputs were reviewed by researchers; classifier used only for Lessons and Recommendations (the two well-performing classes).
- Iterative search refinement: Searches were adjusted after reviewing initial results—decreasing adjacency distance, removing problematic terms.
- Error analysis on translations: Compared original-language searches with translated versions; decided to use only translated English text after confirming minimal unique hits were missed.
- Open-source transparency: All code and trained models were published to enable replication and adaptation.
How does this compare to other UN AI tools (e.g., UNDP AIDA)?
| Feature | UNDP AIDA | UNICEF method |
| Scope | Only UNDP evaluations | Any evaluation dataset |
| Open source | No | Mix: free third-party tool + open-source scripts |
| Search | Keywords | Boolean + adjacency + wildcards |
| Granularity | Paragraph-level | Sections, sentences, tagged keywords |
| Synthesis | Fully AI-automated (LLM summaries) | Semi-automated: filtering + highlighting + human review |
| Translation | Planned | Implemented via Google Translate API |
Note: The UNICEF team emphasized that fully automated LLM summaries (like AIDA) raise validation and hallucination concerns. Their semi-automated approach prioritized human oversight and traceability.
What are the key lessons for evaluators?
✅ AI enables large-scale synthesis
Processing 631 reports across 64 outcomes became feasible, reducing text by 92%.
⚠️ Not all concepts work equally
Well-defined, distinct categories (Lessons, Recommendations) performed well; ambiguous ones (Enablers, Barriers) failed.
🧠 Hybrid > fully automated
Combining AI extraction with human review maintained rigour and avoided hallucination risks of pure LLM synthesis.
🔍 Search strategy matters
Adjacency searches in SWIFT-Review improved precision over simple Boolean; information specialist input is valuable.
🌐 Translation works but needs checks
Automatic translation enabled inclusion of non-English reports, but error analysis is essential.
📂 Open-source enables replication
Published code and models allow other organizations to adapt the methods to their own evaluation datasets.
Frequently asked questions
What is AI-powered evaluation synthesis at UNICEF?
What AI tools did UNICEF use?
How accurate was the AI sentence classification?
Can this method be replicated by other organizations?
What were the main limitations of this approach?
Main reference & original source
📘 This case study is based on the working paper "Accelerating evaluation synthesis analysis: A case study on data extraction from 631 UNICEF evaluation reports" by Schmidt, Addis, Mattellone, et al., published by the UNICEF Evaluation Office in 2024.
Sources: Full PDF · UNICEF summary · GitHub repository
Resources for further learning
- UNICEF Evaluation Office
- SWIFT-Review (free tool)
- GitHub: Code & methods
- OECD DAC Evaluation
- EvalCommunity
Advance your skills in AI-powered evaluation
Learn how to apply text mining, NLP, and semi-automated synthesis in your own evaluation work. Access courses, templates, and expert guidance.
