UNICEF – Operational Framework for Machine Learning in Evaluation – Case Study
UNICEF – Operational Framework for Machine Learning in Evaluation
What is UNICEF's Operational Framework for Machine Learning in Evaluation?
UNICEF's operational framework is a practical guide for evaluation managers on applying machine learning (ML) methods in evaluation studies. Developed through literature review and three pilot projects within UNICEF's Evaluation Office, the framework categorizes applications by data type—text data, traditional quantitative data, and big data—and maps them to evaluation questions aligned with OECD-DAC criteria. It emphasizes that ML can enhance efficiency by automating analytical tasks, but requires foundational understanding, replicable approaches, and attention to ethics and data privacy.
Why did UNICEF develop this framework?
UNICEF country offices generate vast amounts of textual monitoring and reporting data—evaluation reports, annual progress statements, and country programme documents (CPDs)—that are resource-intensive to analyze manually. Multi-country evaluations and assessments in fragile settings often face constraints where traditional approaches (surveys, interviews) are inadequate or prohibitively expensive. The framework responds to the need for cost-effective, robust analytical methods that can leverage existing data assets, reduce time spent on repetitive tasks, and strengthen evidence derived from traditional evaluation methods.
What ML methods and data types does the framework cover?
📄 Text Data
Evaluation reports, CPDs, monitoring narratives. Methods: Supervised NLP (classification), unsupervised NLP (topic modeling), keyword search, summarization (BART, LLMs)
📊 Traditional Quantitative
Surveys, administrative data, census. Methods: Counterfactual identification (matching), heterogeneous treatment effects (causal trees, X-learners), small area estimation
🛰️ Big/Frontier Data
Satellite imagery, social media, mobile phone data, financial transactions. Methods: Poverty prediction, climate shock estimation, slum mapping, sentiment analysis
What were the three pilot projects?
Pilot 1: Scoping and synthesis of evaluation reports (Access to Justice)
📌 Objective
Identify UNICEF evaluation reports on 'child access to justice' and synthesize findings.
⚙️ Method
Supervised NLP model trained on 527 manually labelled reports (98% accuracy); BART large-language model for summarization.
📈 Result
59 reports identified; cost-effective scoping by one manager and one data scientist. Summaries had 6-7/59 missing information—decided not to scale summarizer without higher accuracy.
Pilot 2: Activity mapping (Violence Against Girls and Boys, VAGBaW)
📌 Objective
Identify which country office progress statements describe VAGBaW activities and classify activity types.
⚙️ Method
Supervised model (1,589 labelled statements) to identify VAGBaW statements; unsupervised topic modeling to group activities.
📈 Result
Mapped VAGBaW activities across countries; unsupervised model revealed coherent topics (e.g., capacity development, training) without pre-labelled data.
Pilot 3: Tracking country priorities (Child Marriage)
📌 Objective
Assess whether UNICEF-UNFPA Global Programme influenced national prioritization of child marriage elimination.
⚙️ Method
Keyword search (with stemming) on CPDs from 72 countries, 2010-2022; compared with intelligent semantic search.
📈 Result
Basic keyword search returned more relevant results than advanced semantic search; replicated across other topics (out-of-school children, health).
What were UNICEF's key learnings from the pilots?
| Managerial ML knowledge | Managers need foundational understanding of ML to collaborate with technical experts and design effective solutions. |
| Replicability over complexity | Basic methods (keyword search) can be highly effective and replicable across evaluations—choose the most resource-saving option. |
| Capitalize on processed data | Once documents are cleaned/processed, reuse them across different analyses and thematic areas to maximize ROI. |
| Leverage existing labelled data | Prior qualitative coding work can train supervised models; explore open-source labelled datasets (e.g., OSDG for SDG classification). |
What are the applications by data type (the framework)?
Text Data Applications
NLP methods enable scoping and synthesis of prior evaluative evidence, mapping programme activities and priorities across country offices. Examples include UNICEF's pilot projects and World Bank's use of NLP to classify project reports (Franzen et al., 2022). UNDP's AIDA platform similarly uses NLP for evaluation synthesis.
Traditional Quantitative Data Applications
ML strengthens impact evaluation through:
- Counterfactual identification: ML matching techniques (e.g., Linden & Yarnold, 2016) select balanced comparison groups when control groups are structurally diverse—demonstrated in Indonesia health insurance evaluation (Kreif & DiazOrdaz, 2019).
- Heterogeneous treatment effects: Causal trees and X-learners identify which subgroups benefit most from interventions without pre-specifying groups (Athey et al., 2019; McKenzie, 2018).
Big/Frontier Data Applications
| Satellite imagery | Poverty estimation (Jean et al., 2016; Chi et al., 2022); agricultural productivity (Benos et al., 2021); slum mapping (Müller et al., 2020); climate shock estimation (Macours et al., 2022) |
| Mobile phone data | Wealth estimation (Blumenstock, 2018); migration patterns (Lai et al., 2019) |
| Social media | Public sentiment analysis (Gorodnichenko et al., 2021); needs assessment during crises |
| Google Trends | Proxy for public needs during emergencies |
Note: These methods require advanced analytical expertise (geospatial analysis) and have margins of error that must be reported transparently.
What limitations and risks does the framework identify?
📉 Data quality & access
ML models are only as good as the data—inconsistent document structures required 40% manual extraction in Pilot 1. Big data access varies (public vs. restricted).
🧠 Technical capacity
Requires data science expertise; internal teams need foundational ML knowledge to collaborate effectively with external experts.
🔍 Methodological alignment
Not all ML methods fit evaluation questions—e.g., intelligent search underperformed basic keyword search in Pilot 3.
⚖️ Ethics & privacy
Generative AI tools (ChatGPT) pose data privacy risks when used with sensitive identifiable data; adherence to ethical principles (transparency, fairness, privacy, human rights) is critical.
📏 Accuracy thresholds
Pilot 1 summarizer missed information in ~10% of reports—insufficient for decision-making at scale.
What mitigation strategies does UNICEF recommend?
- Build in-house ML literacy: Managers should understand ML capabilities to design solutions and choose external experts.
- Start with low-cost, replicable pilots: Focus on tailor-made solutions for specific projects with potential for replication (e.g., keyword search on CPDs).
- Strengthen data management: Invest in systems (e.g., Databricks) for easy data access, interoperability, and reduced manual extraction.
- Engage in knowledge sharing: Participate in UN and cross-organizational ML working groups to learn from successes and failures.
- Be transparent about limitations: Report model accuracy levels and how they affect insights; learn from failed experiments.
- Build agile systems: Allow for integration of new ML tools as the field evolves rapidly.
- Adhere to ethical principles: Transparency, fairness, privacy, human rights, and human-centered approaches—especially with generative AI.
What are the key lessons for evaluators?
✅ ML enables scale and efficiency
Pilots demonstrated cost-effective scoping (527 reports labelled, 59 identified) and activity mapping across countries.
⚠️ Not all problems need advanced ML
Basic keyword search outperformed semantic search in Pilot 3—choose the simplest method that answers the question.
🧠 Managerial literacy is foundational
Evaluation managers must understand ML basics to design, oversee, and validate AI applications.
📚 Data preparation is the heavy lift
Text extraction, cleaning, and processing consume most effort—reuse processed data across analyses.
🔍 Accuracy thresholds matter
Pilot 1 summarizer missed information in ~10% of reports—insufficient for decision-making; human review remains essential.
🤝 Collaboration drives success
Interdisciplinary teams (evaluators + data scientists) produce more robust, practical solutions.
📊 Big data offers new frontiers
Satellite, mobile, and social media data can fill gaps where traditional data is unavailable—but require advanced skills and error reporting.
Frequently asked questions
What is UNICEF's Operational Framework for Machine Learning in Evaluation?
What ML methods did UNICEF test in its pilots?
How can ML strengthen impact evaluation with quantitative data?
What are the main risks of using ML in evaluation?
Can ML replace human evaluators?
Main reference & original sources
📘 This case study synthesizes documentation from UNICEF's Evaluation Office:
1. UNICEF (May 2024). An operational framework for Machine Learning in evaluation. New York: United Nations Children's Fund. Full PDF
2. UNICEF Evaluation Office. An operational framework for Machine Learning in evaluation (summary). UNICEF.org summary
Resources for further learning
Advance your skills in ML for evaluation
Learn how to apply supervised/unsupervised learning, NLP, and big data methods in your own evaluation work. Access courses, templates, and expert guidance aligned with UNICEF's framework.
