UNDP’s Latin America and Caribbean evaluation used AI-assisted qualitative coding
AI-Assisted Qualitative Coding in Evaluation: The UNDP Regional Programme Case
How an independent UNDP evaluation used AI-assisted qualitative analysis to structure interviews, survey responses and documents while keeping interpretation human-led.
Primary sources:
Evidano case study
Artificial intelligence is beginning to move beyond analysing completed qualitative datasets and into the
qualitative analysis workflow itself.
This UNDP case provides a documented example of an independent evaluation team using
AILYZE, now Evidano, for structured qualitative analysis. The published case describes AI-assisted coding,
thematic categorization, pattern recognition and focused tagging, while stating that analytical judgments,
synthesis and interpretation remained the responsibility of the evaluation team.
The evaluation at a glance
| Element | Reported information |
|---|---|
| Evaluation | Final Evaluation of the UNDP Regional Programme for Latin America and the Caribbean 2022–2025 |
| Informants | 103 |
| Key informant interviews | 70+ |
| Country Offices surveyed | 26 |
| Subregions | 4 |
| Evaluation approach | Mixed methods |
| AI-assisted platform | AILYZE, now Evidano |
What was evaluated?
The case concerns the Final Evaluation of the UNDP Regional Programme for Latin America and the Caribbean
2022–2025, commissioned by UNDP’s Regional Bureau for Latin America and the Caribbean.
The UNDP Evaluation Resource Centre records the evaluation as completed on
31 December 2025. It identifies the evaluation as completed, records a budget of
US$60,000 and expenditure of US$62,450, and indicates that a quality assessment was conducted.
The Evidano case study reports that the evaluation involved more than 70 key informant interviews,
open-ended responses from a survey of all 26 Country Offices, two deep-dive case studies and document reviews.
The qualitative-analysis challenge
The evaluation brought together several forms of qualitative evidence:
- Key informant interviews
- Open-ended survey responses
- Case-study inputs
- Documents
According to the published case description, the evaluation team used AILYZE to support structured
content analysis of this qualitative material.
The stated purpose was not simply to produce summaries. The platform was used to support
systematic coding, thematic categorization, pattern recognition, theme clustering and focused tagging.
How AI fit into the workflow
Unstructured qualitative evidence
Interviews, open-ended survey responses, case-study inputs and documents entered the analysis workflow.
Systematic coding and thematic analysis
AILYZE supported coding, thematic categorization, pattern recognition, theme clustering and identification
of emerging insights across interviews and documents.
Focused tagging
The published case specifically identifies tagging related to
gender, inclusion, partnerships and performance.
Human interpretation
The case states that analytical judgments, synthesis and interpretations remained the responsibility
of the evaluation team.
Human-led, not AI-led evaluation
One of the most important features of the case is the boundary between
AI-assisted analysis and evaluation judgment.
The published case states that
“All analytical judgments, synthesis, and interpretations remained the sole responsibility
of the Evaluation Team.”
A multilingual evidence chain
The published case also reports that Spanish-language transcripts were translated and
human-verified before analysis.
This creates an important methodological consideration for international evaluations:
translation can become another stage in the evidence chain before coding and interpretation.
Participant language → Transcription → Translation → Verification → Coding → Themes → Interpretation → Finding
The available public case material does not provide quantitative estimates of translation errors
or comparative coding accuracy by language. Those claims should therefore not be inferred from this case.
EvalCommunity AI Evidence Chain
The following is an EvalCommunity adaptation, not a framework reported by UNDP.
Qualitative Data
→
AI-Assisted Coding
→
Themes & Patterns
→
Human Verification
→
Synthesis
→
Interpretation
→
Evaluation Finding
The key principle is simple:
an AI-generated theme is not automatically an evaluation finding.
The evaluator still needs to determine whether the evidence supports the interpretation,
whether alternative explanations exist, and whether contradictory evidence has been adequately considered.
Why traceability matters
The published case states that AI use enhanced the reliability and transparency of qualitative analysis
through consistent data treatment and traceable coding logic, while helping manage evaluator workload.
For M&E professionals, this raises a broader methodological question:
Can an AI-assisted workflow make qualitative analysis easier to audit?
Potentially, if analytical outputs can be connected back to the underlying evidence.
But traceability is not the same as validity.
A traceable coding process can still contain inappropriate codes, missed evidence,
contextual errors or weak interpretations. The evaluator therefore needs to assess both
the audit trail and the quality of the analytical judgment.
EvalCommunity AI Quality Framework for Qualitative Coding
The framework below is proposed by EvalCommunity for M&E practice. It is not presented as a UNDP framework.
| Dimension | Question for evaluators |
|---|---|
| Coding accuracy | Does AI coding agree with validated human coding? |
| Coverage | Are important pieces of evidence being missed? |
| Context sensitivity | Does coding preserve meaning and context? |
| Theme quality | Are identified themes analytically meaningful? |
| Contradiction preservation | Are divergent and contradictory views retained? |
| Traceability | Can themes be traced back to source evidence? |
| Human oversight | Where does expert review occur? |
| Interpretive validity | Are conclusions grounded in the evaluation evidence? |
Possible indicators for AI-assisted qualitative analysis
For teams implementing AI-assisted coding, EvalCommunity proposes monitoring indicators such as:
- Coding agreement: agreement between AI-assisted coding and validated human coding.
- Critical omission rate: important evidence segments missed by the AI-assisted process.
- Traceability rate: proportion of analytical outputs that can be linked to identifiable source evidence.
- Human correction rate: proportion of AI-assisted codes or themes requiring substantive modification.
- Contradiction preservation: extent to which divergent evidence remains visible.
- Cross-source consistency: whether comparable themes are identified across different evidence sources without erasing source differences.
- Interpretation verification: whether AI-supported analytical outputs are reviewed before entering the final evaluation synthesis.
What this case does — and does not — demonstrate
The documented evidence supports:
- AI-assisted qualitative analysis was used in the evaluation.
- The evaluation involved more than 70 key informant interviews.
- All 26 Country Offices were surveyed.
- AI supported coding, thematic categorization, pattern recognition and focused tagging.
- Human experts retained responsibility for analytical interpretation.
- The published case reports benefits relating to consistency, traceability and evaluator workload.
The available sources do not establish:
- that AI coding was more accurate than human coding in this evaluation;
- that AI independently produced the evaluation findings;
- that AI eliminated evaluator bias;
- that AI reduced total evaluation cost by a specified percentage; or
- that AI can replace expert qualitative researchers.
A practical workflow for M&E teams
- Define the analytical framework. Establish evaluation questions, themes and coding criteria.
- Prepare the evidence. Organize transcripts, survey responses and documents while preserving source identifiers.
- Establish a human baseline. Manually code a sample before scaling AI-assisted analysis.
- Run AI-assisted coding. Use AI to identify potential codes, patterns and themes.
- Review. Have qualified evaluators examine the AI-assisted outputs.
- Investigate disagreement. Examine why AI and human coding differ.
- Preserve traceability. Maintain links between codes, themes, quotations and original sources.
- Synthesize. Use validated analytical material to develop findings.
- Apply evaluator judgment. Consider context, competing explanations and evidence limitations.
- Document AI use. State where AI was used and where human judgment remained responsible.
AI should assist the evidence process — not become the evaluator
The UNDP case illustrates a potentially important model for AI in qualitative evaluation:
use AI to help organize, code and interrogate large bodies of qualitative evidence,
while keeping interpretation, synthesis and evaluative judgment with qualified human experts.
Key takeaway
The important innovation is not simply using AI to code qualitative data.
It is creating a workflow in which AI-assisted analysis remains
traceable, reviewable and human-led.
For evaluators, the question is therefore moving from
“Can AI code qualitative data?”
toward:
“How can we demonstrate that AI-assisted coding produces trustworthy evidence?”
Evidence boundary
Source-reported: the evaluation scope, number of informants and interviews,
Country Offices surveyed, qualitative data sources, use of AILYZE/Evidano,
reported analytical functions, human verification of Spanish-language transcripts,
and the stated responsibility of the evaluation team for analytical judgments.
EvalCommunity interpretation: the AI Evidence Chain,
proposed quality framework, proposed indicators and recommended workflow.
Not established by the available sources: a quantified comparison of AI coding accuracy
against human coding for this specific evaluation, causal evidence that AI improved evaluation quality,
or evidence that AI can replace expert qualitative analysis.
Learn how to evaluate and use AI in M&E
Explore practical approaches to AI-assisted qualitative analysis, AI evaluation,
evidence quality, human oversight and responsible AI use in Monitoring & Evaluation.
