EvalCommunity Academy Case Study

AI vs. Traditional Qualitative Interviews: What Did UNICEF Learn?

How UNICEF is experimenting with AI-assisted qualitative interviews and what this means for evaluation design, evidence quality, bias, participant experience and human judgment.

Primary source:

UNICEF — AI in Evaluation: From Interviews to Insights

Case at a Glance

UNICEF’s 2026 evaluation session presents a field experiment comparing
AI-assisted qualitative interviews with traditional qualitative interviews.

The same session also showcases an AI social-media listening tool developed by UNICEF’s Social and Behaviour Change team and an AI-based equity-analysis tool from India.

1. The UNICEF Case

UNICEF’s AI in Evaluation: From Interviews to Insights session presents a
field experiment in which AI-assisted and traditional qualitative interviews are compared.
The session was part of gLOCAL Evaluation Week 2026.

According to UNICEF, the broader objective of the session was to illustrate how AI can expand data sources, deepen insights and support more adaptive and inclusive evaluation practice.

Read the UNICEF Source →

2. The Evaluation Question

At the centre of this case is a practical question:

Can AI-assisted qualitative interviews generate useful evaluation evidence compared with traditional qualitative interviews?

This is more important than simply asking whether an AI system can conduct an interview. The relevant M&E question is whether the resulting evidence is sufficiently credible, relevant, useful and ethically generated for evaluation purposes.

3. Traditional Qualitative Interviews

In a conventional qualitative interview, the researcher or interviewer manages the interaction with the participant.

The interviewer typically:

introduces the interview;
asks the core questions;
listens to responses;
asks follow-up questions;
clarifies unexpected information;
manages the flow of the conversation;
records or supervises the recording of the interview; and
helps ensure that important contextual information is captured.

4. The AI-Assisted Approach

The UNICEF experiment introduces AI into the data-collection stage of the evaluation rather than limiting AI to transcription, coding or analysis.

Evaluation purpose → Interview design → AI-assisted interview → Participant responses → Evaluation analysis → Human interpretation

This is significant because AI is being considered as part of the process through which evaluation evidence is generated.

5. What Should Evaluators Compare?

Dimension	Traditional interview	AI-assisted interview
Consistency	Can vary by interviewer	Potentially more standardized
Probing	Human contextual judgment	AI-generated follow-up questions
Scale	Constrained by interviewer time	Potential for greater scalability
Bias	Interviewer effects are possible	Model and system effects are possible
Participant experience	Human interaction	Human–AI interaction
Quality assurance	Human supervision	AI + human quality assurance

EvalCommunity note: This comparison is an analytical framework developed for this case study. It is not a table of findings reported by UNICEF.

6. Quality: The Critical M&E Question

AI-assisted interviewing should not be evaluated only by asking whether the technology works.

Evaluators should examine whether the interview produces credible and sufficiently rich evidence.

Important dimensions include:

question coverage;
quality of follow-up probing;
depth of responses;
contextual understanding;
neutrality of questioning;
accuracy of transcripts and summaries;
traceability to participant evidence; and
human verification.

7. The Probing Problem

Qualitative interviewing is not simply a process of asking a predetermined list of questions.

The interviewer listens and decides when an answer requires clarification or deeper exploration.

Participant:

“The programme worked, but there were some problems.”

A useful follow-up might explore:

What problems?
Can you provide an example?
Who was affected?
Why do you think this happened?
Did the situation change over time?

The quality of AI-assisted interviewing therefore depends partly on whether AI can ask useful, neutral and contextually appropriate follow-up questions.

8. Bias: Human and AI Sources

AI should not be presented as automatically eliminating interviewer bias.

Traditional interviews can involve:

interviewer effects;
differences in interviewer skill;
inconsistent probing;
social desirability effects; and
differences in interpretation.

AI-assisted interviews can introduce:

model bias;
language limitations;
cultural misunderstandings;
systematic probing patterns;
leading or inappropriate questions; and
automation bias in subsequent analysis.

The M&E question is therefore not simply whether bias exists, but where bias enters the evidence-generation process and whether it can be detected.

9. Participant Experience

Qualitative interviews are also social interactions. The participant experience therefore matters.

Evaluators should ask:

Did participants understand that AI was involved?
Did they feel comfortable interacting with the system?
Did they feel that they were being heard?
Did AI change what they were willing to disclose?
Did the approach work equally well across participant groups?

10. Ethics, Privacy and Confidentiality

Introducing AI into qualitative interviews changes the data-processing environment and therefore raises additional questions about confidentiality, consent and data governance.

Where are interview data stored?
Who can access them?
Is the AI provider processing the data?
How long are transcripts retained?
Are participants clearly informed about AI involvement?
Can participants withdraw?
How are sensitive disclosures handled?

Evidence boundary: The UNICEF public source documents the AI-assisted interview field experiment, but does not provide enough detail on this page to characterize every privacy, consent or data-governance procedure used in the experiment. The questions above are therefore proposed M&E considerations, not claims about UNICEF’s implementation.

11. The AI Evidence Chain

AI-assisted interviewing adds another layer to the process through which evaluation evidence is generated.

Evaluation Question
→ Interview Design
→ AI Interaction
→ Participant Response
→ AI Probe
→ Transcript
→ Analysis
→ Interpretation
→ Evaluation Finding

At each stage, evaluators can ask:
What could go wrong here, and how would we know?

12. EvalCommunity Quality Framework

Based on the UNICEF case, EvalCommunity proposes eight dimensions for evaluating AI-assisted qualitative interviews:

Protocol fidelity — Did the AI follow the approved interview design?
Probing quality — Were follow-up questions relevant and neutral?
Coverage — Were important topics adequately explored?
Depth — Did participants provide sufficiently detailed evidence?
Neutrality — Did questioning avoid leading participants?
Participant experience — How did participants perceive the interaction?
Equity — Did performance differ across groups or languages?
Traceability — Can findings be traced back to participant evidence?

This is an EvalCommunity adaptation for M&E practice and is not presented as a UNICEF framework.

13. What Could Evaluators Measure?

Area	Possible indicator
Protocol	% of required questions delivered correctly
Probing	% of probes judged relevant by independent reviewers
Neutrality	% of questions classified as non-leading
Coverage	% of predefined topics adequately covered
Efficiency	Time required per completed interview
Participant experience	Participant comfort or satisfaction measure
Equity	Differences in quality or completion across participant groups
Traceability	% of findings traceable to original participant evidence

These are proposed EvalCommunity indicators, not indicators reported by UNICEF.

14. A Practical Comparison Design

The UNICEF experiment provides a useful model for a broader M&E approach: evaluate the introduction of AI rather than assuming its value.

Traditional interviews

↓

AI-assisted interviews

↓

Compare quality, depth, probing, participant experience, equity, time and cost

15. A Human-Centred AI Evaluation Workflow

A practical workflow for evaluators could be:

Define purpose — establish the evaluation question and evidence needs.
Design — develop the interview guide and safeguards.
Test — conduct controlled AI-assisted interviews.
Compare — compare AI-assisted and traditional approaches.
Verify — have human evaluators review interview quality.
Monitor — track quality, bias, equity and participant experience.
Learn — improve the AI workflow and evaluation design.

16. UNICEF’s Broader AI-in-Evaluation Work

The interview experiment was presented alongside two other AI applications:

an AI social-media listening tool developed by UNICEF’s Social and Behaviour Change team; and
an AI-based equity-analysis tool from India.

Together, these examples illustrate that AI can enter the evaluation ecosystem at multiple points—from data collection and listening to analysis and equity assessment.

This broader interpretation is based on the scope of the UNICEF session; it does not imply that all three tools have the same level of deployment, evidence or maturity.

17. Lessons for Evaluators

1. Evaluate AI as part of the evidence system.

2. Do not confuse greater scale with better evidence.

3. Evaluate the quality of AI probing.

4. Measure participant experience.

5. Examine equity across participant groups.

6. Preserve traceability from findings to source evidence.

7. Keep human judgment at the centre of evaluation.

18. Evidence Boundary

What UNICEF documents: a field experiment comparing AI-assisted and traditional qualitative interviews, together with demonstrations of an AI social-media listening tool and an AI-based equity-analysis tool. :contentReference[oaicite:1]{index=1}

What this case study adds: an EvalCommunity interpretation of what the experiment means for evaluation design, quality assurance, bias assessment, participant experience and evidence generation.

The proposed indicators, AI Evidence Chain and quality framework in this article are EvalCommunity adaptations. They should not be presented as UNICEF’s own framework or as empirical findings from the UNICEF experiment.

Key Takeaway

The important question is not simply whether AI can conduct a qualitative interview.

The real M&E question is: Can AI-mediated conversations produce evidence that evaluators can trust—and under what conditions?

EvalCommunity Academy

Learn How to Put AI to Work in Your M&E Practice

Learn how to use AI across evaluation workflows—from evaluation design and evidence synthesis to qualitative analysis, reporting, ethics, bias and human oversight—while keeping professional judgment in charge.

Explore the AI in Monitoring & Evaluation Certificate →