Multi-Agent Qualitative Analysis System
EvalCommunity Academy · AI Agents for Evaluators
Build a Multi-Agent Qualitative Analysis System for Evaluators
From interview transcripts to coding, thematic analysis, triangulation and evidence-based findings. Build a coordinated AI workflow that supports qualitative research without replacing the evaluator’s judgement.
No-code prototype
90–120 minutes
Beginner to intermediate
What is a multi-agent qualitative analysis system?
It is a coordinated workflow in which specialised AI roles support different stages of qualitative research, such as transcript preparation, codebook development, coding, theme development, cross-case comparison, quality review and reporting. Each output is linked to its source evidence and reviewed by a human researcher before it becomes an accepted finding.
1. Why use multiple AI roles for qualitative analysis?
Qualitative research helps evaluators understand how people experience programmes, why implementation succeeds or fails, how change happens and what unexpected effects emerge.
But analysing interviews, focus group discussions, field notes and open-ended survey responses takes time. Researchers must read and interpret material, develop codes, compare cases, examine contradictory evidence and explain how their findings are supported.
A single AI prompt may generate a convincing summary, but it can also hide important analytical decisions. It may overlook minority perspectives, merge different experiences into broad themes, misinterpret context or generate quotations that do not match the original transcript.
A coordinated workflow makes these tasks more transparent. Each role has a defined responsibility, a specific output and a review requirement. The objective is not to make AI independently decide what participants mean. It is to make the analytical process more organised, traceable and easier to challenge.
The central principle
AI can help organise evidence and propose interpretations. The researcher remains responsible for the methodology, context, interpretation and final findings.
2. What you will build
You will create a seven-role qualitative analysis workflow, coordinated by a separate manager role and supervised by a human evaluation lead.
Prepare
Organise transcripts and flag quality or privacy issues.
Structure
Develop a transparent coding framework.
Code
Apply approved codes to source-linked passages.
Interpret
Develop and challenge candidate themes.
Compare
Examine differences across cases and sources.
Validate
Check quotations, evidence and analytical limitations.
Expected outputs:
- Transcript inventory and quality report.
- Draft codebook and coding instructions.
- Structured coded dataset.
- Candidate thematic framework.
- Cross-case and triangulation matrix.
- Quality assurance report.
- Draft qualitative findings report.
3. First choose the qualitative method
A coding workflow is not automatically appropriate for every qualitative methodology. The research question, theoretical position and analytical approach must determine how AI is used.
This tutorial demonstrates an applied hybrid deductive–inductive coding workflow. It starts with provisional concepts informed by evaluation questions and allows additional codes to emerge from the material.
Important: do not impose a fixed codebook, independent coding agreement target or consensus procedure on reflexive thematic analysis without considering whether these procedures fit the chosen methodology. Reflexive thematic analysis treats researcher interpretation and reflexivity differently from structured codebook-based approaches.
4. Understand the main analytical approaches
| Approach | Purpose | Possible AI support | Key consideration |
|---|---|---|---|
| Codebook-based thematic analysis | Develop and apply a structured coding framework to identify patterns. | Code suggestions, structured coding and comparisons. | Define and review coding rules. |
| Reflexive thematic analysis | Interpret patterns of shared meaning through researcher engagement. | Familiarisation support, reflective questioning and alternative interpretations. | Do not treat themes as mechanically discovered or validated by agreement scores. |
| Framework analysis | Organise cases and categories in a systematic analytical framework. | Indexing, charting and matrix preparation. | Keep summaries connected to source material. |
| Qualitative content analysis | Classify text according to defined analytical categories. | Category suggestions and structured classification. | Define the unit of analysis and category rules. |
| Grounded theory | Develop concepts and theory through systematic engagement with data. | Memo organisation and comparison support. | Avoid imposing predetermined categories that conflict with the chosen grounded theory approach. |
For the practical exercise, we use an applied hybrid coding approach. The codebook is provisional and must be reviewed before full-scale coding.
5. The practical case: women’s economic empowerment
We will use the hypothetical programme introduced in Tutorial 2. It operates in three rural districts and provides vocational training, business mentoring, financial literacy, savings groups and market access support.
Illustrative evidence sources
- Interviews with women participating in programme activities.
- Interviews with trainers, mentors and field staff.
- Interviews with relevant community stakeholders.
- Focus group discussions, where available.
- Programme monitoring records and other authorised documents.
These are proposed data categories, not actual collected research evidence. No real participant transcripts, consent records or programme findings have been supplied for this tutorial.
Fictional training excerpt · P01
“The business training helped me understand how to calculate my costs. Before, I sometimes sold products without knowing whether I was making a profit. But getting to the market is still difficult. I need to arrange childcare, and transport costs reduce what I earn.”
Fictional example created for training. Not a real participant quotation or research finding.
6. Design the multi-agent architecture
The system has seven specialist roles. A separate workflow manager coordinates them, while a human evaluation lead approves the method, analytical decisions and final findings.
Human Evaluation Lead
Methodology, ethics, interpretation and approval
Qualitative Workflow Manager
Task coordination, versions, evidence and status tracking
Transcript preparation
Codebook development
Qualitative coding
Thematic analysis
Triangulation
Quality review
Findings and reporting
These are specialist AI roles in a coordinated workflow. They are not automatically independent autonomous agents with persistent memory, independent judgement or guaranteed access to the same data.
7. Establish shared instructions for every role
Use a common set of rules across all prompts. This reduces inconsistent outputs and makes the workflow easier to audit.
- Use only authorised source material supplied for the task.
- Never invent quotations, participant characteristics, sample sizes or findings.
- Preserve original wording and relevant context.
- Link each coded passage to a transcript ID and stable segment reference.
- Distinguish participant statements from analyst interpretations.
- Preserve contradictory, unexpected and minority perspectives.
- Do not assume that frequency equals importance.
- Do not infer sensitive characteristics that are not appropriately available.
- Flag ambiguous, incomplete, translated or uncertain passages.
- Mark generated codes and themes as provisional until reviewed.
- Never claim that AI agreement proves independent validation.
- Do not process confidential data through an unapproved service.
8. Agent 1 — Transcript Preparation Specialist
Objective: organise transcripts, check their quality and flag potential privacy issues without changing participant meaning.
This role checks speaker labels, missing sections, transcription issues, language and document structure. It flags potentially identifying information for authorised human review; it does not independently decide what can be deleted from the research record.
Copy-ready prompt
ROLE: Qualitative Transcript Preparation Specialist. PROJECT ID: [INSERT] TRANSCRIPT ID: [INSERT] SOURCE: [INSERT AUTHORISED TRANSCRIPT] TASK: Prepare a transcript quality report and a structured version suitable for analysis. RULES: 1. Preserve original wording and meaning. 2. Retain speaker labels and paragraph boundaries. 3. Identify missing, unclear or incomplete passages. 4. Flag possible transcription and translation errors. 5. Flag potential identifying information for authorised human review. 6. Do not infer or reconstruct participant identities. 7. Do not silently correct uncertain wording. 8. Assign stable paragraph or segment references. 9. Do not generate findings or interpret experiences. 10. Do not overwrite the original source. OUTPUT: - Project and transcript IDs - Source file reference - Completeness assessment - Speaker and paragraph structure - Unclear or missing passages - Potential transcription issues - Potential identifying information requiring review - Recommended corrections for human approval - Readiness: ready for review / correction required / incomplete If a structured transcript is requested, retain the exact source text and add stable segment IDs.
Human review gate: approve data handling, transcript preparation and anonymisation decisions before sending material to an AI service.
9. Agent 2 — Codebook Development Specialist
Objective: develop a provisional coding framework with clear definitions and rules for applying each code.
A codebook should define more than keywords. It should explain what each code means, when it should be used, when it should not be used and how overlapping concepts should be distinguished.
Copy-ready prompt
ROLE: Qualitative Research Codebook Developer. EVALUATION PURPOSE: [INSERT] EVALUATION QUESTIONS: [INSERT] APPROVED CONCEPTUAL FRAMEWORK: [INSERT IF RELEVANT] AUTHORised TRANSCRIPT SAMPLE: [INSERT APPROVED SAMPLE] TASK: Develop a provisional hybrid deductive-inductive codebook. INSTRUCTIONS: 1. Propose deductive codes informed by the evaluation questions. 2. Identify possible inductive codes emerging from the supplied data. 3. Keep conceptually different codes distinct. 4. Avoid unnecessary duplication and overly broad categories. 5. Define every code in plain language. 6. Include inclusion and exclusion criteria. 7. Provide source-grounded examples where available. 8. Identify overlapping codes and explain their differences. 9. Preserve unexpected and contradictory concepts. 10. Do not assume that every participant shares the same experience. 11. Mark all proposed codes as provisional. 12. Do not claim that the codebook is validated. OUTPUT TABLE: - Code ID - Code name - Deductive or inductive - Definition - Inclusion criteria - Exclusion criteria - Example source reference - Related codes - Researcher review question Then identify: A. Possible code clusters B. Missing concepts C. Ambiguous definitions D. Potentially sensitive categories E. Recommendations for pilot coding.
Illustrative starter codebook
| ID | Provisional code | Meaning | Fictional example |
|---|---|---|---|
| C01 | Business skills | Skills used to manage business activities. | Learning to calculate costs. |
| C02 | Financial management | Understanding costs, revenue and profit. | Not knowing whether a sale generated profit. |
| C03 | Market access | Barriers to reaching buyers or markets. | Difficulty getting to the market. |
| C04 | Care responsibilities | Unpaid care affecting participation or work. | Arranging childcare. |
| C05 | Transport costs | Transport expenditure affecting access or earnings. | Transport reducing what the participant earns. |
These codes are illustrative only. They are not a validated coding framework.
Human review gate: approve the coding approach and pilot codebook before full-scale coding.
10. Agent 3 — Qualitative Coding Specialist
Objective: apply the approved codebook to transcript segments and preserve traceability to the original material.
Copy-ready prompt
ROLE: Qualitative Research Coding Specialist. APPROVED CODEBOOK: [PASTE APPROVED VERSION] TRANSCRIPT: [PASTE AUTHORISED TRANSCRIPT OR CHUNK] PROJECT ID: [INSERT] TRANSCRIPT ID: [INSERT] SOURCE VERSION: [INSERT] TASK: Apply the approved codebook to the supplied material. INSTRUCTIONS: 1. Read the supplied material in context. 2. Identify relevant text segments. 3. Preserve exact wording in quotations. 4. Assign codes only when definitions are met. 5. Explain each coding decision briefly. 6. Include transcript and segment references. 7. Flag material that does not fit existing codes. 8. Suggest new codes separately, not as approved codes. 9. Preserve negative and contradictory evidence. 10. Do not infer motivations or personal characteristics. 11. Flag ambiguity instead of inventing meaning. 12. Do not generate themes at this stage. 13. Do not treat code frequency as importance. OUTPUT: - Project ID - Transcript ID - Source version - Segment ID - Exact source excerpt - Code ID and name - Coding rationale - Context note - Uncertainty flag - Proposed new code, if applicable - Human review status Finish with a summary of codes applied, uncoded material, possible new codes and issues requiring review.
Example coding output
| Segment | Fictional excerpt | Code | Reason |
|---|---|---|---|
| P01-003 | “The business training helped me understand how to calculate my costs.” | C01 | Describes a business skill. |
| P01-004 | “I sometimes sold products without knowing whether I was making a profit.” | C02 | Describes a financial management issue. |
| P01-005 | “Getting to the market is still difficult.” | C03 | Describes a market access barrier. |
| P01-006 | “I need to arrange childcare.” | C04 | Describes care responsibilities. |
| P01-007 | “Transport costs reduce what I earn.” | C05 | Connects transport costs with earnings. |
The segment IDs are illustrative. Real projects must use verified transcript locations.
11. Agent 4 — Thematic Analysis Specialist
Objective: examine relationships between codes and propose candidate patterns of meaning relevant to the evaluation questions.
A theme should express a meaningful analytical idea. It should not simply repeat a code name or group together frequently mentioned topics.
Copy-ready prompt
ROLE: Qualitative Thematic Analysis Specialist. ANALYTICAL APPROACH: [INSERT APPROVED APPROACH] RESEARCH QUESTIONS: [INSERT] APPROVED CODEBOOK: [INSERT] CODED DATA WITH SOURCE REFERENCES: [INSERT] TASK: Develop provisional analytical patterns or candidate themes appropriate to the stated method. INSTRUCTIONS: 1. Examine relationships between codes and cases. 2. Identify candidate themes relevant to the research questions. 3. Explain the central organising idea of each theme. 4. Distinguish a theme from a broad topic or code cluster. 5. Include supporting source references. 6. Preserve contradictory and negative cases. 7. Identify differences between participant experiences. 8. Consider alternative interpretations. 9. Do not assume that frequent codes are the most important themes. 10. Do not generalise beyond the available evidence. 11. Distinguish descriptive observations from interpretive claims. 12. Do not invent quotations or source references. 13. State when evidence is insufficient. 14. Explain which decisions require researcher interpretation. OUTPUT: - Candidate theme ID and title - Central organising concept - Definition - Related codes - Supporting source references - Contradictory evidence - Differences across cases - Alternative interpretations - Relevance to evaluation questions - Limitations - Human review questions Mark all themes as provisional.
Illustrative movement from codes to themes
Candidate Theme 1: New skills and business confidence
Possible related codes: business skills, financial management and decision-making.
Candidate Theme 2: Economic participation constrained by everyday barriers
Possible related codes: market access, care responsibilities and transport costs.
These are examples based on one fictional excerpt. They are not actual programme findings.
Human review gate: examine whether the candidate themes are analytically meaningful, supported by the complete dataset and consistent with the chosen methodology.
12. Agent 5 — Triangulation and Cross-Case Specialist
Objective: compare evidence across participants, stakeholder groups, locations, time periods and data sources.
Triangulation is not simply finding multiple sources that agree. Differences can be equally important because they may reveal unequal experiences, implementation barriers or conflicting interpretations.
Copy-ready prompt
ROLE: Qualitative Evidence Triangulation Specialist. APPROVED THEMES: [INSERT] SOURCE-REFERENCED QUALITATIVE EVIDENCE: [INSERT] OTHER AUTHORISED SOURCES: [INSERT SURVEYS, MONITORING DATA, DOCUMENTS OR OTHER EVIDENCE] COMPARISON DIMENSIONS: [INSERT ONLY RELEVANT, AUTHORISED DIMENSIONS] TASK: Compare evidence across cases and sources. INSTRUCTIONS: 1. Identify converging evidence. 2. Identify conflicting or divergent evidence. 3. Identify complementary information. 4. Identify missing evidence. 5. Compare groups only where the data support the comparison. 6. Preserve minority and negative cases. 7. Do not assume one participant represents a group. 8. Distinguish absence of evidence from evidence of absence. 9. Do not invent explanations for differences. 10. Do not claim causality from source agreement. 11. Preserve source references. 12. Identify questions requiring additional research. OUTPUT TABLE: - Evaluation question or theme - Evidence source - Relevant case or group - Converging evidence - Divergent evidence - Complementary insights - Missing evidence - Possible explanations requiring investigation - Interpretation limitations Finish with unresolved questions and recommendations for human follow-up.
Four possible evidence relationships
Convergence
Different sources support a similar interpretation.
Divergence
Sources report different or conflicting experiences.
Complementarity
Sources illuminate different aspects of an issue.
Missing evidence
Available sources do not answer an important question.
13. Agent 6 — Quality Assurance Reviewer
Objective: challenge the analytical process and identify weaknesses in coding, evidence, interpretation and reporting.
Copy-ready prompt
ROLE: Independent Qualitative Analysis Quality Reviewer. SOURCE MATERIAL OR VERIFIED SOURCE REFERENCES: [INSERT] CODEBOOK: [INSERT] CODED DATA: [INSERT] THEMES: [INSERT] TRIANGULATION MATRIX: [INSERT] TASK: Critically assess the quality, traceability and limitations of the analysis. REVIEW: 1. Do coded excerpts match the original source? 2. Are transcript and segment references correct? 3. Are codes applied consistently with their definitions? 4. Are there overlapping, vague or overly broad codes? 5. Is important material overlooked? 6. Are themes supported by the evidence? 7. Are themes merely topic labels? 8. Are negative cases and contradictory accounts retained? 9. Are group comparisons supported by the data? 10. Are interpretations clearly separated from participant statements? 11. Are quotations accurate and in context? 12. Are translation limitations acknowledged? 13. Are causal claims justified? 14. Are ethical or confidentiality risks visible? 15. Are important limitations disclosed? OUTPUT: - Critical errors - Major concerns - Minor improvements - Source references requiring verification - Coding decisions requiring review - Themes requiring revision - Missing or contradictory evidence - Potential bias and overgeneralisation - Recommended corrections - Human decisions still required For every issue, provide its source reference, severity, reason and recommended action. Do not declare the analysis valid merely because another AI role produced it.
14. Agent 7 — Findings and Reporting Specialist
Objective: prepare an evidence-linked draft findings report from reviewed analytical material.
Copy-ready prompt
ROLE: Qualitative Findings and Reporting Specialist. APPROVED EVALUATION QUESTIONS: [INSERT] RESEARCH METHOD: [INSERT] APPROVED THEMES: [INSERT] VALIDATED SOURCE-REFERENCED EVIDENCE: [INSERT] TRIANGULATION RESULTS: [INSERT] QUALITY REVIEW AND RESOLUTIONS: [INSERT] TASK: Prepare a draft qualitative findings report for researcher review. INSTRUCTIONS: 1. Organise findings around evaluation questions and approved themes. 2. State each analytical claim clearly. 3. Explain the evidence supporting each claim. 4. Use direct quotations only when verified against original sources. 5. Include transcript and segment references. 6. Preserve differences and contradictory evidence. 7. Distinguish perceptions from independently established facts. 8. Distinguish reported change from demonstrated programme contribution or impact. 9. Avoid generalising beyond the available evidence. 10. Disclose sampling and data limitations. 11. Do not invent participant characteristics. 12. Do not conceal uncertainty or unresolved disagreement. OUTPUT: A. Summary of qualitative findings B. Analytical approach C. Theme-by-theme findings D. Supporting evidence and verified quotations E. Differences and contradictory cases F. Triangulation findings G. Limitations and uncertainties H. Implications for programme implementation I. Recommendations for evaluator consideration J. Unresolved questions Label the entire report as a draft for human review.
15. The workflow manager: coordinate the seven roles
The manager controls the sequence, passes approved information between roles and maintains the project record. It does not replace the human evaluation lead.
Copy-ready prompt
ROLE: Qualitative Analysis Workflow Manager. PROJECT ID: [INSERT] APPROVED PROJECT METHOD: [INSERT] AVAILABLE OUTPUTS: [INSERT TRANSCRIPT INVENTORY, CODEBOOK, CODED DATA, THEMES, TRIANGULATION AND QUALITY REVIEW] TASK: Coordinate specialist outputs and prepare a traceable analysis package. INSTRUCTIONS: 1. Maintain project and transcript IDs. 2. Track which source records have been processed. 3. Check that the methodology and codebook have human approval. 4. Record source and output versions. 5. Identify incomplete or failed tasks. 6. Preserve contradictory findings. 7. Never alter original quotations. 8. Keep unresolved analytical decisions visible. 9. Incorporate review corrections only when supported. 10. Record which outputs remain unreviewed. 11. Maintain a decision log. 12. Do not treat agreement between AI roles as independent validation. 13. Prepare a final package for human review. OUTPUT: - Project status - Transcript processing status - Approved method and codebook version - Coding completion and review status - Candidate and approved themes - Triangulation summary - Quality issues and corrective actions - Evidence traceability report - Unresolved analytical decisions - Human approval checklist STATUS: Draft analysis for human review / additional evidence required / methodological revision required / ready for researcher interpretation. Never label findings final without explicit human approval.
16. Connect the workflow in the correct order
The roles should not all execute simultaneously. Each stage depends on approved outputs from earlier stages.
Consent, access and data management confirmed
Quality and privacy checks
Approve data handling and transcript readiness
Provisional coding framework
Approve method and pilot codebook
Source-linked records
Candidate patterns
Cross-source comparison
Challenge and correct the analysis
Evidence-linked draft
17. Pilot and validate before processing the full dataset
Do not apply an untested prompt and codebook to every transcript. Start with a small, varied subset and examine the quality of the outputs.
- Select a small set of authorised transcripts that captures relevant variation.
- Have a qualified researcher analyse the selected material using the chosen method.
- Run the AI workflow on the same material.
- Compare coding decisions, segment boundaries, omissions and interpretations.
- Investigate disagreements rather than assuming either the human or AI output is automatically correct.
- Revise the prompts and codebook where appropriate.
- Repeat the pilot and document remaining limitations.
- Use ongoing human checks during full processing.
Agreement is not the same as validity.
For structured codebook-based approaches, agreement measures such as Cohen’s kappa may be useful when their assumptions are met. However, an agreement score does not establish that the coding captures participants’ meanings or that the resulting interpretation is valid. Reflexive thematic analysis uses different quality principles and should not automatically be judged by mechanical agreement targets.
What should you test?
| Dimension | Validation question | Action |
|---|---|---|
| Coding accuracy | Does the passage fit the code definition? | Review against human analysis. |
| Quotation fidelity | Does it match the original transcript? | Verify exact source text. |
| Overcoding | Are irrelevant passages being coded? | Check definitions and boundaries. |
| Undercoding | Are meaningful passages being missed? | Review uncoded material. |
| Context | Does the interpretation preserve surrounding meaning? | Review full case context. |
| Bias | Are perspectives being represented unevenly? | Examine differences across relevant cases. |
18. Build the first version without coding
Option A — Manual prototype
- Create a secure project folder and assign a project ID.
- Prepare an authorised transcript sample.
- Run the Transcript Preparation prompt.
- Review and approve the codebook.
- Run pilot coding and compare the results with human coding.
- Revise the coding instructions where necessary.
- Develop candidate themes and compare cases.
- Run triangulation where other authorised sources are available.
- Run the quality review and resolve critical issues.
- Prepare the draft findings report for human approval.
Option B — Automate the workflow
| Component | Possible technology | Purpose |
|---|---|---|
| Secure storage | Approved organisational storage | Manage authorised source files. |
| Project tracking | Controlled spreadsheet or database | Track IDs, versions and review status. |
| Orchestration | Make.com or n8n | Coordinate task execution and handoffs. |
| AI processing | Approved model API | Run role-specific prompts. |
| Qualitative analysis software | NVivo, MAXQDA, ATLAS.ti or equivalent | Manage and retrieve qualitative material. |
These are possible components, not mandatory tools. Confirm organisational approval, data residency, retention, access controls and service terms before selecting a platform.
19. Technical rules for reliable automation
A workflow tool such as Make.com or n8n is an orchestrator, not automatically an AI agent. It executes the sequence, transfers approved outputs and manages task status.
Recommended implementation rules
- Use stable IDs. Every project, transcript, segment, code and theme should have a persistent identifier.
- Process transcripts in manageable chunks. Preserve paragraph boundaries and include enough surrounding context to avoid misleading interpretations.
- Use controlled overlap. If long transcripts are split into overlapping chunks, deduplicate repeated passages by their stable segment IDs.
- Use structured outputs. Request JSON or another consistent schema for automated handoffs. Validate required fields before continuing.
- Maintain source references. Never detach a quotation or coded segment from its original transcript and location.
- Version the workflow. Record the model, prompt version, codebook version and source version used for each output.
- Handle failures. Record failed tasks, retry safely and prevent incomplete outputs from being marked approved.
- Control access. Keep API keys outside prompts and restrict access to confidential files.
- Control cost. Avoid repeatedly sending the complete corpus when only a specific source-linked subset is required.
- Keep human checkpoints. Do not automatically move draft outputs into final reporting.
A production workflow needs more than seven prompts. Each module must be configured with input mapping, output parsing, error handling, access control and an approval state. This tutorial provides the architecture and prompt templates, not an importable Make.com or n8n workflow.
20. Build an evidence traceability structure
Every reported finding should be traceable to the original source material. Use a consistent data structure throughout the project.
| Field | Illustrative value | Purpose |
|---|---|---|
| project_id | QUAL-2026-001 | Links project records. |
| transcript_id | P01 | Identifies a source document. |
| segment_id | P01-003 | Locates a passage. |
| code_id | C01 | Links the segment to a code. |
| theme_id | T01 | Links codes to themes. |
| source_version | Transcript-v2 | Tracks source revisions. |
| review_status | Draft / reviewed / approved | Distinguishes unreviewed and approved work. |
Keep original transcripts immutable. Store prepared transcripts, coded records, analytical memos and revisions separately. Record who or what generated each output, when it was generated and which instructions were used.
21. Protect participants and research data
Interview transcripts can contain sensitive information even when the research topic appears routine. Names, occupations, locations, family circumstances and combinations of details may make someone identifiable.
Do not upload confidential transcripts to an unapproved AI service.
Confirm research consent, ethics requirements, organisational policy, service terms, retention arrangements, access controls and security requirements before processing. Removing names alone may not sufficiently anonymise a transcript.
- Process only information required for the analytical task.
- Use project-specific IDs instead of names and contact details.
- Store identity keys separately with restricted access.
- Assess re-identification risks from locations, occupations and personal stories.
- Confirm approved service settings and data-processing terms.
- Restrict access and maintain appropriate access records.
- Establish secure retention and deletion procedures.
- Check whether direct quotations could identify participants.
- Use competent human review for translation and local context.
- Ensure participant information and consent cover the intended data processing.
22. Understand the risks of AI-assisted qualitative analysis
Research on AI-assisted qualitative analysis reports both useful applications and significant limitations. Some studies show that AI can generate plausible descriptive insights; others identify overcoding, weak agreement or systematic bias. Results depend on the dataset, analytical method, model and validation procedure.
| Risk | Possible problem | Safeguard |
|---|---|---|
| Overcoding | Too much text is assigned to a code. | Check boundaries and definitions. |
| Overgeneralisation | Different experiences become one broad theme. | Compare cases and preserve variation. |
| Confirmation bias | Expected programme results dominate interpretation. | Look deliberately for disconfirming evidence. |
| Quotation errors | Altered wording appears as a direct quotation. | Verify every quote against the source. |
| Context loss | A passage is interpreted in isolation. | Review surrounding discussion and case context. |
| Language bias | Idioms, dialects or translation are misunderstood. | Involve competent language reviewers. |
| False consensus | AI roles repeat the same unsupported claim. | Trace claims to original evidence and human review. |
23. Test your multi-agent system
Use challenging examples to find weaknesses before scaling up.
| Test | Input | Expected behaviour |
|---|---|---|
| Ambiguous passage | An unclear participant statement. | Flag uncertainty. |
| Contradictory evidence | Different participants report different experiences. | Preserve and compare both accounts. |
| Missing source ID | A coded passage has no reference. | Flag as untraceable. |
| Incorrect quotation | Proposed quotation differs from the transcript. | Reject and record the error. |
| Unexpected concept | A passage does not fit the codebook. | Suggest a new code separately. |
| Sensitive information | A transcript contains identifying details. | Follow approved privacy procedures. |
24. Practical challenge: analyse a small interview dataset
Build a manual prototype using the fictional programme or an authorised dataset that you are permitted to process.
- Prepare two or three authorised, anonymised training transcripts.
- Run the Transcript Preparation Specialist.
- Develop and review the provisional codebook.
- Manually code a sample of the material.
- Run the AI coding role on the same sample.
- Compare the outputs and investigate disagreements.
- Revise the prompts or codebook where appropriate.
- Develop candidate themes and examine differences across cases.
- Run triangulation if other authorised evidence is available.
- Run quality assurance and resolve critical issues.
- Prepare a draft findings report with source references.
- Document human interpretation and remaining limitations.
Suggested time: 90–120 minutes for a small manual prototype. Larger datasets and automated workflows require additional time.
25. What this system cannot reliably do alone
- Determine the true meaning of participant experiences without contextual interpretation.
- Establish theme importance simply from frequency.
- Demonstrate statistical representativeness from a purposive qualitative sample.
- Verify a quotation without checking the original source.
- Resolve cultural and linguistic ambiguity without relevant expertise.
- Turn reported change into proven causal impact.
- Guarantee that multiple AI roles are independent or unbiased.
- Replace research ethics approval, informed consent or secure data governance.
The value of the system is in supporting disciplined work: organising material, proposing codes, identifying patterns, comparing evidence and making analytical decisions easier to inspect.
Frequently asked questions
What is AI-assisted qualitative data analysis?
It uses AI to support tasks such as transcript organisation, coding, comparison, summarisation and candidate theme development. The researcher remains responsible for the method, interpretation and findings.
Can AI code interview transcripts?
Yes. AI can propose and apply codes to text segments, but performance depends on the dataset, codebook, context and model. Pilot testing and human review are necessary.
Can AI generate themes automatically?
It can propose candidate themes. Researchers must assess whether they represent meaningful patterns, preserve differences and answer the research questions. A generated theme list is not automatically a valid thematic analysis.
What is the difference between a code and a theme?
A code labels a meaningful segment of data. A theme expresses a broader pattern of meaning developed through analysis. A theme should have a clear central idea rather than simply group similar words.
Does AI coding require inter-coder reliability statistics?
Not in every methodology. Agreement measures may be appropriate for some structured coding approaches, but other approaches, including reflexive thematic analysis, use different quality principles. Select validation procedures that fit the method.
Can AI analyse interviews in different languages?
Some models support multiple languages, but performance varies across languages, dialects and idioms. Preserve original-language material where possible and use competent human reviewers for translation and interpretation.
Can I upload confidential transcripts to ChatGPT?
Only when the relevant consent, organisational policies, service terms and security arrangements permit it. Anonymisation, access controls and re-identification risk must be considered before processing.
Can multi-agent AI replace NVivo, MAXQDA or ATLAS.ti?
Not necessarily. AI agents can support analytical tasks, while qualitative software provides data organisation, retrieval, coding and collaboration functions. They may complement one another.
Further reading and research
These resources support the methodological and technical principles used in this tutorial.
- CDC (2024). Collecting and Analyzing Qualitative Data. Guidance on data management, coding, memos, interpretation and verification.
- NICE (2022). Conduct of Qualitative Research Studies. Guidance on methodology, reflexivity, sampling, ethics, negative cases and triangulation.
- UN Women (2022). UN Women Evaluation Handbook. Guidance on gender-responsive evaluation and ethical evaluation practice.
- Wachinger et al. (2025). Prompts, Pearls, Imperfections: Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis. Examines the opportunities and limitations of AI-supported analysis compared with human interpretation.
- Boettinger, Bürgel and Bartsch (2026). A Hybrid Intelligence Approach to Qualitative Data Analysis. Examines manual and AI coding, overcoding, agreement and validation.
- Jafarian (2026). Human-in-the-Loop LLM-Assisted Thematic Analysis. A recent tutorial on structured AI-assisted thematic analysis with iterative human verification.
From transcripts to evidence-based findings
A multi-agent qualitative analysis system can help evaluators organise complex material, structure coding, compare experiences and prepare evidence-linked findings.
But success is not measured by how many transcripts AI can process. It is measured by whether the analysis preserves participant meaning, recognises differences, makes uncertainty visible and supports better evaluation decisions.
Continue learning with EvalCommunity Academy
Explore practical learning resources on AI agents, qualitative analysis, Monitoring and Evaluation and international development.
EvalCommunity Academy · Practical AI learning for Monitoring, Evaluation, Accountability and Learning (MEAL) and international development professionals.
