ChatGPT and Coding Framework
EvalCommunity Tutorial
How to Use ChatGPT to Develop a Coding Framework for Qualitative Research and Thematic Analysis
A practical guide for evaluators, M&E professionals, researchers, and development practitioners who want to use ChatGPT responsibly to create, refine, test, and document a qualitative coding framework.
Part of the EvalCommunity tutorial series: AI-Integrated Tools for Qualitative Research and Thematic Analysis
Tutorial Summary
A coding framework, also called a coding scheme or codebook, helps evaluators organize qualitative data into meaningful categories. It defines the themes, codes, sub-codes, inclusion rules, exclusion rules, and examples used to analyze interviews, focus groups, open-ended survey responses, case notes, reflection documents, and stakeholder feedback.
ChatGPT can support this process by helping evaluators generate an initial coding structure, refine code definitions, identify overlaps, create sub-codes, test the framework against sample responses, and prepare documentation for transparent reporting.
Key message: ChatGPT can help speed up the development of a coding framework, but it should not replace evaluator judgement. The final coding framework should always be reviewed, tested, refined, and validated by humans who understand the evaluation purpose, context, language, and ethical risks.
What You Will Learn
- How to use ChatGPT to create an initial qualitative coding framework.
- How to move from evaluation questions to themes, codes, and sub-codes.
- How to use AI to refine code definitions and reduce overlap between codes.
- How to test a coding framework on sample qualitative data.
- How to document AI-assisted coding transparently and responsibly.
- How to avoid common mistakes when using ChatGPT for thematic analysis.
Why Coding Frameworks Matter in Evaluation
Qualitative data can be rich, detailed, and highly contextual. However, without a clear coding framework, analysis can become inconsistent, subjective, or difficult to explain. A coding framework helps evaluation teams apply a structured approach to qualitative evidence.
For M&E teams, a good coding framework makes it easier to identify patterns, compare stakeholder perspectives, connect findings to evaluation questions, and generate evidence-based recommendations.
Consistency
A clear codebook helps different evaluators apply the same coding logic across interviews, focus groups, or survey responses.
Transparency
Documented codes, definitions, and examples make the analysis process easier to audit, explain, and defend.
Learning
A coding framework helps teams move from raw qualitative data to practical insights, lessons, and recommendations.
What ChatGPT Can and Cannot Do
ChatGPT can be useful during the design and refinement of a coding framework, especially when evaluators need a first draft, alternative categories, clearer definitions, or a structured codebook format. However, ChatGPT does not understand the full field context unless you provide it carefully, and it can miss cultural meaning, power dynamics, local terminology, or sensitive issues.
Use ChatGPT For
- Creating a first-draft codebook.
- Suggesting possible themes and sub-themes.
- Improving code definitions.
- Checking for overlapping codes.
- Generating inclusion and exclusion criteria.
- Preparing documentation for analysis methods.
Do Not Use ChatGPT As
- A replacement for qualitative expertise.
- A final decision-maker for findings.
- A tool for analyzing sensitive data without safeguards.
- A substitute for participant validation or sensemaking.
- A source of automatic conclusions.
- A place to upload confidential data without approval.
Before You Start: Prepare Your Qualitative Data
Before using ChatGPT, prepare a safe and structured input. You do not need to upload the full dataset to develop a coding framework. In many cases, it is better to use a small, anonymized sample of responses and a clear description of the evaluation context.
Recommended Input Structure
| Input | Purpose |
|---|---|
| Evaluation purpose | Explains why the analysis is being conducted. |
| Evaluation questions | Guides the coding framework toward the evidence needed. |
| Type of data | Interview transcripts, focus group notes, open-ended survey responses, complaints, or reflection notes. |
| Stakeholder groups | Participants, staff, partners, government officials, community members, donors, or service users. |
| Sample excerpts | A small, anonymized sample helps ChatGPT suggest relevant codes. |
| Analytical approach | Inductive, deductive, hybrid, thematic, framework-based, outcome-focused, or learning-oriented. |
Data Safety Checklist
- Remove names, phone numbers, email addresses, exact locations, and other personal identifiers.
- Do not upload confidential transcripts unless your organization has approved the tool and data handling process.
- Use small anonymized samples when developing the framework.
- Separate sensitive safeguarding, protection, legal, or misconduct cases for human-led review.
- Document what was uploaded, why it was uploaded, and how the AI output was used.
Step-by-Step Workflow: Using ChatGPT to Develop a Coding Framework
Step 1: Define the Evaluation Focus
Start with the evaluation questions. A coding framework should not be a generic list of themes. It should help answer specific questions about relevance, effectiveness, outcomes, barriers, equity, sustainability, accountability, or learning.
Step 2: Decide the Coding Approach
Choose whether your coding framework will be deductive, inductive, or hybrid.
- Deductive coding: Codes are based on the evaluation questions, theory of change, OECD-DAC criteria, programme logic, or donor reporting framework.
- Inductive coding: Codes emerge from the data itself.
- Hybrid coding: Some codes are predefined, while others emerge from participant responses.
Step 3: Ask ChatGPT for a First-Draft Framework
Provide ChatGPT with the evaluation purpose, questions, type of qualitative data, stakeholder groups, and a small anonymized sample. Ask it to produce themes, codes, sub-codes, definitions, inclusion rules, exclusion rules, and example excerpts.
Step 4: Review the First Draft
Do not accept the first output automatically. Review whether the codes are relevant, clear, distinct, and aligned with the evaluation questions.
- Are some codes too broad?
- Are some codes duplicated or overlapping?
- Are important stakeholder perspectives missing?
- Are sensitive issues treated appropriately?
- Are the definitions clear enough for multiple coders?
Step 5: Improve the Codebook
Ask ChatGPT to identify overlapping codes, merge duplicates, split broad themes, strengthen definitions, and add inclusion and exclusion criteria. This helps transform a simple theme list into a usable coding framework.
Step 6: Test the Framework on Sample Responses
Use a small sample of anonymized responses and ask ChatGPT to apply the coding framework. Then check whether the assigned codes make sense. This helps identify unclear definitions, missing codes, and inconsistent coding rules.
Step 7: Validate with Human Review
The evaluation team should manually review the coding framework, test it on a sample, compare coding decisions, and agree on final definitions. For participatory evaluations, stakeholders may also be invited to review the themes during sensemaking sessions.
Step 8: Document the Process
Document how ChatGPT was used, what data was provided, how outputs were reviewed, what changes were made by humans, and how final codes were validated. This improves transparency and trust in the analysis.
Prompt Templates for Evaluators
The quality of ChatGPT’s output depends on the quality of your prompt. Use structured prompts that explain the evaluation context, the type of data, the coding approach, and the output format you need.
Prompt 1: Generate an Initial Coding Framework
You are supporting a qualitative evaluation analysis. Evaluation purpose: [Insert purpose] Evaluation questions: [Insert questions] Type of data: [Interviews / focus groups / open-ended survey responses / reflection notes] Stakeholder groups: [Insert groups] Coding approach: [Deductive / inductive / hybrid] Sample anonymized excerpts: [Insert 5–10 short excerpts] Please develop a first-draft coding framework with: 1. Main themes 2. Codes 3. Sub-codes where useful 4. Definitions 5. Inclusion criteria 6. Exclusion criteria 7. Example excerpts 8. Notes for human reviewers
Prompt 2: Improve Code Definitions
Review the coding framework below. Please improve it by: 1. Making each code definition clearer 2. Identifying overlapping codes 3. Suggesting which codes should be merged 4. Suggesting which codes should be split 5. Adding inclusion and exclusion rules 6. Making the framework easier for multiple coders to apply consistently Coding framework: [Paste coding framework]
Prompt 3: Test the Framework on Sample Data
Apply the coding framework below to the anonymized sample excerpts. For each excerpt, provide: 1. Suggested code or codes 2. Brief justification 3. Any uncertainty 4. Suggestions to improve the framework Coding framework: [Paste framework] Sample excerpts: [Paste anonymized excerpts]
Prompt 4: Draft the Methodology Note
Help me draft a transparent methodology note for an evaluation report. The qualitative analysis used: - [Interviews / focus groups / open-ended survey responses] - A coding framework developed with AI assistance - Human review and refinement - Manual validation of final themes Please write a concise methodology paragraph that explains how ChatGPT supported the coding framework development, while making clear that final interpretation and validation were conducted by human evaluators.
Worked Example: Coding Framework for Beneficiary Feedback
Imagine an NGO is evaluating a livelihood training programme. The evaluation team has collected open-ended survey responses from participants. The team wants to understand what changed for participants, what barriers they faced, and what should be improved.
| Evaluation Question | Possible Theme | Example Codes |
|---|---|---|
| What changed for participants? | Participant outcomes | Improved income, confidence, technical skills, business planning, social connections |
| What barriers did participants face? | Implementation barriers | Transport cost, childcare, timing, unclear communication, limited follow-up |
| What should improve? | Programme improvement | More mentoring, practical exercises, local language support, longer training, financial support |
Human Reviewer Question
After ChatGPT suggests these themes, the evaluator should ask: Do these codes reflect the actual programme logic, participant language, local context, and intended use of the findings?
Coding Framework Template
Use this structure to document your AI-assisted coding framework. This table can be copied into Word, Excel, Google Sheets, MAXQDA, NVivo, ATLAS.ti, Dedoose, Taguette, or another qualitative analysis tool.
| Theme | Code | Definition | Include When | Exclude When | Example |
|---|---|---|---|---|---|
| Participant outcomes | Improved confidence | Participant describes increased confidence, self-belief, or willingness to act. | The response mentions confidence, motivation, self-esteem, or feeling able to try something new. | The response only mentions technical skills without confidence or motivation. | “I now feel confident to start my small business.” |
| Implementation barriers | Transport cost | Participant identifies travel cost or transport access as a barrier. | The response mentions travel cost, distance, transport availability, or difficulty reaching the activity. | The response mentions general financial challenges unrelated to attending the programme. | “It was hard to attend because transport was expensive.” |
| Programme improvement | Need for mentoring | Participant asks for follow-up guidance, coaching, or continued support. | The response mentions mentoring, coaching, follow-up visits, or help after the training. | The response asks for more training content but not follow-up support. | “We need someone to guide us after the training ends.” |
Quality Checks for an AI-Assisted Codebook
Before using the coding framework for full analysis, apply quality checks. These checks help ensure that the framework is usable, clear, and aligned with the evaluation purpose.
Relevance Check
Each code should help answer an evaluation question or support a specific learning need.
Clarity Check
Each code should have a clear definition that another evaluator can apply consistently.
Overlap Check
Codes should not duplicate each other. If two codes are too similar, merge or clarify them.
Context Check
Codes should reflect the programme context, participant language, and local meaning.
Sensitivity Check
Sensitive issues should be coded carefully and handled through appropriate human-led protocols.
Evidence Check
Final themes should be supported by actual excerpts, not only by AI-generated summaries.
Responsible AI Use in Qualitative Analysis
Using ChatGPT for qualitative analysis requires careful attention to privacy, consent, data protection, transparency, and human oversight. This is especially important when working with vulnerable populations, sensitive topics, complaints, safeguarding information, or politically sensitive contexts.
Important Safeguards
- Do not upload personally identifiable information unless your organization has approved the tool and data handling process.
- Use anonymized excerpts instead of raw transcripts whenever possible.
- Do not rely on AI-generated themes as final findings without human review.
- Do not use ChatGPT to make decisions about individuals, eligibility, protection cases, misconduct, or safeguarding incidents.
- Keep a record of prompts, outputs, human changes, and validation steps.
- Explain in the evaluation methodology how AI was used and how human evaluators validated the outputs.
Suggested Disclosure Statement
ChatGPT was used to support the development and refinement of the qualitative coding framework. The evaluation team provided anonymized contextual information and sample excerpts, reviewed AI-generated suggestions, revised code definitions, tested the framework against source responses, and retained responsibility for final coding decisions, interpretation, findings, and recommendations.
Common Mistakes to Avoid
Mistake 1: Asking ChatGPT for themes without context
A weak prompt such as “create themes from this data” often produces generic results. Provide evaluation questions, programme context, stakeholder groups, and the intended use of findings.
Mistake 2: Using the first output as the final codebook
The first AI output should be treated as a draft. Evaluators should refine it, remove weak codes, add missing codes, and test it on real excerpts.
Mistake 3: Uploading sensitive transcripts without safeguards
Qualitative data can include personal, political, emotional, or protection-related information. Use anonymization, organizational approval, and appropriate data controls before using any AI tool.
Mistake 4: Reporting AI-generated themes without evidence
Evaluation findings should be grounded in source evidence. Always check themes against actual excerpts and explain how interpretations were validated.
Frequently Asked Questions
Can ChatGPT create a coding framework for qualitative analysis?
Yes. ChatGPT can help create a first-draft coding framework, suggest themes and sub-codes, improve definitions, and test the framework on sample excerpts. However, the final framework should be validated by human evaluators.
Can ChatGPT replace manual qualitative coding?
No. ChatGPT can support coding framework development and analysis, but qualitative interpretation requires human judgement, contextual knowledge, ethical awareness, and evidence validation.
Should I upload full interview transcripts to ChatGPT?
Only if your organization permits it and the data handling process is appropriate. In many cases, it is safer to use anonymized excerpts or synthetic examples when developing the coding framework.
Can I use the coding framework in tools like NVivo, MAXQDA, ATLAS.ti, or Excel?
Yes. ChatGPT can help create a structured codebook that can be transferred into qualitative analysis tools, spreadsheets, or collaborative review documents.
Final Takeaway
ChatGPT can help evaluators move faster from evaluation questions to a structured coding framework. It can support brainstorming, codebook development, refinement, testing, and documentation.
The strongest approach is not to let AI decide the findings, but to combine AI-assisted organization with human-led interpretation, contextual knowledge, ethical safeguards, and transparent validation.
Sources and Further Reading
This tutorial was developed as an original EvalCommunity learning resource for evaluators and M&E professionals. The responsible AI and data-use guidance should always be adapted to your organization’s policies and legal obligations.
“`
