Raw Feedback Sample

“The training was excellent. I learned new farming techniques. My income increased. The trainers were very helpful.”

AI Word Frequency Output

training: 1 | excellent: 1 | learned: 1 | farming: 1 | techniques: 1 | income: 1 | increased: 1 | trainers: 1 | helpful: 1

STEP 1

Define Your Content Analysis Questions

What to do: Be specific about what you want to count or measure.

How to do it: Write 3-5 quantitative questions about your text data.

Example questions: How many respondents mention increased income? What percentage of feedback is positive? Which keywords appear most frequently?

STEP 2

Select Your Content Analysis Approach

What to do: Choose between concept frequency, category coding, sentiment analysis, word frequency, or comparative analysis.

How to do it: Match your research questions to the appropriate method.

Example: “How many mention income?” → Concept frequency. “What is the overall sentiment?” → Sentiment analysis.

STEP 3

Prepare and Clean Your Text Data

What to do: Consolidate all text data into a single organized format.

How to do it: Create a spreadsheet where each row contains one response or document with metadata.

Example format: Respondent ID | Gender | Location | Response Text

STEP 4

Review a Sample Manually

What to do: Read 50-100 responses to understand the language and potential categories.

How to do it: As you read, note recurring words, phrases, and concepts.

Example observations: “Income,” “money,” “earnings” appear frequently. “Skills,” “training,” “learned” also common.

STEP 5

Create Your Content Analysis Categories

What to do: Define the concepts or categories you want to count.

How to do it: Based on your manual review, create a list of target concepts with synonyms.

Example category: “Income Increase” – Include: income, earnings, salary, profit, revenue, money. Exclude: cost, expense, payment.

STEP 6

Build a Gold-Standard Coded Sample

What to do: Manually code a representative sample of 100-200 responses.

How to do it: For each response, mark which categories/concepts are present.

Example: “My income increased and I learned new skills.” → Categories: Income Increase, New Skills.

STEP 7

Design the AI Prompt for Content Analysis

What to do: Write structured instructions for counting concepts.

How to do it: Include categories, definitions, and output format (counts or binary indicators).

You are a content analysis assistant for M&E.
For the text below, count whether these concepts appear:
- Income Increase: Mentions earning more money, higher salary, increased profits
- New Skills: Mentions learning new techniques, knowledge, or abilities
- Challenges: Mentions barriers, problems, or difficulties
Return JSON with each concept as true/false and a confidence score.
Text: [INSERT RESPONSE]

STEP 8

Test AI on Your Coded Sample

What to do: Process your gold-standard sample through the AI model.

How to do it: Compare AI counts/classifications to human coding.

STEP 9

Calculate Agreement and Accuracy

What to do: Measure how well AI matches human coding.

How to do it: Calculate percent agreement and Cohen’s Kappa for each category.

Interpretation: Aim for 85%+ agreement for reliable content analysis.

STEP 10

Refine Categories Based on Errors

What to do: Review cases where AI disagreed with human coding.

How to do it: Identify unclear category definitions or missing synonyms.

Example refinement: Add “earnings” and “wages” to the Income Increase category.

STEP 11

Scale to Full Dataset

What to do: Once validated, process all text data through AI.

How to do it: Use batch processing for thousands of responses.

Example: Process 10,000 open-ended survey responses.

STEP 12

Generate Frequency Tables

What to do: Count how many responses contain each concept.

How to do it: Sum the binary indicators for each category.

STEP 13

Generate Word Clouds and Visualizations

What to do: Create visual representations of word frequencies.

How to do it: Use AI to extract top keywords and frequency data for visualization tools.

Tools: Python wordcloud library, WordArt.com, or Tableau.

STEP 14

Create Comparative Content Tables

What to do: Compare concept frequencies across subgroups.

How to do it: Use metadata to disaggregate results by gender, location, age, etc.

STEP 15

Validate Findings with a Second Sample

What to do: Randomly select an additional 100-200 responses for verification.

How to do it: Compare AI counts to human coding on this new sample.

Example: If agreement remains high (85%+), findings are reliable.

STEP 16

Report Content Analysis Findings Transparently

What to do: Document your methodology and results clearly.

How to do it: Include category definitions, validation results, frequency tables, and limitations.

Example statement: “Content analysis of 5,000 open-ended responses identified that 86% of beneficiaries mentioned increased income (Cohen’s Kappa = 0.85 with human-coded validation sample).”

Baseline (Before Program)

“I struggle to grow enough food for my family. We often go hungry during dry season.”

Endline (After Program)

“I learned new farming techniques. My harvest has doubled. I now earn enough to send my children to school.”

import openai
import pandas as pd
import json
client = openai.OpenAI(api_key="your-api-key")
prompt_template = """
You are a content analysis assistant.
For the text below, indicate whether each concept appears.
Concepts: Income Increase, New Skills, Empowerment, Challenges
Return JSON with each concept as true/false.
Text: {text}
"""
def analyze_text(text):
    response = client.chat.completions.create(
        model="your-selected-model",
        messages=[{"role": "user", "content": prompt_template.format(text=text)}],
        temperature=0
    )
    return json.loads(response.choices[0].message.content)
df = pd.read_csv("responses.csv")
results = []
for _, row in df.iterrows():
    result = analyze_text(row["response_text"])
    results.append(result)
results_df = pd.DataFrame(results)
frequency = results_df.sum() / len(results_df) * 100
print(frequency)