STEP 1

Define the Outcome You Want to Measure

What to do: Write a clear, specific definition of the outcome you want to measure.

How to do it: Ask: What change are we trying to measure? Who is affected? What does success look like?

Example: “Trust in local health services: the degree to which community members believe healthcare providers are competent, reliable, and responsive.”

STEP 2

Build a Measurement Framework

What to do: Break the outcome into measurable components or dimensions.

How to do it: Identify 2-5 sub-dimensions that together define the outcome.

Example – Trust in Health Services: Competence (good treatment, accurate diagnosis), Reliability (regular attendance, available medicines), Respect (fair treatment, listening to concerns).

STEP 3

Identify Data Sources

What to do: Create an inventory of all available qualitative data.

How to do it: List each data source, its format, language, volume, and suitability for AI analysis.

Example: 5,000 open-ended survey responses (English, CSV format), 200 interview transcripts (Spanish, Word documents).

STEP 4

Design a Coding Scale

What to do: Create a numeric scale with clear anchor points.

How to do it: Define each score level with a label and example response.

Example Trust Scale: 0=Strong distrust (“Health workers never come”), 1=Distrust (“The clinic is unreliable”), 2=Neutral (“Some good, some bad”), 3=Trust (“Generally reliable”), 4=Strong trust (“Completely trust them”).

STEP 5

Create a Gold Standard Dataset

What to do: Manually code a representative sample of 100-500 responses.

How to do it: Have two independent coders each score the sample. Resolve disagreements through discussion.

Example: Select 200 responses covering all score levels, all respondent types, and all time periods.

STEP 6

Build the AI Prompt

What to do: Write structured instructions for the AI model.

How to do it: Include role definition, scale with examples, output format, and edge case handling.

You are an M&E assistant. Score the following text on trust in health services.
Scale: 0=Strong distrust, 1=Distrust, 2=Neutral, 3=Trust, 4=Strong trust
Return ONLY valid JSON with "score" and "reasoning".
Do not include any text outside the JSON.
Text: [INSERT RESPONSE]

STEP 7

Run AI Classification on Test Sample

What to do: Process your gold standard dataset through the AI model.

How to do it: Use API calls or batch processing to get AI scores for each response.

Example: Run 200 gold standard responses through the latest GPT model using your prompt.

STEP 8

Review AI Explanations

What to do: Examine the AI’s reasoning for each score.

How to do it: For a sample of responses, compare AI reasoning to human reasoning.

Example: “Score: 4. Reason: Respondent mentions regular service availability and strong confidence in healthcare workers.”

STEP 9

Calculate Validation Metrics

What to do: Compare AI scores to human gold standard scores.

How to do it: Calculate percent agreement, Cohen’s Kappa, and accuracy.

Example: 87% agreement, Kappa=0.82, indicating substantial agreement.

STEP 10

Improve the Prompt and Assess Bias

What to do: Analyze where AI got it wrong and why, then test for bias across demographic groups.

How to do it: Review disagreement cases. Add clarifying examples. Adjust scale definitions. Then calculate agreement separately for subgroups.

Example prompt improvement: AI confuses “trust” with “satisfaction”. Add examples distinguishing them.

STEP 11

Scale to Full Dataset

What to do: Once validated, process all remaining data.

How to do it: Use batch API processing. Include error handling and rate limiting.

Example: Process 10,000 responses in batches of 100, saving outputs to CSV.

For large-scale processing, see our AI for Data Extraction from Transcripts tutorial.

STEP 12

Create Composite Indicators

What to do: Combine multiple related variables into a single index.

How to do it: Average or sum component scores after normalizing scales.

Example: Resilience Index = (Trust + Participation + Cohesion + Agency) ÷ 4

STEP 13

Analyze the Variable

What to do: Use the AI-generated variable in statistical analysis.

How to do it: Import the variable into Excel, R, Python, SPSS, or Stata.

Example questions: Do women report higher trust? Does trust differ by region? Does trust predict program retention?

STEP 14

Document Everything

What to do: Create a transparent record of your methodology.

How to do it: Record model, prompt version, validation results, limitations, and human review process.

Example documentation: “Model: Latest GPT model. Prompt version v3. Validation: 87% agreement, Kappa=0.82 on 200 responses.”

For guidance on documentation standards, see Designing Informed Consent for AI Tools.

STEP 15

Report Findings Responsibly

What to do: Disclose AI use transparently in reports.

How to do it: Include methodology, validation, limitations, and human oversight.

Example statement: “Trust scores were generated using AI-assisted classification of open-ended beneficiary feedback. A manually coded validation sample of 200 responses achieved 87% agreement. Human review was conducted on uncertain cases.”

import openai
import pandas as pd
import json
import time
client = openai.OpenAI(api_key="your-api-key")
# Define the prompt template
prompt_template = """
You are an M&E assistant. Score the following text on trust in health services.
Scale: 0=Strong distrust, 1=Distrust, 2=Neutral, 3=Trust, 4=Strong trust
Return ONLY valid JSON with "score" and "reasoning".
Text: {text}
"""
# Use the latest model supported by your provider
MODEL_NAME = "your-selected-model"  # Replace with current model name
def score_response(text, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model=MODEL_NAME,
                messages=[{"role": "user", "content": prompt_template.format(text=text)}],
                temperature=0.2
            )
            result = json.loads(response.choices[0].message.content)
            return result.get("score", None), result.get("reasoning", ""), result.get("confidence", None)
        except json.JSONDecodeError:
            return None, "JSON parsing error", None
        except Exception as e:
            if attempt < max_retries - 1:
                time.sleep(2 ** attempt)
            else:
                return None, f"Error: {str(e)}", None
    return None, "Max retries exceeded", None
# Process batch
df = pd.read_csv("responses.csv")
results = []
for _, row in df.iterrows():
    score, reasoning, confidence = score_response(row["response_text"])
    results.append({
        "id": row["respondent_id"],
        "trust_score": score,
        "reasoning": reasoning,
        "confidence": confidence
    })
output_df = pd.DataFrame(results)
output_df.to_csv("trust_scores.csv", index=False)
print(f"Processed {len(results)} responses")

from sklearn.metrics import cohen_kappa_score
import pandas as pd
df = pd.read_csv("validation_dataset.csv")
agreement = (df["human_score"] == df["ai_score"]).mean() * 100
kappa = cohen_kappa_score(df["human_score"], df["ai_score"], weights="quadratic")
print(f"Agreement Rate: {agreement:.1f}%")
print(f"Weighted Cohen's Kappa: {kappa:.2f}")
# Assess bias across groups
for group in ["gender", "region"]:
    group_agreement = df.groupby(group).apply(
        lambda x: (x["human_score"] == x["ai_score"]).mean() * 100
    )
    print(f"Agreement by {group}:\n{group_agreement}")
# Optional: if your model returns confidence scores
if "confidence" in df.columns:
    low_confidence = df[df["confidence"] < 0.8]
    low_confidence.to_csv("needs_review.csv", index=False)
    print(f"Low confidence cases requiring review: {len(low_confidence)}")
else:
    print("Confidence scores not available - review a random sample instead")

Primary Source

3ie Blog: Making AI work: Practical applications in impact evaluation

AI Governance

NIST AI Risk Management Framework | OECD AI Principles

Evaluation Standards

OECD DAC Evaluation Criteria | UNEG Ethical Guidelines

Qualitative Methods

BetterEvaluation: Qualitative Data Coding | Cohen’s Kappa Documentation

Responsible Data and AI Bias

Humanitarian Data Centre | UNICEF: AI and Children’s Rights