Raw Interview Excerpt

“Before the project, women in our village never spoke at community meetings. Now, we have five women on the village council. They speak up about health issues and children’s education. Some men were resistant at first, but now they see the value. We didn’t expect this to happen so quickly.”

AI Harvested Outcomes

• Behavioral: Women now speak at community meetings
• Structural: Five women elected to village council
• Attitudinal: Men now see value in women’s participation
• Unexpected: Rapid change (unexpected timeline)

STEP 1

Define the Scope of Your Harvest

What to do: Decide which programs, time periods, and outcome types you want to harvest.

How to do it: Ask: What programs are we evaluating? What time period? Which stakeholders? What outcome types interest us?

Example scope: “Harvest outcomes from the agricultural livelihoods program (2020-2025) across all 10 districts, focusing on behavioral and systemic changes.”

STEP 2

Gather Your Source Materials

What to do: Collect all documents and transcripts relevant to the harvest.

How to do it: Gather program reports, evaluation reports, case studies, interview transcripts, focus group notes, and stakeholder feedback.

Example sources: 50 quarterly reports, 30 stakeholder interviews, 15 case studies, 200 beneficiary feedback comments.

STEP 3

Review a Sample Manually

What to do: Read a small sample of documents to understand the language and potential outcomes.

How to do it: Review 10-20 pages or 5-10 interviews manually to identify outcome patterns.

Example observations: “Women’s leadership,” “farmers formed groups,” “policy changed,” “unexpected cooperation between communities.”

STEP 4

Create Outcome Harvesting Categories

What to do: Define the categories you will use to classify outcomes.

How to do it: Based on your manual review, create categories like: Outcome Type (behavioral/relational/systemic), Domain (health/education/livelihoods), Significance (high/medium/low).

Example categories: Behavioral Change, Relational Change, Attitudinal Change, Capability Change, Systemic Change, Unintended.

STEP 5

Create Category Definitions with Examples

What to do: Write clear definitions for each category with inclusion/exclusion criteria.

How to do it: For each category, specify what counts and provide example outcome statements.

Example: “Behavioral Change” – Include: Changes in actions, practices, behaviors. Exclude: Changes in attitudes alone. Example: “Farmers now use drought-resistant seeds.”

STEP 6

Build a Gold-Standard Harvested Sample

What to do: Manually harvest outcomes from a representative sample of documents.

How to do it: Review 20-50 pages or 10-20 interview excerpts and identify all outcome statements manually.

Example: Create a spreadsheet with columns: Outcome Statement, Outcome Type, Domain, Significance, Evidence Quote.

STEP 7

Design the AI Outcome Harvesting Prompt

What to do: Write structured instructions for outcome extraction.

How to do it: Include outcome types, categories, output format, and confidence scoring.

You are an outcome harvesting assistant for M&E.
From the text below, extract ALL outcome statements.
For each outcome, identify:
- Outcome statement (a specific change)
- Outcome type (behavioral/relational/attitudinal/capability/systemic/unintended)
- Significance (high/medium/low)
- Evidence quote
Return as JSON array of outcomes.
Text: [INSERT TEXT]

STEP 8

Test AI on Your Gold-Standard Sample

What to do: Process your manually harvested sample through the AI model.

How to do it: Compare AI-extracted outcomes to human-extracted outcomes.

Metrics to calculate: Outcome detection rate (did AI find all outcomes?), classification accuracy (did AI categorize correctly?), false positive rate (did AI invent outcomes?).

STEP 9

Calculate Outcome Harvesting Accuracy

What to do: Measure how well AI performs at outcome detection and classification.

How to do it: Calculate precision (accuracy of outcomes found), recall (completeness), and F1 score.

Interpretation: Aim for recall > 85% (finding most outcomes) and precision > 80% (not inventing false outcomes).

STEP 10

Refine the Prompt Based on Errors

What to do: Review cases where AI missed outcomes or misclassified them.

How to do it: Identify unclear definitions, missing outcome types, or ambiguous language.

Example refinement: Add “now” and “currently” as indicators of change. Add “unexpectedly” as a signal for unintended outcomes.

STEP 11

Scale to Full Document Corpus

What to do: Process all documents and transcripts through AI.

How to do it: Use batch processing for hundreds of pages or documents.

Example: Process 500 pages of reports and 200 interview transcripts.

STEP 12

Deduplicate and Consolidate Outcomes

What to do: Merge similar outcomes mentioned across multiple sources.

How to do it: Use AI to identify duplicate or similar outcome statements.

Example consolidation: “Women joined village council” (Interview) + “Five women elected to council” (Report) → Combined outcome with multiple evidence sources.

STEP 13

Extract Contribution Evidence

What to do: Identify how the program contributed to each outcome.

How to do it: Ask AI to extract statements about program contribution (e.g., “because of the training,” “as a result of the project”).

Example contribution evidence: “After attending the leadership workshop, women began speaking at meetings.”

STEP 14

Categorize and Prioritize Outcomes

What to do: Organize outcomes by type, domain, and significance.

How to do it: Generate summary tables of outcomes by category.

STEP 15

Verify Outcomes with Stakeholders

What to do: Share harvested outcomes with stakeholders for verification.

How to do it: Use AI to generate summary reports for stakeholder review.

Example verification process: Send outcome summary to program staff and beneficiaries to confirm accuracy.

STEP 16

Report Harvested Outcomes

What to do: Document harvested outcomes for evaluation and learning.

How to do it: Create an outcome report with categories, evidence, and contribution statements.

Example statement: “AI-assisted outcome harvesting of 50 program reports and 30 interviews identified 112 outcomes, including 47 behavioral changes, 31 relational changes, and 12 systemic changes. Of these, 38 outcomes were unexpected.”

Program Activity Mentioned

“The training taught us improved farming methods.”

Contribution Statement

“The training contributed to farmers adopting new methods.”

Alternative Factors

“Better rainfall also contributed to improved harvests.”

import openai
import pandas as pd
import json
client = openai.OpenAI(api_key="your-api-key")
prompt_template = """
You are an outcome harvesting assistant.
Extract all outcome statements from this text.
For each outcome, provide:
- outcome_statement: what changed
- outcome_type: behavioral/relational/attitudinal/capability/systemic/unintended
- evidence_quote: the exact text supporting this outcome
Return as JSON array of outcomes.
Text: {text}
"""
def extract_outcomes(text):
    response = client.chat.completions.create(
        model="your-selected-model",
        messages=[{"role": "user", "content": prompt_template.format(text=text)}],
        temperature=0.2
    )
    return json.loads(response.choices[0].message.content)
# Process document
with open("program_report.txt", "r") as f:
    text = f.read()
outcomes = extract_outcomes(text)
df = pd.DataFrame(outcomes)
df.to_csv("harvested_outcomes.csv", index=False)
print(f"Harvested {len(outcomes)} outcomes")