Original Story (Beneficiary)

“Before the program, I had no income. I couldn’t feed my children. After the training, I started a small business selling vegetables. Now I earn enough to pay school fees and have savings. My children are healthier. I feel like a different person. This changed everything for my family.”

AI MSC Scoring Output

• Significance Score: 9.5/10
• Change Domains: Livelihoods, Health, Empowerment
• Depth of Change: High (transformative)
• Reasoning: Story shows complete life transformation across multiple domains with clear before/after contrast.

STEP 1

Define MSC Domains and Questions

What to do: Define what types of change you want to capture.

How to do it: Create 3-8 domains of change (e.g., Livelihoods, Health, Empowerment) and a simple story collection question.

Example question: “Since participating in our program, what is the most significant change that has happened in your life?”

STEP 2

Design Story Collection for AI Compatibility

What to do: Structure story collection to maximize AI usability.

How to do it: Collect stories in plain text. Include metadata: storyteller ID, location, gender, date.

Example format: Each story as a separate row in a spreadsheet with columns: ID, Date, Location, Gender, Story Text.

STEP 3

Review a Sample of Stories Manually

What to do: Read 20-50 stories manually to understand the range of changes.

How to do it: As you read, note patterns in change types, language, and depth of impact.

Example observations: Many stories mention income increase. Empowerment stories are often from women. Some stories describe unexpected community-level changes.

STEP 4

Create a Gold-Standard MSC Scoring Sample

What to do: Manually score a representative sample of 50-100 stories.

How to do it: Have two team members independently score each story on significance (1-10). Resolve disagreements through discussion.

Example scoring criteria: 1-3 = Minor change, 4-6 = Moderate change, 7-8 = Significant change, 9-10 = Transformative change.

STEP 5

Define MSC Scoring Criteria for AI

What to do: Write clear criteria that AI can use to score stories.

How to do it: Define what makes a story more significant: depth of change, breadth of change (multiple domains), unexpectedness, number of people affected.

Example criteria: Significance score based on: (1) Depth: surface vs. life-changing, (2) Breadth: 1 domain vs. multiple domains, (3) Unexpectedness: expected vs. surprising.

STEP 6

Design the AI MSC Scoring Prompt

What to do: Write structured instructions for AI to score stories.

How to do it: Include scoring criteria, domain identification, and output format.

You are an MSC assistant for M&E.
Score this story on a scale of 1-10, where:
1-3 = Minor change (small, surface-level)
4-6 = Moderate change (noticeable improvement)
7-8 = Significant change (major improvement)
9-10 = Transformative (life-changing)
Also identify:
- Change domain(s): Livelihoods/Health/Education/Empowerment/Relationships/Resilience
- Depth of change: Low/Medium/High
- Unexpectedness: Expected/Unexpected
Return as JSON with: story_id, significance_score, domains, depth, unexpectedness, reasoning.
Story: {story}

STEP 7

Test AI on Your Gold-Standard Sample

What to do: Process your manually scored sample through AI.

How to do it: Compare AI significance scores to human scores.

Example comparison: Human score: 8, AI score: 7 (close). Human score: 9, AI score: 4 (disagreement to investigate).

STEP 8

Calculate MSC Scoring Agreement

What to do: Measure how well AI matches human scoring.

How to do it: Calculate correlation between human and AI scores. Identify systematic biases (e.g., AI consistently scores lower on empowerment stories).

Interpretation: Aim for correlation > 0.8. If correlation is low, review disagreements and refine criteria.

STEP 9

Refine Scoring Criteria Based on Errors

What to do: Review stories where AI and human scores diverge significantly.

How to do it: Identify what the human noticed that AI missed (or vice versa).

Example refinement: If AI misses “unexpected” changes, add “unexpectedly,” “surprisingly,” “never thought” to the criteria.

STEP 10

Scale to All Stories

What to do: Process all collected stories through AI.

How to do it: Use batch processing for hundreds or thousands of stories.

Example: Process 2,000 stories in batches of 50.

STEP 11

Generate MSC Story Rankings

What to do: Sort stories by AI significance score.

How to do it: Create a spreadsheet with stories sorted from highest to lowest score.

STEP 12

Extract Change Domains by Group

What to do: Analyze which change domains are most common across stories.

How to do it: Group stories by domain and count frequencies.

STEP 13

Identify Stories for Participatory Selection

What to do: Shortlist top stories for stakeholder selection.

How to do it: Select top 20-50 stories by AI score. Also include stories that represent each domain and diverse storytellers.

Example shortlist: Top 15 by score + 5 from each domain + 5 from underrepresented groups.

STEP 14

Generate MSC Summaries for Stakeholders

What to do: Create summaries of each shortlisted story.

How to do it: Use AI to generate concise summaries (2-3 sentences) of each story.

Example summary: “A woman who started a vegetable business after training now earns enough to send her children to school. She reports increased confidence and community respect.”

STEP 15

Facilitate Participatory Selection

What to do: Present shortlisted stories to stakeholders for final selection.

How to do it: Share summaries (not AI scores) with stakeholders. Ask them to select the 5-10 most significant stories. Discuss selections as a group.

Documentation: Record which stories were selected and why.

STEP 16

Report MSC Findings

What to do: Document and share the most significant changes.

How to do it: Create an MSC report with the selected stories, analysis of change domains, and aggregate statistics.

Example statement: “AI-assisted MSC analysis of 2,000 stories identified that 84% of beneficiaries reported livelihood improvements. Stakeholder selection of the top 20 stories highlighted empowerment and health as the most valued changes.”

AI Role

• Score all stories on significance
• Extract change domains
• Shortlist top stories
• Generate summaries
• Aggregate statistics

Human Role

• Select final most significant stories
• Discuss why changes matter
• Interpret stories in context
• Make program decisions based on findings
• Share stories with stakeholders

import openai
import pandas as pd
import json
client = openai.OpenAI(api_key="your-api-key")
prompt_template = """
Score this story on significance (1-10).
1-3: Minor change
4-6: Moderate change
7-8: Significant change
9-10: Transformative change
Also identify: domains (Livelihoods/Health/Education/Empowerment/Relationships/Resilience),
depth (low/medium/high), unexpectedness (expected/unexpected).
Return as JSON.
Story: {story}
"""
def score_story(story):
    response = client.chat.completions.create(
        model="your-selected-model",
        messages=[{"role": "user", "content": prompt_template.format(story=story)}],
        temperature=0.2
    )
    return json.loads(response.choices[0].message.content)
df = pd.read_csv("stories.csv")
results = []
for _, row in df.iterrows():
    result = score_story(row["story_text"])
    results.append({
        "id": row["story_id"],
        "significance": result["significance"],
        "domains": ", ".join(result["domains"]),
        "depth": result["depth"],
        "unexpectedness": result["unexpectedness"]
    })
output_df = pd.DataFrame(results)
output_df.to_csv("msc_scores.csv", index=False)
print(f"Scored {len(results)} stories")