Case Overview

A small think tank focused on international development policy in Australia and the Indo-Pacific produces analysis, policy briefs, and strategic dialogues intended to inform decision-making across government and development actors.

The organization wanted to strengthen its approach to understanding whether its outputs contributed to changes in policy decisions, policy debates, or learning. Developing a Monitoring, Evaluation, and Learning Framework (MELF) was selected as the first step.

The case examines how GPT-4 was used as part of this design process. The purpose was not to delegate evaluative reasoning to AI, but to explore how generative AI could support the evaluator while identifying where human judgment remained essential.

1. Study Context

The case was undertaken as a formative design exercise by a solo evaluator. Approximately two days of work were invested over a period of six weeks.

The organization wanted the MELF to provide an overarching evaluative structure, establish standards, and develop a theory of change that could later underpin implementation.

The exercise was designed to test how ChatGPT-4 could support MELF development, explore methodological adaptations, and examine the evaluator competencies required for responsible integration of AI into evaluation design.

2. Why Use Generative AI?

Designing a MELF requires bringing together theoretical literature, organizational priorities, evaluative standards, and information about the organization’s work.

Generative AI offered potential advantages in several areas:

• Accelerating synthesis of diverse evidence.
• Generating initial framework prototypes.
• Processing large amounts of organizational text.
• Identifying patterns and recurring themes.
• Supporting alternative formulations of evaluation concepts.
• Providing material that could subsequently be challenged and refined by the evaluator.

The purpose was explicitly not to delegate evaluative reasoning. Instead, the exercise examined how AI might extend, and sometimes constrain, evaluator practice.

The case reports that the AI-assisted exercise took approximately two days, whereas a traditional MELF design process in this context was estimated at between five and ten days.

3. The Initial AI Prototype

The evaluator began by asking GPT-4 to create a baseline MELF with minimal prompting.

The resulting framework provided a generic structure but revealed weaknesses in evaluation criteria and contextual sensitivity.

• Vague evaluation criteria.
• Insufficient sensitivity to organizational context.
• Generic rather than organization-specific logic.

The initial draft was therefore treated as diagnostic rather than final. Its weaknesses became the basis for subsequent refinement.

4. The Five-Stage Design Process

Stage 1 — Initial Prototype Generation

GPT-4 generated a baseline MELF with minimal prompting. The draft exposed weaknesses in criteria and contextual sensitivity.

Stage 2 — Content Modeling of Past Outputs

More than 100 outputs from the think tank were analyzed, including website content, policy briefs, and analytical reports.

GenAI was used to identify dominant themes and patterns of engagement and to cluster them into preliminary evaluative domains.

Stage 3 — Literature Synthesis

GenAI supported a structured review of evaluation theory and think tank performance literature. Draft criteria and standards were developed drawing on Scriven’s logic of evaluation and realist principles.

Stage 4 — Realist-Informed Theory of Change

Prompts were used to structure potential Context–Mechanism–Outcome (CMO) pathways.

However, the initial AI-generated pathways tended toward linear and mechanistic models. One important assumption was that producing high-quality, evidence-based analysis would itself trigger policy uptake.

The evaluator subsequently challenged these assumptions using organizational feedback and documentation concerning policy impact, reflecting the contingent nature of policy contribution.

Stage 5 — Staff Feedback and Redrafting

Feedback was collected from nine staff members across analytical, engagement, and leadership functions through short written reflections and optional voice notes.

GPT-4 was used to identify common concerns and suggestions. Staff highlighted:

• The importance of shaping policy discourse alongside formal policy uptake.
• The importance of relationships and networks as conditions for impact.
• The need to account for informal and relational forms of influence.

Staff judgment was then used to moderate and reinterpret AI-generated themes where they risked over-generalization or insufficient contextualization.

5. A Key Problem: Frequency Is Not Significance

One important weakness was GenAI’s tendency to associate frequency with significance.

For example, “regional partnerships” appeared repeatedly and was therefore treated as a central theme. Less frequently mentioned issues, including “budgetary trade-offs” and “policy coordination failures,” received less emphasis even though they could be strategically significant.

The evaluator therefore had to reassert interpretive authority and determine whether frequently repeated themes represented meaningful contributions or simply reflected organizational habits.

Student reflection: Why might an AI system identify frequently mentioned topics more readily than strategically important but less frequently mentioned issues?

6. The Theory of Change Challenge

The initial AI-generated logic tended toward a simplified causal relationship:

The case identified this as an overly linear assumption.

Policy contribution may depend on context, relationships, networks, engagement, timing, and other conditions. The evaluator therefore used a realist-informed approach to interrogate potential Context–Mechanism–Outcome pathways.

Student Task

Context: What conditions might affect whether evidence contributes to change?

Mechanism: What might cause decision-makers, institutions, or policy communities to respond?

Outcome: What changes could occur beyond formal policy adoption?

Consider changes in policy discourse, awareness, relationships, policy options considered, evidence use, organizational learning, or eventual policy change.

7. What Human Participants Added

The staff feedback helped identify dimensions of impact that could be under-emphasized by a more mechanical analysis.

8. The Resulting MELF

The resulting MELF was approximately 3,500 words and contained six sections:

1. Context and scope
2. Theory of change
3. Evaluation questions and criteria
4. Indicators and data sources
5. Methods and analysis
6. Learning and use

The framework was coherent and included explicit criteria, evaluative domains, and a basic realist-informed theory of change.

However, the case concluded that the framework was not particularly fit for purpose. It lacked the contextual nuance and contextual fit required for a high-quality operational product.

9. What GenAI Did Well

• Accelerated synthesis.
• Generated initial drafts and prototypes.
• Processed large volumes of text.
• Surfaced patterns and themes.
• Supported alternative design options.
• Helped process staff feedback.
• Supported iterative interrogation of the framework.

10. What GenAI Did Poorly

• Produced vague outputs.
• Generated overly generic frameworks.
• Tended toward linear causal assumptions.
• Could give disproportionate weight to frequently occurring themes.
• Could over-generalize organizational patterns.
• Could under-emphasize informal and relational forms of impact.
• Did not independently produce a framework sufficiently fit for operational use.

11. Evaluator Competencies

The case identifies evaluator competencies as a key safeguard when GenAI is incorporated into MELF development.

Evaluative Reasoning

The ability to distinguish surface-level adequacy from substantive quality and to interrogate assumptions in AI-generated outputs.

Contextual Literacy

The ability to understand and incorporate the realities of the organization, policy environment, relationships, and networks.

Responsible AI Literacy

The ability to understand AI limitations, verify outputs, document decisions, and maintain transparency and accountability.

The case also highlights critical experimentation, including testing prompting approaches, approaches for eliciting human input, and the effectiveness of different GenAI tools in quality-assurance functions.

12. Student Exercise — Critique the AI

Imagine that you are the evaluator reviewing the initial AI-generated MELF.

Identify five weaknesses you would look for.

1. Are the evaluation criteria sufficiently specific?
2. Does the framework reflect the organization’s context?
3. Are the causal assumptions realistic?
4. Are important issues being identified, or simply frequently mentioned issues?
5. Are indicators and evidence requirements sufficiently clear for operational use?

13. Student Exercise — Human or AI?

For each activity, decide whether it should be primarily AI-supported, evaluator-led, or shared between AI and the evaluator.

Discussion: What should never be delegated entirely to generative AI in evaluation design?

14. Final Student Challenge

Prepare a one-page recommendation to the organization addressing:

1. What AI should be used for.
2. What AI should not be trusted to do.
3. How human review should be incorporated.
4. How stakeholder input should be incorporated.
5. How AI-generated assumptions should be tested.
6. How the final framework should be validated.
7. What safeguards should be established.

End your recommendation with one of these decisions:

Justify your decision using evidence and reasoning from the case.

15. Responsible AI and Future Evaluation Practice

The case considers the possibility that increasingly capable AI systems will become more agentic, with capabilities involving planning, memory, multimodal reasoning, and real-world action execution.

Such systems could automate substantial components of data gathering, coding, and synthesis. At the same time, they introduce concerns around verification, attribution, transparency, accountability, and opaque decision-making.

The case therefore emphasizes the importance of contestability: stakeholders should be able to question and challenge evaluation results rather than simply accept automated outputs.

As AI capabilities increase, evaluators may need stronger skills in interrogating AI outputs, identifying problematic causal reasoning, monitoring audit trails, and validating AI-supported processes.

16. Key Takeaway

The central lesson of the case is that the value of GenAI in evaluation design lies less in producing a ready-to-use framework and more in accelerating the process of surfacing weaknesses, testing alternative design logics, and interrogating assumptions.

The approach is particularly useful for early-stage tasks such as mapping evaluative domains, stress-testing theories of change, and identifying implicit assumptions. It is less suitable when evaluation purposes are poorly specified, institutional context is weakly articulated, or AI outputs are treated as authoritative rather than provisional.

Recommended Course

Source & Further Reading

Pertinent Sources from the Chapter

The following references are selected from the bibliography provided at the end of Fung’s chapter. They are particularly relevant to the themes explored in this case: generative AI in evaluation, evaluation design, AI ethics, realist evaluation, and the changing role of evaluators.