From AI Answers to Systematic Analysis: Using QInsights in Evaluation

A practical case study on using AI to support qualitative evaluation analysis while keeping the evaluator in control.

The case in brief

Generative AI has made it much easier to work with large amounts of qualitative material. Evaluation reports, transcripts and project documents can be uploaded to an AI system and explored in a matter of seconds.

That is useful, but it also raises a methodological question: is an AI-generated answer the same thing as an analysis?

In evaluation work, the answer is not necessarily yes. Analysis involves decisions about what to examine, which cases to compare, what evidence is relevant, how much context needs to be preserved and whether an emerging interpretation actually holds across the material.

This case study describes how Dr. Susanne Fries used QInsights in a meta-analysis of 23 project evaluations. The approach was deliberately iterative: AI was used to explore and question the material, while the evaluator remained responsible for the analytical direction and interpretation.

The case study is based on the document From AI Answers to Systematic Analysis: Using QInsights in Evaluation by Dr. Susanne Fries.

Why the analysis environment matters

A general-purpose chatbot can produce a surprisingly sophisticated response to a large collection of evaluation reports. It can identify apparent themes, compare projects and suggest recommendations.

The problem is what can happen along the way. Several analytical decisions may be made at once, without the evaluator seeing how the model arrived at its overall interpretation.

For example, when 23 reports are analysed together, differences between projects can easily disappear. Two programmes may appear to have the same outcome but have arrived there through very different processes. A weakness identified in one context may also have a completely different meaning in another.

There is another risk. If the evaluator simply follows the themes and questions suggested by the AI, the process can gradually become AI-led rather than researcher-led.

This is why the surrounding analysis environment matters. The issue is not only whether AI can produce a good answer. It is whether the evaluator can remain actively involved in the analytical process.

The QInsights case: analysing 23 evaluations

For the Misereor meta-analysis, Dr. Fries began with the evaluation material itself rather than with AI.

1. Understand the material first

All 23 evaluation summaries were read before AI was introduced into the analysis. This provided an independent understanding of the projects, their aims and their contexts.

That first step was important because an evaluator needs enough familiarity with the material to judge whether an AI-generated interpretation is useful, superficial, distorted or simply wrong.

A project was then created in QInsights, with background information supplied to the AI assistant. Dr. Fries suggests thinking about this information in much the same way as briefing a research assistant who is joining the project at the analysis stage.

2. Start with open exploration

The next step was to look for an overarching topic across the projects. QInsights’ Topic Analysis was used for this initial exploration.

Importantly, the topics were not treated as final findings. They were used as starting points for further questions: What looks important? What is unexpected? Where are the tensions? What is missing?

Three topics appeared in approximately 70–90% of the reports:

• Deficits and learning needs in monitoring and proof of impact
• Gender equality and inclusion of marginalized groups
• Strengthening institutional and organisational capacities

These topics then became the basis for a more focused conversational analysis. :contentReference[oaicite:3]{index=3}

3. Develop analytical questions

Rather than asking AI simply to “analyse the evaluations”, Dr. Fries developed a set of questions to guide the analysis.

The questions explored whether projects and partner organisations were able to learn from experience, adapt to changing circumstances and develop their work independently.

They also examined:

• Monitoring and proof of effectiveness
• Gender justice and inclusion
• Institutional and organisational capacities
• Fragile states and crises
• The extent to which institutional strengthening was achieved

The analysis also asked for summaries containing supporting quotations, making the results easier to compare later. :contentReference[oaicite:4]{index=4}

4. Analyse individual cases

This was a particularly important methodological decision.

Instead of immediately putting all 23 evaluations into one analysis, the first detailed stage focused on 10 projects individually.

Working case by case helped preserve the logic and context of each project before moving towards cross-case conclusions.

When QInsights produced an interpretation, the evaluator could return to the supporting source material and ask whether the interpretation was actually justified.

For example, if AI suggested that a project had a strong adaptive learning culture, the evaluator could inspect the evidence and decide whether “adaptive learning” was really an appropriate interpretation or whether the evidence only demonstrated informal problem-solving.

5. Bring the cases together

After the individual analyses, the 10 conversations were brought together to look for similarities and differences.

Several recurring tensions emerged, including:

• Informal learning versus systematic documentation
• Professional strength versus organisational maturity
• Empowerment versus continuing dependency

One important conclusion was that learning and adaptability were strengths, particularly in dynamic and fragile contexts. At the same time, institutional learning remained limited where learning was not systematically reflected on, documented and transferred.

Another finding concerned institutional change. Projects could strengthen institutional skills and room for manoeuvre, but those gains could remain fragile without parallel investment in internal structures, knowledge management and financial diversification. :contentReference[oaicite:5]{index=5}

6. Identify broader patterns

The synthesis was used to develop a broader interpretation of how different projects approached learning, work and impact.

Three different logics were identified:

• Transformation-oriented: learning as a political and discursive process
• Service- and output-oriented: effect as a demonstrable result
• Mediation and accompaniment: effect through stabilisation and relationships

These categories were not simply accepted because the AI had suggested them. They became something to test against additional evidence.

7. Test the interpretation against the full dataset

The final stage returned to all 23 evaluations.

The synthesis document and the complete dataset were brought together, including the 13 reports that had not been part of the initial case-by-case analysis.

The purpose was to ask a simple but important question: Does the interpretation still hold when it is tested against the wider dataset?

The evaluator could then examine whether the previously identified patterns appeared in the additional projects, whether new patterns emerged, or whether the original interpretation needed to be changed.

This final step is important because a coherent AI-generated narrative can be persuasive. Testing it against additional evidence creates an opportunity to challenge that narrative rather than simply accepting it. :contentReference[oaicite:6]{index=6}

Beyond this particular workflow

The seven stages above should not be understood as a universal QInsights workflow. Different evaluation questions and methodologies require different analytical approaches.

The original case study points to several possibilities, including query-based analysis, narrative-integrated thematic analysis, grounded theory and Outcome Harvesting. The important principle is that the methodology should determine the analytical logic; the software should support that logic rather than prescribe it. :contentReference[oaicite:7]{index=7}

Working with context and metadata

One useful feature described in the case study is the ability to define characteristics of documents and respondents as part of the project’s Data Context.

This matters because evaluation datasets often contain meaningful groups: programme staff, implementing partners, beneficiaries, government actors, countries, regions or different intervention types.

Instead of repeatedly explaining these characteristics in individual prompts, they can be structured as part of the analysis. This allows researchers to deliberately compare groups or create subsets of the data.

For example, an evaluator might compare implementing partners with programme staff, examine one region separately, or explore whether perspectives differ according to participant characteristics.

The important point is that these analytical groupings remain researcher-led rather than being inferred and imposed by the AI. :contentReference[oaicite:8]{index=8}

Other ways to use QInsights

The case study focuses mainly on Topic Analysis and Conversational Analysis, but the document also describes other QInsights options.

• Grid Analysis for structured cross-case comparisons using consistent questions.
• Guided Analysis for a more focused, step-by-step exploration of a selected topic.
• Sentiment Analysis for semi-structured data such as open-ended survey responses, stakeholder feedback and social media comments.

Grid Analysis, for example, can provide an answer for each respondent or case, with access to supporting quotations and follow-up conversations. :contentReference[oaicite:9]{index=9}

What QInsights does not replace

It is tempting to think that a specialised AI tool solves the methodological problems associated with generative AI. It does not.

QInsights still uses large language models, and their outputs remain probabilistic. The software cannot decide which evaluation question is important, whether a finding is significant in its programme context, or whether an apparent weakness should be interpreted as a failure.

Those decisions still belong to the evaluator.

What a purpose-built environment can do is make some parts of the process easier: controlling the material being analysed, preserving cases and contextual distinctions, organising different analytical strands, inspecting evidence, comparing groups, revisiting earlier analyses and following up on emerging questions. :contentReference[oaicite:10]{index=10}

The main lesson

The experience described by Dr. Fries offers a useful way of thinking about AI in qualitative evaluation.

AI can make it much faster to work through large amounts of material. But speed should not be the only measure of success. The more important question is whether AI reduces the mechanical workload while preserving the evaluator’s ability to think critically about the evidence.

In this case, the evaluator did not simply ask AI for themes and recommendations. The analysis moved back and forth between cases, questions, evidence, interpretation and validation.

That is the idea behind what Dr. Fries describes as hybrid intelligence: the analytical process develops through the interaction between researcher and AI rather than through a simple division of tasks between the two. :contentReference[oaicite:11]{index=11}

For evaluators beginning to work with AI, this may be one of the most useful lessons of all: AI does not have to replace analytical thinking to be valuable. It can give evaluators more capacity to do the thinking that matters.