Can AI Create an Illusion of Consensus in Evaluation?

A practical M&E experiment on stakeholder disagreement, minority perspectives and AI-generated synthesis.

When AI summarizes interviews, consultations or open-ended survey responses, it can make complex evidence easier to understand.

But there is another question evaluators should ask:

Does the AI synthesis make stakeholders appear to agree more than the original evidence shows?

This is different from asking whether the AI invented information.

An AI summary can accurately describe the dominant theme while giving less visibility to disagreement, exceptions, minority experiences or competing interpretations.

In evaluation, that matters because disagreement can itself be evidence.

Core question: Can an AI-generated synthesis be factually defensible while creating an overly unified picture of stakeholder views?

1. What Is an Illusion of Consensus?

An illusion of consensus occurs when a synthesis makes a set of views appear more unified or consistent than the underlying evidence supports.

It does not necessarily require fabricated information.

For example, an AI system may correctly identify that a majority of interview participants described a programme positively. But if the summary gives little visibility to participants who reported serious problems, the final reader may come away with a stronger impression of agreement than the original evidence warrants.

The underlying research on the Accuracy Paradox identifies illusion of consensus among broader hallucination and sociotechnical risks that can be missed when AI is evaluated only through narrow factual-accuracy measures. :contentReference[oaicite:2]{index=2}

The research also argues that AI outputs can be accurate in individual elements while becoming misleading when context and relationships between those elements are lost. :contentReference[oaicite:3]{index=3}

Read:
Accuracy Paradox of AI: What It Means and How to Fix It.

2. A Practical M&E Experiment

Consider a fictional evaluation of a community health programme.

For this teaching experiment, imagine that an evaluation team has interviewed four stakeholder groups:

• Community members: 20 interviews
• Frontline workers: 10 interviews
• Programme managers: 5 interviews
• Local partners: 5 interviews

Important: These numbers are illustrative for the experiment. They are not a population estimate and should not be interpreted as evidence that one stakeholder group represents the views of another.

Suppose the evidence contains the following perspectives:

Now ask an AI system:

Prompt 1: “Summarize the main themes emerging from these stakeholder interviews.”

A possible output might be:

“Stakeholders generally reported that the programme improved access to services, strengthened service delivery and increased collaboration among local actors.”

For the purposes of this teaching experiment, assume that the sentence is broadly supported by the evidence.

Now compare the summary with the original material.

3. Where Did the Disagreement Go?

The AI summary identifies the dominant positive themes.

But several important qualifications are less visible:

• Some remote households still experienced access problems.
• Frontline workers reported increased workload.
• Staffing constraints affected implementation.
• Some implementation targets required adaptation.
• Partners differed in their assessment of local ownership.

The issue is not necessarily that the AI has fabricated a statement.

The issue is that the synthesis may have compressed meaningful variation into a smoother narrative.

A reader who sees only the summary could reasonably infer greater agreement than exists in the underlying evidence.

4. Why Consensus Needs to Be Tested

Consensus is not a problem when the evidence genuinely shows consensus.

The problem is assuming consensus because an AI-generated summary sounds unified.

In evaluation, stakeholder differences may reveal:

• different experiences of programme implementation;
• differences in access or exposure;
• implementation constraints;
• geographic variation;
• different incentives or roles;
• unintended effects; or
• alternative explanations for the observed result.

Therefore, the evaluator should not ask only:

“What is the main theme?”

The evaluator should also ask:

“Who expressed it, who disagreed, and does the difference matter?”

5. A Second Prompt: Ask AI to Find Disagreement

Now repeat the analysis using a more demanding prompt.

Prompt 2: “Analyze these stakeholder interviews. Identify areas of agreement, disagreement, contradiction and uncertainty. Separate dominant views from minority views. Identify which stakeholder groups expressed each view. Do not assume that a less frequent view is unimportant. Identify any minority or contradictory perspective that could materially change the interpretation of the main finding. Do not invent evidence.”

This changes the analytical task.

Instead of asking only:
“What is the main theme?”

the evaluator is asking:
“Where do stakeholders agree, where do they disagree, and what might those differences tell us?”

6. Compare the Two Approaches

7. The “Who Disagrees?” Test

Whenever AI produces a smooth stakeholder synthesis, ask:

1. Who agrees with the dominant finding?
2. Who disagrees?
3. Which stakeholder groups report different experiences?
4. Are minority and negative cases visible?
5. Are contradictory statements being examined or simply compressed?
6. Could a disagreement materially change the evaluation interpretation?

Evaluation rule: A minority view should not automatically become the conclusion. But it should not disappear merely because it is less frequent.

8. Frequency Is Not the Same as Importance

A frequent theme is not automatically the most important finding.

Conversely, a less frequent view is not automatically more important simply because it is a minority view.

The evaluator needs to consider both frequency and significance.

A less frequent issue may deserve closer examination when it concerns:

• a potentially important unintended effect;
• a specific population experiencing a different outcome;
• a serious implementation problem;
• an important ethical or safeguarding concern;
• a significant geographic or contextual difference; or
• evidence that challenges the dominant interpretation.

The correct response is therefore not to give every opinion equal weight.

It is to make important variation visible before deciding what the evidence means.

9. Why Stakeholder Groups Matter

A single overall theme can conceal differences between groups.

For example, a programme may be described positively overall while experiences differ by:

• location;
• age;
• gender or other relevant demographic characteristics;
• disability status;
• programme participation or exposure;
• stakeholder role; or
• implementation site.

The relevant dimensions depend on the evaluation question and the evidence available.

If those differences disappear during AI-assisted synthesis, the evaluation may lose information that is important for understanding implementation, equity or outcomes.

EvalCommunity’s Responsible AI Field Manual specifically recommends checking whether minority, dissenting and marginalized perspectives remain visible and whether AI has flattened important contextual differences.

See:
Responsible AI Field Manual for Monitoring & Evaluation.

10. AI Should Not Decide What Counts as Important

AI can help identify patterns across large volumes of qualitative data.

But identifying a pattern and deciding its evaluative importance are different tasks.

For example, AI might report:

“Most participants described the programme positively.”

The evaluator still needs to ask:

• Who expressed this view?
• Which stakeholder groups are represented?
• Who was not represented?
• What did negative cases report?
• Are the experiences different by context or group?
• Could the minority evidence change the interpretation?

EvalCommunity’s guidance on using AI in evaluation emphasizes that AI can assist with qualitative analysis and stakeholder feedback analysis, but evaluators remain responsible for verifying outputs, identifying bias, interpreting evidence and ensuring that findings are appropriate for the evaluation context.

See:
How to Use AI in Evaluation Practice.

11. A Reproducible M&E Experiment

You can test this problem using your own authorized interview or consultation data.

1. Select qualitative evidence. Use interviews, focus groups, consultation notes or open-ended survey responses.
2. Run a conventional AI synthesis. Ask for the main themes.
3. Save the exact output.
4. Run a disagreement-focused analysis. Ask for agreement, disagreement, minority views, negative cases and contradictions.
5. Compare the outputs. Record which perspectives became more or less visible.
6. Return to the original evidence. Check whether the AI accurately represented each perspective.
7. Check relevant subgroup differences. Use dimensions appropriate to the evaluation question and data.
8. Assess significance. Determine whether any less frequent view could materially affect interpretation.
9. Document AI use. Record prompts, outputs, review steps and decisions.
10. Make the final interpretation. The evaluator decides how the evidence should be interpreted.

This approach is consistent with EvalCommunity’s AI-assisted consultation framework, which recommends examining accuracy, coverage, nuance, dissent, traceability, human review and influence.

See:
AI-Assisted Consultation Documentation.

12. Add a “Disagreement Layer” to AI Analysis

A practical workflow can separate theme extraction from disagreement analysis:

AI Theme Extraction → Disagreement Check → Evidence Verification → Human Interpretation

The first stage identifies recurring patterns.

The second deliberately searches for information that the first stage may have made less visible.

The third returns to the source evidence.

The final interpretation remains with the evaluator.

This is consistent with the broader principle behind structured AI evaluation: testing should examine more than whether an output looks plausible. NIST’s ARIA approach combines model testing, red teaming and user testing and is designed to examine technical and contextual robustness. :contentReference[oaicite:4]{index=4}

13. A Practical Prompt for Evaluators

Prompt:

“You are assisting with qualitative evaluation analysis. Do not assume that the most frequent view is the most important view.

Identify:

1. dominant themes;

2. minority themes;

3. contradictory evidence;

4. negative cases;

5. differences between stakeholder groups;

6. evidence that challenges the dominant interpretation;

7. important contextual differences; and

8. issues that could materially change the evaluation finding.

For each important point, identify the supporting source, passage or evidence unit. Do not invent evidence. Distinguish what participants said from your interpretation.”

The purpose of this prompt is not to make AI automatically produce a “balanced” answer.

The purpose is to make disagreement visible enough for an evaluator to examine it.

14. Keep the Evidence Traceable

Every important AI-generated theme should remain connected to the underlying evidence.

Stakeholder Evidence → AI Synthesis → Disagreement Check → Human Review → Evaluation Finding

If an evaluator cannot identify where an AI-generated theme came from, the synthesis should not automatically be treated as evidence.

EvalCommunity’s AI Evidence-Synthesis Agent guidance similarly emphasizes registered sources, traceable evidence, preserved contradictions and limitations, and human review before findings become conclusions.

See:
AI Evidence-Synthesis Agent for Evaluation.

15. The Five-Question Consensus Test

Before accepting an AI-generated stakeholder synthesis, ask:

1. Does the synthesis show where stakeholders agree?
2. Does it show where they disagree?
3. Are minority and negative cases visible?
4. Are relevant differences between stakeholder groups preserved?
5. Could an omitted disagreement change the evaluation interpretation?

If an AI synthesis looks unusually smooth, ask where the variation went.

16. Check the Sources, Not Just the Summary

AI-generated qualitative synthesis should remain traceable to the original evidence.

Do not rely on AI-generated quotations, references or claims without verification.

EvalCommunity’s citation-verification guidance recommends checking the original source, verifying the claim and reviewing its context before using an AI-generated source, quotation or evidence claim.

See:
Avoid Fake Citations When Using AI.

17. What the Experiment Does—and Does Not—Show

This experiment does not show that AI always creates an illusion of consensus.

It shows why consensus should be tested against the underlying evidence when AI is used to synthesize diverse stakeholder perspectives.

AI may correctly identify a dominant pattern.

The evaluator still needs to determine whether that pattern adequately represents the evidence and whether important differences have been preserved.

The practical distinction is:

• Dominant theme: What pattern appears repeatedly?
• Distribution: Which stakeholder groups expressed it?
• Variation: How did experiences differ?
• Dissent: What evidence challenges the dominant view?
• Significance: Could the less frequent evidence materially affect interpretation?

18. What NIST Adds to the Discussion

NIST’s AI measurement and evaluation programme emphasizes that different AI characteristics require different measurements and that context is crucial when deciding how AI should be evaluated. :contentReference[oaicite:5]{index=5}

NIST’s ARIA approach goes beyond system performance alone and combines model testing, red teaming and user testing to examine technical and contextual robustness. :contentReference[oaicite:6]{index=6}

For M&E professionals, the implication is practical: evaluating an AI-assisted qualitative workflow should not stop at asking whether individual statements are accurate. The evaluator should also examine what information becomes visible, what becomes less visible, and whether the synthesis changes how the underlying evidence is understood.

19. Practical Rule for M&E Professionals

Do not confuse a smooth synthesis with stakeholder agreement.

When AI summarizes interviews, consultations or open-ended responses:

• identify the dominant pattern;
• identify who expressed it;
• actively search for disagreement;
• preserve relevant minority and negative cases;
• check differences between stakeholder groups;
• trace important claims to original evidence; and
• keep final interpretation with the evaluator.

The goal is not to give every opinion equal weight.

The goal is to ensure that important disagreement is visible before the evaluator decides what the evidence means.

20. Continue the Series

This article is part of the EvalCommunity series on AI accuracy, verification and evidence quality in M&E.

• Article 1: Accuracy Paradox of AI: What It Means and How to Fix It
• Article 2: How to Verify an AI-Generated Evaluation Finding: A 10-Step Tutorial
• Article 3: Can AI Be Accurate and Still Mislead an Evaluator? An M&E Experiment
• Article 4: When AI Summarizes Evaluation Evidence, What Gets Left Out?
• Article 5: Can AI Create an Illusion of Consensus in Evaluation?

Related EvalCommunity Resources

• AI-Assisted Consultation Documentation
• How GenAI Works for Evaluators
• How to Use AI in Evaluation Practice
• Responsible AI Field Manual for Monitoring & Evaluation
• AI Evidence-Synthesis Agent for Evaluation
• Avoid Fake Citations When Using AI
• How to Evaluate AI Tools for Monitoring & Evaluation
• An AI Prompt That Finds Its Own Mistakes

Authoritative External Sources

• NIST — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
  
• NIST — AI Measurement and Evaluation
  
• NIST — ARIA Evaluation Planning Manual: Elements of ARIA-Style AI Evaluations
  
• Accuracy Paradox: Addressing Epistemic, Manipulative, and Societal Risks of Hallucination in AI Governance
  
• WHO — Ethics and Governance of Artificial Intelligence for Health: Guidance on Large Multi-Modal Models
  

Key Takeaway

AI can make disagreement less visible without inventing a single fact.

For M&E professionals, the question is therefore not only:
“Is the AI summary accurate?”

It is also:

“Does the synthesis accurately represent both agreement and disagreement in the evidence?”

A strong AI-assisted evaluation workflow does not simply identify what appears most frequently.

It also makes it possible to see who disagreed, how their experiences differed, and whether those differences matter for the evaluation.

Editorial note: The community-health programme scenario in this article is a teaching example created to demonstrate the method. It is not presented as an empirical evaluation study. The prompts, tests and workflow described as EvalCommunity adaptations are educational applications and are not presented as methodologies reported by the external sources.