Can AI Fill the Gaps in Evaluation Evidence? A Practical M&E Test

When evaluation evidence is incomplete, AI can produce a plausible explanation that quietly fills the gaps. This practical test helps evaluators distinguish what the evidence actually says from what AI is inferring, assuming or inventing.

From weak evidence to missing evidence

The previous tutorial examined whether AI can make a limited evidence base appear stronger than it really is:

Can AI Sound More Certain Than the Evidence? A Practical M&E Test

That tutorial focused on the strength of the language used in an evaluation conclusion.

This tutorial goes one step earlier:
What happens when important evidence is missing?

AI systems are designed to produce coherent responses. In an evaluation context, that can create a problem when the available evidence does not contain enough information to answer the question being asked.

The model may produce a plausible explanation that sounds like an evidence-based finding even though part of the explanation comes from inference or assumption rather than the evaluation data.

A simple M&E example

Imagine a fictional youth employment programme:

• 70% of surveyed participants reported improved employment prospects.
• Programme staff said mentoring was important.
• There is no information about programme dropouts.
• There is no comparison group.
• The evaluation does not measure the amount or quality of mentoring received.

An AI system might produce:

The problem is not necessarily a false number. The problem is the unsupported bridge between the available evidence and the explanation.

The evidence tells us that participants reported improved prospects and that staff considered mentoring important. It does not, by itself, establish that mentoring caused the improvement or that it worked through the mechanisms described by the AI.

Evidence, inference or assumption?

When evidence is incomplete, separate three things:

The quality problem occurs when AI presents all three in the same confident voice.

If a material statement cannot be traced to the evidence, it should be identified as an inference, hypothesis or unsupported claim rather than presented as an established finding.

Step 1: Define the evidence boundary

Before asking AI to explain an evaluation finding, establish what the evaluation actually contains.

• Known: What is directly supported?
• Unknown: What information is missing?
• Unresolved: What explanations remain possible?

For the example:

This creates an explicit boundary that AI should not cross without additional evidence.

Step 2: Run the “Where Did This Come From?” test

Break the AI-generated conclusion into individual claims and classify each one.

• Source-supported — directly supported by the evidence.
• Inference — a reasoned interpretation of the evidence.
• Hypothesis — a possible explanation requiring further investigation.
• Unsupported — no adequate evidence identified.

For example:

Step 3: Look for AI-generated “bridges”

Some of the most difficult AI errors are not obvious false facts. They are statements that connect two pieces of evidence without sufficient support.

The bridge may be plausible, but plausibility is not evidence.

The same problem can occur with explanations of mechanisms, theories of change, implementation factors and causal pathways.

Step 4: Ask AI what it cannot know

A useful prompt is:

This is more useful than simply asking AI:
“What explains the finding?”

Step 5: Run the Missing-Evidence Test

1. What evidence would we expect if this explanation were correct?
2. Do we actually have that evidence?
3. What relevant information is missing?
4. Could the conclusion change if that information became available?
5. Should the gap become a limitation, additional data-collection priority or research question?

This turns “we don’t know” into an explicit and useful evaluation result rather than allowing AI to fill the gap.

Step 6: Do not confuse missing evidence with evidence of no effect

AI should not silently convert the first situation into the second.

For example:

is not necessarily equivalent to:

The second statement makes a substantive evaluation claim. The first may simply indicate that the relevant evidence was unavailable or not examined.

Step 7: Turn evidence gaps into explicit limitations

A missing piece of evidence does not automatically make an evaluation unusable. But important gaps should remain visible.

Instead of allowing AI to silently fill the gap, write it explicitly:

This preserves the boundary between what is known and what remains unresolved.

Step 8: Ask what additional evidence would resolve the gap

The next question should often be:
“What evidence would allow us to answer this properly?”

Depending on the evaluation question, this could involve:

• additional interviews;
• administrative data;
• follow-up measurement;
• comparison data;
• subgroup analysis;
• observation;
• document review;
• additional qualitative inquiry.

The appropriate response depends on the evaluation question, design and context. AI can help identify options, but the evaluator should determine what additional evidence is methodologically appropriate.

Implementation: build an Evidence-Gap Check into your workflow

AI Synthesis → Claim Extraction → Evidence Trace → Gap Detection → Inference Check → Human Review

1. Extract the claims

Ask AI to list every substantive claim in its proposed finding.

2. Trace each claim

Require a source, dataset, interview, document or analysis for every material claim.

3. Mark unsupported bridges

Highlight statements that connect pieces of evidence without adequate support.

4. Identify missing information

Ask:

5. Decide what to do with the gap

The evaluator can decide whether the gap should be:

• reported as a limitation;
• addressed through additional analysis;
• addressed through additional data collection;
• treated as an unresolved question;
• or left outside the evaluation scope.

6. Preserve the boundary

The final AI-assisted narrative should not become more complete than the evidence base itself.

A 10-step Evidence-Gap Test

1. Identify the AI-generated conclusion.
2. Break it into individual claims.
3. Trace each claim to its supporting evidence.
4. Separate evidence from inference and hypothesis.
5. Identify missing data and perspectives.
6. Identify unsupported connections between evidence.
7. Distinguish missing evidence from evidence of no effect.
8. Identify what additional evidence could resolve important gaps.
9. Revise the conclusion to make unresolved issues visible.
10. Document the human verification decision.

What AI can help with

• Identifying claims that require evidence.
• Tracing statements back to supplied sources.
• Finding missing information.
• Separating observations from interpretations.
• Generating alternative explanations.
• Identifying questions requiring additional evidence.
• Stress-testing evaluation conclusions.

What AI should not silently do

• Assume missing participants had the same experience as respondents.
• Assume an unmeasured mechanism caused an observed outcome.
• Treat stakeholder opinion as independently verified evidence.
• Convert an absence of data into evidence that something did not happen.
• Invent details that make an evaluation narrative more complete.
• Resolve an unanswered question simply because a plausible answer can be generated.

NIST describes confabulation as a GenAI phenomenon in which systems can generate and confidently present erroneous or false content. NIST also notes that generated reasoning or citations can create additional opportunities for inappropriate reliance. :contentReference[oaicite:3]{index=3}

A useful distinction for evaluators

Sometimes the most defensible evaluation statement is:

That is not a failure of evaluation. It is a transparent statement about the boundary of the available evidence.

For impact questions, this distinction is particularly important because causal conclusions require appropriate designs and counterfactual reasoning. The World Bank’s Impact Evaluation in Practice explains that simply observing an outcome change is not sufficient to establish that a programme caused it. :contentReference[oaicite:4]{index=4}

Why this matters for AI-assisted evaluation

NIST’s Generative AI Profile is designed to help organizations incorporate trustworthiness considerations into the design, development, use and evaluation of generative AI systems. :contentReference[oaicite:5]{index=5}

For M&E, the practical implication is straightforward: an AI-assisted workflow should make important uncertainty and evidence limitations visible rather than hiding them behind a complete-sounding narrative.

UNDP’s revised Evaluation Guidelines 2026 became effective on September 30, 2026. UNDP states that the revision includes strengthened guidance on the responsible and transparent use of artificial intelligence in evaluation activities. :contentReference[oaicite:6]{index=6}

This reinforces the importance of documenting how AI is used while keeping methodological and professional judgment with the evaluator.

Connect This Tutorial With the Previous Series

This tutorial is the ninth part of the EvalCommunity AI + M&E evidence-quality series.

• 1. Accuracy Paradox of AI: What It Means and How to Fix It
  Why apparently accurate AI output can still create problems of trust, context and interpretation.
• 2. How to Verify an AI-Generated Evaluation Finding: A 10-Step Tutorial
  How to trace an AI-generated finding back to the underlying evidence.
• 3. Can AI Be Accurate and Still Mislead an Evaluator? An M&E Experiment
  How accurate information can become incomplete, overconfident or misleading when context is lost.
• 4. When AI Summarizes Evaluation Evidence, What Gets Left Out?
  How omission, lost context and missing nuance can change the meaning of evaluation evidence.
• 5. Can AI Create an Illusion of Consensus in Evaluation?
  How AI-generated synthesis can make stakeholder views appear more unified while disagreement and minority perspectives become less visible.
• 6. Can AI Turn Correlation Into Causation? A Practical M&E Test
  How to detect when AI moves from observed change or association to a stronger causal interpretation.
• 7. Can AI Sound More Certain Than the Evidence? A Practical M&E Test
  How to detect and correct certainty inflation when AI-generated evaluation language is stronger than the evidence.
• 8. Can AI Make Weak Evidence Look Strong? A Practical M&E Test
  How to test whether AI synthesis makes a limited evidence base appear stronger than it actually is.
• 9. Can AI Fill the Gaps in Evaluation Evidence? A Practical M&E Test
  This tutorial: how to detect when AI crosses the boundary between evidence, inference, assumption and unsupported explanation.

The Evidence-Quality Workflow

Accuracy → Verification → Context → Completeness → Disagreement → Causality → Certainty → Evidence Strength → Evidence Gaps → Human Judgment

• Accuracy: Is the information correct?
• Verification: Can the claim be traced to its source?
• Context: Has important context been preserved?
• Completeness: What has been omitted?
• Disagreement: Have different perspectives been preserved?
• Causality: Is AI claiming more than the evaluation design supports?
• Certainty: Is the language stronger than the evidence?
• Evidence strength: Is the underlying evidence sufficiently strong?
• Evidence gaps: Is AI filling information that the evaluation does not contain?
• Human judgment: Can the evaluator defend the final conclusion?

Related EvalCommunity Resources

• Use of AI in Evaluation
• How GenAI Works for Evaluators
• Evaluate AI Tools
• AI Tool Evaluation Protocol for M&E
• AI Evidence-Synthesis Agent for Evaluation
• How to Avoid Fake Citations When Using AI
• Responsible AI Field Manual for M&E
• AI-Assisted Consultation Documentation
• AI Prompt That Finds Its Own Mistakes
• Principles for AI Use in Monitoring & Evaluation

Authoritative External Sources

• NIST — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
  
• NIST — AI Measurement and Evaluation
  
• NIST — ARIA Evaluation Planning Manual, 2026
  
• UNDP — Evaluation Guidelines 2026
  
• UNDP Independent Evaluation Office — Impact Evaluation Guidelines, 2026
  
• World Bank — Impact Evaluation in Practice, Second Edition
  

The Practical Rule

AI can help evaluators organize evidence, identify patterns, compare sources, detect gaps and formulate hypotheses.

But it should not silently turn missing information into a plausible fact.

Sometimes the most defensible evaluation statement is simply:
“The available evidence is insufficient to determine this.”

The evaluator remains responsible for keeping the boundary between evidence, inference and assumption visible.

Note: The numerical examples in this tutorial are illustrative. They demonstrate a quality-assurance method and are not empirical evaluation results.