An AI-generated evaluation report can be accurate sentence by sentence and still be incomplete.

That is one of the most important risks to understand when using AI for qualitative analysis, interviews, document review, and evaluation reporting.

In the first two tutorials in this series, we looked at whether AI can produce an accurate evaluation report and how to verify whether reported claims are supported by evidence.

This tutorial examines the other direction:

This is the problem of recall, omission, and completeness.

Learning Objectives

• Understand precision versus recall in AI-assisted evaluation.
• Recognise why report-to-source checking can miss omissions.
• Design a source-to-finding completeness check.
• Use explicit coding to make omissions easier to detect.
• Understand sparse and dense qualitative coding tasks.
• Use sampling appropriately as a quality-assurance tool.
• Apply recall-oriented AI prompts.
• Build a two-direction verification workflow.

1. The Problem With a “Correct” AI Report

Imagine an evaluator has 24 interview transcripts from farmers participating in a training programme.

The evaluation question is:

The AI produces this finding:

A checker opens the transcripts for F3 and F7 and confirms that both support the statement.

The report passes.

But what if F9 and F15 also made the same claim?

The checker may never open those transcripts because the report never mentioned them.

The report can therefore be precise but incomplete.

2. Precision and Recall Ask Different Questions

Precision asks:

Recall asks:

3. Why Omissions Are Easy to Miss

Checking a reported claim is relatively straightforward.

If the report says:

“F3 credits the training with increasing harvests.”

the checker can search for F3 and verify the evidence.

But if the report says nothing about F9, there is no F9 claim to verify.

This creates a structural problem:

Research on summarisation has treated omission as a distinct quality problem. For example, research published in the ACL Anthology examines omission in dialogue summarisation, while AbsenceBench examines whether language models can identify information that is missing.

The practical lesson for evaluation is simple:

Completeness needs its own quality-assurance process.

4. The 24-Farmer Example

Suppose four farmers make the relevant claim:

The report says:

“Two farmers credit the training for their bigger harvest: F3 and F7.”

F3 and F7 are correctly reported. But F9 and F15 are missing.

So the report has:

• Good precision: the reported cases are supported.
• Incomplete recall: relevant cases were omitted.

This is why “Only two of 24 farmers…” requires much stronger checking than “Some farmers…”.

5. Stronger Claims Create a Stronger Evidence Burden

Words such as only, nobody, all, none, exactly, every and majority should trigger additional scrutiny.

6. Reverse the Direction of Checking

If report-to-source checking is insufficient for recall, reverse the direction.

Instead of asking:

“Does F3 support the report?”

ask:

“What findings does F3 contribute to?”

Then repeat the process for F7, F9, F15 and the other sources.

This makes omissions easier to identify before they disappear into the final narrative.

7. Make Recall Measurable With Explicit Coding

One of the strongest approaches described in the source material is to represent the underlying evidence explicitly instead of asking AI to freestyle the final finding.

For example:

The important point is not the exact format.

The important point is that the finding can be traced back to an explicit evidence record.

If a report says “four farmers,” the evaluator can identify those four sources.

If the report says “one farmer disagreed,” that source can also be identified.

8. Code Supporting, Contradictory and Qualifying Evidence

Recall is not only about finding more positive cases.

A report can also become misleading when contradictory or qualifying evidence disappears.

For each finding, consider:

• Supporting evidence — consistent with the finding.
• Contradictory evidence — points in another direction.
• Qualifying evidence — narrows or conditions the finding.
• No relevant evidence — the source does not address the issue.

This matters particularly when AI turns diverse interview responses into a smooth narrative.

The PCAOB AS 1105: Audit Evidence provides an authoritative example from auditing of the importance of evaluating evidence in relation to assertions. The auditing context is different from evaluation, but the evidence-quality principle is useful by analogy.

9. Sparse and Dense Coding Tasks

Not every qualitative question has the same coding burden.

A sparse task contains relatively few relevant passages within a large amount of text.

Example:

“Which farmers mention drainage as a factor affecting harvests?”

A dense task contains many potentially relevant pieces of evidence.

Example:

“What causal explanations do participants give for changes in agricultural outcomes?”

A task that appears cheap because it has few obvious hits may actually be risky because much of the source material is being passed over.

10. Why Smaller, Auditable Units Can Help

Giving a model an entire corpus and asking for every relevant finding does not guarantee complete extraction.

The source material describes a practical comparison in which processing a very large document as one context produced substantially fewer extracted causal links than processing the material in smaller chunks.

The broader lesson is not that one particular chunk size is always correct.

It is that smaller, auditable units can make omissions easier to detect.

For example, divide a transcript into numbered segments and require the extraction process to address each segment.

If relevant evidence exists, record it.

If no relevant evidence exists, record “none identified.”

This creates an audit trail and makes silent skipping harder.

11. “None Identified” Is Not “Not Checked”

These should never be treated as equivalent.

A transparent AI-assisted workflow should record what was actually read, coded, sampled, or excluded.

The OECD Quality Standards for Development Evaluation provide guidance on evaluation quality across purpose, design, implementation, reporting, and use, including attention to information sources and methodological quality.

12. Use Sampling as a Quality Check

You do not necessarily need to manually recode every source twice.

A practical quality-assurance approach is to select a sample of source material and independently check whether relevant evidence was captured.

1. Select sources independently.
2. Read them without relying on the AI-generated findings.
3. Identify evidence relevant to the evaluation question.
4. Compare your coding with the AI’s coding.
5. Investigate disagreements.
6. Look for systematic patterns of omission.

Sampling can reveal weaknesses in an extraction process, but a small quality-assurance sample should not automatically be presented as a statistically valid estimate of overall recall.

The appropriate design depends on the evaluation objective, population, sampling approach, and desired level of inference.

The PCAOB AS 2315: Audit Sampling illustrates this broader principle in auditing: sampling design, sample size, and evaluation require an explicit objective and professional judgement.

13. Causal Questions Are Particularly Interesting

Many evaluation questions ask about causality or contribution:

• What contributed to improved outcomes?
• Why did some participants experience better results?
• What explains differences between groups?
• What changes did participants attribute to an intervention?

These questions can produce rich evidence containing both supporting and contradictory accounts.

For example:

A simple summary can flatten these differences.

Explicit causal coding can preserve them.

14. Causal Mapping: From Search to Structured Evidence

The source material describes causal mapping as one way to make causal analysis more systematic.

Instead of repeatedly asking an AI to search a corpus for different causal findings, the analyst can represent causal claims explicitly:

For example:

A later question can then query the coded relationships instead of asking the model to rediscover the evidence from scratch.

This can be particularly useful for dense causal data, although it does not replace professional interpretation.

15. Give AI a Narrower Job

If AI is asked to freestyle an entire finding, it has to decide:

• what evidence matters;
• what evidence can be ignored;
• which themes deserve attention;
• which quotations should be selected;
• which contradictions should be reported; and
• which findings deserve emphasis.

A narrower workflow reduces the number of hidden decisions.

16. A Practical Recall Audit Workflow

1. Define the evaluation question. State exactly what you want to know.
2. Define relevant evidence. Establish inclusion and exclusion rules.
3. Structure the source material. Use transcripts, passages, cases, or numbered segments.
4. Extract evidence systematically. Ask AI to identify evidence before drafting conclusions.
5. Record contradictions. Do not capture only confirming evidence.
6. Build findings from the coded evidence. Make counts traceable.
7. Audit precision. Go from report to source.
8. Audit recall. Go from source to finding.
9. Document limitations. State what was examined, sampled, and excluded.

17. Prompt: Ask AI to Perform a Recall-Oriented Audit

18. Use Two Directions of Quality Assurance

Neither direction replaces the other.

A report can contain well-supported claims while omitting important evidence. It can also capture many relevant cases while including unsupported interpretations.

The objective is a defensible evidence chain.

19. Completeness Does Not Mean “Include Everything”

A good evaluation report is not expected to reproduce every sentence in every transcript.

Evaluation requires selection, synthesis, interpretation, and judgement.

The objective is not:

“Include everything.”

The objective is:

“Do not systematically omit evidence that is relevant to the evaluation question.”

The evaluator still decides what is relevant, how contradictory evidence should affect interpretation, and what conclusions the evidence can support.

The American Evaluation Association Guiding Principles, updated in 2025, emphasise systematic inquiry, competence, integrity, transparent communication of methods and limitations, and professional judgement in evaluation practice.

20. Interactive Exercise: Find What the Report Missed

Imagine you have 10 interview transcripts.

The AI-generated report says:

The AI cites F2, F5 and F8.

Now conduct a recall audit.

1. Which other participants discussed changes in practice?
2. Did anyone report no change?
3. Did anyone report a negative effect?
4. Did anyone attribute the change to another factor?
5. Is “most” actually supported?
6. Which transcripts were examined?
7. Can the finding be reconstructed from explicit coding?

The key question is not simply whether F2, F5 and F8 were correctly reported.

It is what happened to everyone else.

21. Recall Audit Checklist

22. The Three-Tutorial AI Evaluation QA Framework

Start with Tutorial 1:
AI Write an Evaluation Report: Can You Trust It? – EvalCommunity Academy

Continue with Tutorial 2:
Verification of an AI-Generated Evaluation Report – EvalCommunity Academy

Then use this tutorial:
Audit recall, omissions, contradictory evidence, and completeness.

23. Build It Into Your AI Evaluation Workflow

24. Further Reading

• ACL Anthology — Towards Understanding Omission in Dialogue Summarization
• AbsenceBench — research on identifying missing information.
• PCAOB AS 1105 — Audit Evidence
• PCAOB AS 2315 — Audit Sampling
• OECD — Quality Standards for Development Evaluation
• American Evaluation Association — Guiding Principles

25. Explore More EvalCommunity Resources

How to Use AI in Evaluation Practice
Explore the EvalCommunity guide

AI in Monitoring & Evaluation Certificate
Explore the EvalCommunity Certificate

EvalCommunity Academy
Explore AI and M&E courses, tutorials, and resources

Key Takeaway

This tutorial is part of the EvalCommunity series on AI-assisted evaluation practice.