1. Why Evidence Checking Matters

AI can produce evaluation reports that are fluent, structured and persuasive. It can summarise interviews, identify themes, draft findings, generate quotations and organise evidence into a professional-looking report.

But a professional-looking report is not necessarily an evidence-supported report.

One of the most important quality-assurance tasks is therefore to check the relationship between:

REPORT CLAIM → SOURCE EVIDENCE

This direction of checking asks:

Does the evidence actually support what the report says?

This is the foundation of precision checking: testing whether the claims included in the report are supported by the underlying evidence.

However, precision is only one part of evaluation quality. A report can contain claims that are all correct and still omit important evidence.

If you have not yet read the first tutorial, start here:

AI Write an Evaluation Report: Can You Trust It? – EvalCommunity Academy

2. What Is Precision?

In this context, precision asks whether the information that appears in the report is supported by the available evidence.

Imagine an AI-generated report says:

“Several farmers reported that the training contributed to increased harvests, including F3 and F7.”

A precision check would investigate:

• Did F3 actually say this?
• Did F7 actually say this?
• Are the quotations accurate?
• Does “increased harvests” fairly describe their statements?
• Does “contributed to” accurately reflect the language used?
• Did the AI introduce an interpretation that was not present in the source?

The objective is not simply to determine whether the sentence sounds reasonable.

The objective is to determine whether the sentence can be defended from the evidence.

3. Follow the Evidence Chain

A useful way to think about an AI-generated finding is as an evidence chain:

SOURCE → EXTRACTED EVIDENCE → INTERPRETATION → FINDING → REPORT CLAIM

Every step can introduce an error.

4. Break Findings Into Individual Claims

Do not treat an entire paragraph as one factual claim.

For example:

“The training significantly improved farmers’ harvests. Most participants reported higher yields, particularly in the northern district, and several attributed the improvement directly to the training.”

This contains several separate claims:

1. The training improved harvests.
2. The improvement was significant.
3. Most participants reported higher yields.
4. The northern district showed particularly strong improvement.
5. Several participants attributed the improvement directly to the training.

Each claim requires its own evidence check.

Some parts of a paragraph may be supported while other parts are not.

5. Use an Evidence Verification Table

A simple precision audit can separate the claim from the evidence behind it.

This structure separates:

• what the report claims,
• where the evidence comes from,
• what the evidence actually says, and
• what conclusion the evidence supports.

6. Check Quotations Carefully

Quotations create a strong impression of evidence. But a quotation can be genuine and still be misleading when used to support a broader claim.

For every quotation, check:

• Accuracy: Did the participant actually say this?
• Context: Does the surrounding interview change the meaning?
• Representation: Does the quotation support the claim attached to it?
• Selection: Is the quotation presented as illustrative or representative?

Important: A correct quotation can support an incorrect conclusion if the report generalises beyond what the quotation establishes.

7. Check Numerical Claims

Numbers can make an AI-generated report sound particularly authoritative.

Consider:

“18 of 24 farmers reported improved harvests.”

Ask:

• What is the denominator?
• Who is included in the numerator?
• What exactly counts as “reported improved harvests”?
• Were all 24 participants reviewed?
• How were ambiguous cases treated?
• Were contradictory cases considered?
• Does the wording match the coding rule?

A number is only as defensible as the rule used to generate it.

8. Scrutinise “All,” “Only,” “Most” and “None”

Some words dramatically increase the evidence burden of a finding.

Compare:

Lower evidence burden: “Some participants described improvements in harvests.”

Higher evidence burden: “Only two of 24 participants reported improvements in harvests.”

The second statement may be more informative, but it requires a much more systematic check.

9. Check Causal Claims

There is a major difference between:

• “The participant experienced a higher harvest after the training.”
• “The participant attributed the higher harvest to the training.”
• “The training contributed to the higher harvest.”
• “The training caused the higher harvest.”

These statements make progressively stronger claims. The source may support one without supporting the others.

Precision rule: Do not allow a report to move from association or participant attribution to a causal conclusion without an appropriate evidentiary basis.

10. Look for Unsupported Interpretation

AI does not necessarily make mistakes by inventing facts. It can also make mistakes by adding an interpretation that sounds plausible but is not established by the source.

For example:

Source: “The training helped me understand how to use the new technique.”

AI report: “The training substantially increased participants’ technical capacity.”

The evaluator should ask whether the source establishes:

• increased capacity,
• substantial improvement,
• broader participant-level change,
• or simply one participant’s perception.

Evidence checking must therefore examine both facts and interpretation.

11. Precision Does Not Mean Completeness

Suppose the report says:

“Farmers F3 and F7 attributed higher harvests to the training.”

You inspect both transcripts and find that the statement is accurate.

The finding passes the precision test.

But suppose F9 and F15 made the same attribution.

The report is still correct about F3 and F7. It is simply incomplete.

Precision asks: Is what the report says supported?

Recall asks: What relevant evidence did the report fail to include?

That distinction is central to the AI evaluation-reporting problem.

Read Tutorial 1: AI Write an Evaluation Report: Can You Trust It?

12. The Two Directions of Quality Assurance

REPORT → SOURCE

Does the evidence support what the report says?

SOURCE → REPORT

Did the report capture the relevant evidence?

This tutorial focuses primarily on the first direction.

The next tutorial will focus on the second: recall, omissions and completeness auditing.

13. Practical Precision Audit Workflow

1. Extract the substantive claims. Break findings and conclusions into individual claims.
2. Identify the evidence source. Locate the transcript, document, dataset or other source.
3. Verify the evidence. Read enough of the original source to determine whether the claim is supported.
4. Check quotations. Verify wording, context and interpretation.
5. Check numbers. Verify numerator, denominator, coding rule and population.
6. Check interpretation. Look for conclusions that go beyond the source.
7. Check causal language. Distinguish experience, association, attribution and causal inference.
8. Record the result. Classify each claim as supported, partially supported, unsupported, contradicted or not verifiable.

14. A Simple Evidence Classification

15. Prompt: Ask AI to Perform a Precision Audit

You can use AI as a quality-assurance assistant without allowing it to make the final professional judgement.

Important: This prompt improves precision checking. It does not establish recall or prove that the report is complete.

16. Can AI Be the Final Checker?

AI can be useful for first-pass quality assurance.

It can help:

• identify claims that need verification,
• locate relevant passages,
• compare report language with source language,
• flag possible overstatement,
• check numerical claims,
• identify possible quotation mismatches,
• organise the review into a consistent table.

But the evaluator still needs to determine whether the verification process is adequate.

AI can say: “The claim is supported.”

The evaluator must still ask: “What evidence did you check, what did you not check, and is that enough for this evaluation question?”

This approach is consistent with broader responsible-AI frameworks that emphasise trustworthiness, risk management and human oversight. The NIST AI Risk Management Framework provides a framework for organisations managing AI-related risks.

17. What Evaluation Can Learn From Auditing

Auditing provides a useful conceptual comparison.

The PCAOB AS 1105: Audit Evidence addresses the use of sufficient and appropriate evidence in auditing.

The relevant lesson for AI-assisted evaluation is simple:

Do not check only evidence that confirms the report.

Look for evidence that qualifies, contradicts or changes the interpretation.

Evaluation is not financial auditing, so audit procedures should not simply be copied into evaluation practice. But the underlying discipline of matching conclusions to sufficient and appropriate evidence is highly relevant.

The PCAOB AS 2315: Audit Sampling also provides useful principles concerning samples, populations and the relationship between the audit objective and the evidence examined.

18. What Research on AI Factuality Tells Us

Research on AI-generated text reinforces the value of checking individual claims rather than treating an entire response as one factual object.

FActScore evaluates long-form generated text by breaking it into atomic facts and assessing whether those facts are supported by reliable sources.

The broader lesson for evaluation is straightforward:

A report should not be treated as a single factual object. Its substantive claims need to be examined individually.

Research on omission also shows why precision checking has limits. Towards Understanding Omission in Dialogue Summarization treats omission as a distinct problem rather than simply another form of factual error.

19. Precision vs. Recall

You need both.

A high-precision report can still have poor recall.

20. Interactive Exercise: Audit an AI Finding

Scenario: You have 24 farmer interviews.

An AI-generated report states:

“Farmers F3 and F7 attributed increased harvests to the training.”

You inspect both transcripts and find that the statement is accurate.

Now answer:

1. Does the finding pass the precision check?
2. What evidence supports the claim?
3. Have the quotations, if any, been represented accurately?
4. Does “attributed” accurately describe what the participants said?
5. Does the report make any stronger causal claim elsewhere?
6. Could other farmers have made the same attribution?
7. What would you need to check to answer that question?

Learning point: The finding can be completely accurate about F3 and F7 while still being incomplete. Precision and completeness are different quality questions.

21. Precision Audit Checklist

• ☐ Have all substantive claims been identified?
• ☐ Can every important claim be traced to source evidence?
• ☐ Have quotations been checked against the original source?
• ☐ Has the context surrounding quotations been considered?
• ☐ Have numerical claims been checked?
• ☐ Are denominators and populations clear?
• ☐ Have “all,” “only,” “most” and “none” been scrutinised?
• ☐ Has causal language been checked?
• ☐ Has interpretation been separated from direct evidence?
• ☐ Has contradictory or qualifying evidence been considered?
• ☐ Is it clear which evidence was actually checked?
• ☐ Has precision been kept separate from completeness?

22. Build Precision Checking Into the Workflow

Evaluation Question

↓

Evidence Requirements

↓

AI-Assisted Analysis

↓

Evidence-Linked Findings

↓

Precision Audit

↓

Human Interpretation and Judgement

Precision checking should not be an afterthought added only at the end of report writing.

EvalCommunity’s broader guidance on AI in evaluation similarly emphasises human review, evidence traceability, validation and professional judgement. See:

How to Use AI in Evaluation Practice

23. What the Evaluator Still Has to Decide

AI can help perform many parts of evidence verification.

The evaluator still has to decide:

• What counts as adequate evidence?
• How strong does the evidence need to be for this evaluation question?
• Is the interpretation appropriate to the context?
• Does the methodology support the conclusion?
• What uncertainty needs to be communicated?
• Which evidence is contradictory or qualifying?
• Can the findings be defended to the commissioner and intended users?

These are not simply language-generation problems.

They are professional evaluation judgements.

AI can help verify a claim. The evaluator decides whether the evidence is sufficient for the conclusion.

24. Key Takeaway

An AI-generated evaluation report should not be accepted simply because another AI says it is accurate.

The first question should be:

Can each important claim be traced back to evidence that actually supports it?

That is the precision question.

But precision is only half of the quality-assurance problem.

A report can be highly precise and still miss important evidence.

That leads to the next question:

What did the AI fail to include?

External Authoritative Resources

• NIST AI Risk Management Framework
• PCAOB AS 1105: Audit Evidence
• PCAOB AS 2315: Audit Sampling
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• Towards Understanding Omission in Dialogue Summarization
• AbsenceBench: Language Models Can’t Tell What’s Missing

Continue Learning With EvalCommunity

• AI Write an Evaluation Report: Can You Trust It?
• How to Use AI in Evaluation Practice
• AI in Monitoring & Evaluation Certificate
• EvalCommunity Academy