What Is an AI Evidence-Synthesis Agent for Evaluation?

An AI evidence-synthesis agent is a controlled workflow around an AI model. It reads an approved evidence set, maps relevant material to evaluation questions, records limitations and disagreement, and prepares structured draft findings that remain subject to evaluator review.

The safest architecture separates adaptive AI work from deterministic checking:

Learning objectives

• Distinguish evidence synthesis from automated report writing.
• Create approved evaluation questions and stable source IDs.
• Build a traceable evidence matrix.
• Record source dependence, limitations, contradiction and evidence gaps.
• Apply confidence rules defined before the AI run.
• Verify generated outputs before evaluator review.
• Keep findings, conclusions and recommendations separate.

The OECD DAC criteria—relevance, coherence, effectiveness, efficiency, impact and sustainability—are useful analytical lenses. They are not an automatic report template or a substitute for evaluation questions, context and professional judgement.

1. Choose a Suitable Evidence-Synthesis Task

Start with a narrow, reviewable task. Test it against these six questions:

2. Prepare Your Computer and Create the Project Folder

Download and extract the starter pack. Do not edit files while they are still inside the ZIP.

Beginner glossary

3. Understand the Project Structure

ai-evidence-synthesis-agent-starter-pack/
├── TASK.md
├── EVALUATION_QUESTIONS.csv
├── SOURCE_REGISTER.csv
├── EVIDENCE_CATALOGUE.csv
├── EVIDENCE_RULES.md
├── CONFIDENCE_RULES.json
├── LOOP_INSTRUCTIONS.md
├── PROGRESS.md
├── source-material/
├── tools/verify_traceability.py
└── outputs/

Source files remain unchanged. Generated outputs go only into outputs/. PROGRESS.md records what happened, unresolved issues and what the next run should do.

4. Prepare the Evaluation Questions

Open EVALUATION_QUESTIONS.csv. Each row needs a stable ID, relevant criterion and approved question.

Do not force all six OECD criteria into every evaluation. Use only the criteria and questions relevant to the purpose, intervention, users and available evidence.

5. Register Sources and Create Evidence Units

Register each source

SOURCE_REGISTER.csv records a stable source ID, title, type, date, author or group, file path, independence group, sensitivity and limitations.

Create an evidence catalogue

EVIDENCE_CATALOGUE.csv breaks sources into short evidence units such as E003. Each unit contains a faithful summary and its limitation. The catalogue is an index, not a replacement for the full source.

Treat quotations carefully

Do not ask the AI to reconstruct quotations from summaries or notes. Use a direct quotation only when the exact words appear in an authorised source and have been manually verified.

Assess relevance and limitations

A source may be genuine but still weak for a particular question. Consider method, coverage, date, independence, disaggregation, missing data, self-report, sampling and whether the source measures activities, outputs, outcomes or longer-term change.

6. Build the Evaluation Evidence Matrix

The machine-verifiable matrix is stored in outputs/evidence-matrix.csv. A human-readable version is stored in outputs/evidence-matrix.md.

Keep four products separate

This beginner agent stops at draft findings and interpretations. Conclusions and recommendations remain with the evaluation team.

7. Define Confidence Rules Before the AI Runs

Confidence describes the strength and traceability of evidence supporting a draft finding. It is not the model’s confidence and does not indicate programme success.

The starter rules are training examples. An evaluation team must approve operational confidence rules before use. A large number of weak or dependent sources does not automatically create high confidence.

8. Open the Folder and Run the Agent Manually

Claude Code on desktop

1. Open Claude Code on desktop.
2. Select the extracted project folder.
3. Use a permission mode with strong oversight for sensitive work.
4. Review proposed edits and commands before approval.
5. Do not approve access outside the project folder unless authorised.

Claude Code in a terminal

cd /path/to/ai-evidence-synthesis-agent-starter-pack
claude

On Windows, open the extracted folder in File Explorer, click the address bar, type powershell and press Enter. On macOS, type cd in Terminal and drag the folder into the window.

Copy-paste instruction

Run the evaluation evidence-synthesis workflow in this project folder.
Read TASK.md, the evaluation questions, source register, evidence catalogue, evidence rules, confidence rules, loop instructions, progress file, and registered source files.
Create or update only approved files in outputs/ and PROGRESS.md.
Do not modify source material.
Do not invent evidence, numbers or quotations.
Preserve disagreement, qualifications and limitations.
Do not write final conclusions or recommendations.
Mark every draft finding for human review.
Run python tools/verify_traceability.py after creating the outputs.

9. Run the Evidence-Traceability Verification

Open a terminal in the project folder and run:

python tools/verify_traceability.py

Use python3 if python is not recognised.

The corrected checker verifies:

• required files and exact CSV schemas;
• blank and duplicate question, source, evidence and finding IDs;
• registered source paths and file existence;
• catalogue links between evidence and sources;
• finding links to approved questions, sources and evidence;
• agreement labels, limitations, confidence rationales and human-review flags;
• minimum source-diversity rules for confidence;
• required output files;
• SHA-256 fingerprints for registered inputs and the evidence-matrix output.

Expected practice result

Findings reviewed: 6
Traceable findings: 6
Errors: 0
Warnings: 0
Confidence: High=0, Moderate=3, Low=2, Insufficient=1

10. Review Contradictions, Gaps and Emerging Findings

Contradictory evidence

Do not resolve disagreement by majority vote. Compare what each source measures, its method, timing, coverage, independence and limitations. A management interview and a participant survey may both be valid perspectives on different aspects of the same issue.

Evidence gaps

Write “insufficient evidence” where appropriate. Missing follow-up data do not prove that impact did not occur; they mean the available evidence cannot support the judgement.

Emerging findings

Check every draft against the original sources. Ask whether it preserves qualifying language, distinguishes intended from achieved results, records limitations and avoids causal claims unsupported by the design.

Evaluator decision

For each row, the evaluator should accept, revise, reject or request additional evidence. Only after this review should findings be developed into conclusions, and recommendations should then follow logically from approved findings and conclusions.

11. Academy Exercise: Synthesise Five Conflicting Sources

The starter pack contains five fictional sources with supporting evidence, contradictory stakeholder perspectives, missing outcome data, an unsupported claim and a monitoring result that differs from participant evidence.

Learner task

1. Review the questions, source register and evidence catalogue.
2. Check each evidence unit against the original source.
3. Create or improve the evidence matrix.
4. Record contradictions and evidence gaps separately.
5. Draft emerging findings without conclusions or recommendations.
6. Run the verifier.
7. Explain which issues require evaluator judgement.

Success criteria

• Every draft finding uses valid source and evidence IDs.
• Contradictory sources remain visible.
• No quotation, number or claim is invented.
• Confidence follows the approved training rules.
• Missing evidence is not treated as failure.
• All draft findings are marked for human review.
• The verifier reports zero errors.

Frequently Asked Questions

What is an AI evidence-synthesis agent?

A controlled AI-assisted workflow that organises registered evidence against evaluation questions and creates traceable draft outputs for evaluator review.

Can it write the final evaluation report?

It can support drafting, but final findings, conclusions and recommendations require evaluator judgement, quality assurance and approval.

Should every evaluation use all six OECD criteria?

No. Apply criteria thoughtfully according to the evaluation purpose, questions, context and intended users.

What happens when sources disagree?

Record the disagreement, methods and limitations. Do not automatically choose the source with the strongest language or largest document count.

Does a file fingerprint prove that evidence is accurate?

No. A fingerprint helps show whether a file changed. It does not validate the truth, quality or interpretation of its contents.

Can this workflow use confidential interviews?

Only when your organisation has approved the tool, legal basis, access controls, storage, retention and processing arrangement. The practice pack uses fictional material.

Final Takeaway

The value of an AI evidence-synthesis agent is not faster prose. It is a more inspectable workflow: approved questions, registered sources, traceable evidence, explicit limitations, preserved contradiction, pre-defined confidence rules, deterministic verification and human evaluator judgement.

Official References

• OECD DAC evaluation criteria
• OECD: Applying Evaluation Criteria Thoughtfully
• UNDP Evaluation Guidelines
• Claude Code permissions