The Case

This is an educational simulation developed by EvalCommunity, inspired by the ARISMA framework for AI- and LLM-assisted systematic reviews, scoping reviews and mapping studies.

An international development organization has accumulated a large collection of evaluation reports, research studies, monitoring documents, learning products and qualitative evidence. The M&E team wants to use AI to accelerate evidence synthesis.

The M&E Manager must determine where AI can assist, where it must be validated, and where humans must retain decision authority.

1. What ARISMA Changes

ARISMA does not present AI as an autonomous reviewer. It proposes treating AI as an inspected, benchmarked, logged and reversible assistant.

Its governing principle is that consequential scientific decisions should remain human-interpretable, human-auditable and human-accountable.

The framework was developed from evidence-synthesis guidance, research on AI-assisted reviews and structured consultation with 21 researchers and information specialists. :contentReference[oaicite:2]{index=2}

2. Where Can AI Enter the M&E Evidence Process?

ARISMA specifically organizes AI-assisted evidence synthesis across framing, protocolization, retrieval, selection/enrichment, evidence structuring, and synthesis/reporting. :contentReference[oaicite:3]{index=3}

3. Three Different Roles for AI

ARISMA argues that the higher the epistemic consequence of a task, the stronger the required validation and human oversight. :contentReference[oaicite:4]{index=4}

4. The Four Conditions

These four conditions are central to ARISMA’s approach to responsible AI-assisted evidence synthesis. :contentReference[oaicite:5]{index=5}

5. Case Challenge: AI Screening

Imagine that an M&E team has 10,000 potentially relevant records. An AI system recommends excluding 7,500 and reviewing 2,500.

What should the M&E Manager do first?

1. Deploy the system immediately.
2. Ask the AI how confident it is.
3. Create a human-reviewed pilot set and benchmark its performance.
4. Accept all exclusions below a confidence threshold.

ARISMA describes screening as a particularly consequential point because AI can directly reshape the eventual evidence base. Its proposed workflow therefore separates calibration from production use and retains human review and override. :contentReference[oaicite:6]{index=6}

6. AI Extraction: The Numbers Problem

AI may be useful for identifying descriptive information or prepopulating extraction fields. But critical quantitative fields require stronger controls.

The numbers may look plausible. But what if the 18% refers only to a subgroup, a specific outcome or a particular follow-up period?

M&E principle: the more an extracted field affects aggregation, analysis or a major conclusion, the stronger the requirement for source-level human verification.

ARISMA explicitly distinguishes lower-risk descriptive fields from critical quantitative fields such as sample sizes, event counts, effect estimates and confidence intervals. :contentReference[oaicite:7]{index=7}

7. AI-Assisted Qualitative Coding

AI can also support qualitative evidence synthesis.

ARISMA treats AI-generated keywords and categories as exploratory. The evaluation team must finalize the classification scheme through human consolidation and auditing. :contentReference[oaicite:8]{index=8}

This connects directly with EvalCommunity’s existing work on AI-powered qualitative data analysis in M&E and the AI in Monitoring & Evaluation course.

8. Evidence Provenance

A central question for an AI-assisted evidence process is:

A practical evidence chain is:

Source → Extracted evidence → Verified evidence table → Coding → Synthesis claim → M&E interpretation

If the chain cannot be reconstructed, the team cannot easily determine whether a conclusion is grounded in the evidence or introduced by the AI system.

ARISMA therefore proposes provenance across screening, extraction and synthesis, including records of AI recommendations, human overrides and source verification. :contentReference[oaicite:9]{index=9}

EvalCommunity applies the same principle in its practical AI Evidence Synthesis for M&E Literature Reviews tutorial.

9. Validate Before You Trust

An AI system should not be accepted simply because its outputs look convincing.

• Create a representative human-reviewed validation sample.
• Compare AI outputs with human judgments.
• Examine disagreements and borderline cases.
• Measure task-appropriate performance.
• Document the model, prompts and settings.
• Recalibrate when the model, prompts, criteria or corpus changes.
• Maintain human override.

ARISMA specifically emphasizes recurring calibration rather than treating validation as a one-time exercise. Changes in model, prompts, criteria, corpus or observed performance can trigger recalibration. :contentReference[oaicite:10]{index=10}

For a broader practical approach, see EvalCommunity’s AI Tool Evaluation Protocol for M&E.

10. When Should AI Be Restricted or Disabled?

ARISMA explicitly includes revision or disabling of AI support when validation requirements are not met. :contentReference[oaicite:11]{index=11}

For the data-protection dimension, see EvalCommunity’s How to Use AI Safely with Sensitive M&E Data.

11. You Are the M&E Manager

Your organization wants to introduce AI into evidence synthesis. For each task, decide the appropriate level of AI involvement.

12. AI Evidence-Synthesis Quality Checklist

13. The M&E Professional Takeaway

AI can process evidence at scale. It can identify patterns, organize information and accelerate routine work. But the evaluator remains responsible for defining the evidence question, validating the process, interpreting findings and determining whether conclusions are adequately supported.

14. Questions for Reflection

1. Which parts of your evidence workflow could safely be assisted by AI?
2. Which decisions should remain human-led?
3. What would constitute an unacceptable AI error?
4. Do you have a human-reviewed validation dataset?
5. Can your team trace an important conclusion back to its source evidence?
6. How would you document AI involvement in an evaluation report?
7. What would trigger you to restrict or stop using an AI system?

EvalCommunity Academy · AI in Monitoring & Evaluation