AI Tools for Evidence Synthesis: The Cochrane Case
AI tools for evidence synthesis
The Cochrane case: how AI tools are being evaluated for evidence synthesis
Can AI improve evidence synthesis without compromising rigor?
Artificial intelligence is increasingly being introduced into evidence synthesis.
AI tools can support activities such as literature screening, data extraction,
classification and other stages of the review process.
But the important question is not simply whether AI can make evidence synthesis
faster. The more important question is whether AI can be used while maintaining
the transparency, independence and methodological rigor required for trusted evidence.
In February 2026, Cochrane announced an innovative study to test whether AI tools
could support or enhance key stages of evidence synthesis. The study is designed
around independent, prospective and reproducible evaluation rather than relying
only on claims made by AI tool developers.
Read the original Cochrane announcement
Case at a glance
Organization: Cochrane
Announcement: February 19, 2026
Focus: AI tools for evidence synthesis
Key tasks: Literature screening and data extraction
Study design: Adaptive platform study
Comparison: AI tools and the traditional dual-human standard
Planned scope: Approximately 15 Cochrane review updates
1. Why did Cochrane launch the study?
AI technologies are advancing rapidly. Cochrane notes that a growing number of
studies and preprints are exploring whether AI can reproduce elements of evidence
synthesis in a fraction of the time.
However, Cochrane also identified limitations in many existing evaluations,
including reliance on studies conducted by tool developers themselves.
Speed is only one factor in evidence synthesis. Trusted evidence also depends on
transparency, independence and methodological rigor.
Cochrane therefore sought to evaluate AI tools through a more rigorous,
independent and prospective approach.
Cochrane, 2026
.
2. What parts of evidence synthesis are being examined?
The study focuses on whether AI tools can support or enhance key stages of the
evidence-synthesis lifecycle.
- Literature screening — identifying studies that may be relevant.
- Data extraction — supporting the collection of information from studies.
- Evidence-synthesis workflows — examining how AI can be incorporated while maintaining established standards.
Importantly, the study does not frame AI simply as an automation exercise.
The tools are evaluated within real review workflows and compared with traditional
methods.
3. Cochrane received 48 AI tool proposals
In November 2025, Cochrane invited AI tool developers to submit proposals.
The organization received 48 proposals.
The submissions were then assessed against a set of criteria before tools were
shortlisted.
The initial screening considered:
- Alignment with RAISE principles
- Alignment with Cochrane’s mission, vision and values
- Tool maturity
- Existing evaluation of screening and data-extraction performance
- Affordability
- Data protection and copyright compliance
Cochrane then considered additional evidence about the tools, including existing
evaluation evidence, evidence supporting performance claims, functionality,
human oversight and dependence on user-defined prompts.
An AI tool should not be selected solely because it performs well in a demonstration.
Evidence about how, where and under what conditions the tool performs matters.
4. What did Cochrane look for when assessing AI tools?
The tool-selection process provides a useful model for M&E organizations that
are considering AI systems.
Cochrane considered:
- Existing evidence about performance
- Evidence supporting developers’ performance claims
- The range of functions supported by the tool
- The level of human oversight during automation
- The extent to which results depend on user-defined prompts
- Data protection and copyright considerations
- Affordability and potential scalability
This is particularly relevant for M&E because the same AI system may behave
differently depending on the task, evidence base, prompt, configuration and
level of human oversight.
5. Why use an adaptive platform study?
One of the most distinctive features of the Cochrane approach is its
adaptive platform design.
Multiple interventions — including different AI tools and the dual-human
standard — can be evaluated under a common protocol.
Common protocol → Multiple AI tools → Performance criteria → Comparison → Adaptation
The protocol defines criteria that AI tools are expected to meet, including
performance metrics and uncertainty thresholds.
The platform design also allows new tools to be added and ineffective tools
to be removed over time.
This is especially relevant in AI because tools and capabilities can change
rapidly.
6. AI is being tested against traditional methods
Cochrane planned to test the AI tools across approximately
15 Cochrane review updates.
Performance will be compared with traditional methods through the work of
review author teams.
This comparison matters because an AI system should not be evaluated only by
asking whether it produces an output. The relevant question is how its output
compares with an established review process.
When testing an AI tool, establish a human-reviewed reference process before
deciding whether the AI output is acceptable.
7. The study is designed around real review work
Cochrane’s approach is intended to evaluate AI within actual review workflows,
rather than relying only on isolated demonstrations or simplified benchmark
exercises.
This distinction is important for M&E.
An AI tool may perform well on a carefully prepared test set but behave
differently when working with:
- Different document types
- Different languages
- Inconsistent reporting
- Complex evaluation designs
- Large evidence collections
- Ambiguous inclusion criteria
- Different prompts or configurations
Testing the tool within the workflow therefore becomes part of the evaluation
itself.
8. The role of human oversight
Cochrane explicitly considered the level of human oversight during
automation as part of the tool-selection process.
This is important because AI-supported evidence synthesis is not simply a
question of whether a system can perform a task independently.
The level and type of human involvement can influence how AI outputs should
be interpreted and used.
Define the human role before deploying the AI system — not after an error occurs.
9. The M&E case: reviewing thousands of documents
Imagine an international development organization wants to synthesize evidence
from thousands of evaluation reports, baseline studies, impact assessments and
learning documents.
The team wants AI to help with:
- Finding potentially relevant documents
- Screening records
- Extracting study characteristics
- Identifying outcomes and indicators
- Classifying evidence
- Building an evidence matrix
- Preparing material for synthesis
Instead of asking, “Which AI tool is the most powerful?”,
the M&E team asks:
“Which tool performs adequately for this task, in our evidence environment,
under defined conditions?”
10. What should an M&E team measure?
Cochrane’s approach provides a useful starting point for thinking about
AI-tool evaluation in M&E.
| Dimension | M&E question |
|---|---|
| Accuracy | Does the AI produce sufficiently accurate results? |
| Completeness | Does it miss relevant evidence? |
| Consistency | Does performance remain stable across cases? |
| Human oversight | What must a qualified professional verify? |
| Efficiency | Does the tool reduce workload without creating unacceptable quality risks? |
| Usability | Can M&E professionals use it effectively within their existing workflow? |
11. The hidden question: what happens to the final evidence?
A critical issue in AI-assisted evidence synthesis is not simply whether the
AI makes an error at one stage.
The more important question is whether an early error can influence later
stages of the synthesis.
AI screening error → Missing study → Incomplete evidence base → Different synthesis
This is why M&E teams should think beyond task-level accuracy and consider
the potential consequences of errors for the final evidence and conclusions.
12. Why performance thresholds matter
One challenge identified by Cochrane is that the evidence-synthesis community
does not necessarily have agreement about what constitutes “good enough”
performance for AI tools.
Yet a study evaluating AI needs predefined criteria against which performance
can be assessed.
For an M&E team, this means defining acceptable performance before
the AI becomes part of the operational workflow.
What level of error, uncertainty or disagreement would cause us to modify,
restrict or stop using this AI tool?
13. AI tool selection is itself an M&E activity
There is an important methodological lesson here.
Selecting an AI tool is not just a procurement decision. It can be treated as
a structured assessment process involving:
- Defined criteria
- Evidence review
- Performance assessment
- Risk assessment
- Human oversight requirements
- Validation within the intended workflow
- Ongoing monitoring
This approach is closely aligned with the broader M&E principle of
testing claims against evidence.
14. Apply the lesson to your own AI workflow
Before introducing an AI evidence-synthesis tool into an M&E workflow,
consider the following checklist:
☐ Define the exact task AI will perform.
☐ Identify the evidence base on which the tool will operate.
☐ Review independent evidence supporting performance claims.
☐ Establish a human-reviewed reference sample.
☐ Define acceptable performance thresholds.
☐ Define the human verification process.
☐ Document prompts, settings and relevant tool versions.
☐ Monitor errors and disagreements.
☐ Assess whether errors could affect final conclusions.
☐ Define conditions for modifying or stopping AI use.
15. Related EvalCommunity resources
EvalCommunity has developed practical resources that complement the issues
raised by the Cochrane case.
AI Evidence Synthesis for M&E Literature Reviews
Learn how AI can be incorporated into structured M&E evidence-synthesis workflows.
AI Tool Evaluation Protocol for M&E
A practical approach to testing, comparing and documenting AI tools for M&E tasks.
AI Evidence-Synthesis Agent for Evaluation
Explore a controlled workflow for organizing evaluation evidence into a traceable matrix.
Using AI Safely with Sensitive M&E Data
Consider privacy, confidentiality and responsible AI use when working with M&E data.
M&E Professional Takeaway
The Cochrane case illustrates a broader principle for AI in M&E:
do not evaluate AI only by how impressive or fast it appears.
Evaluate it against defined criteria, compare its performance with established
methods, test it within the intended workflow and establish clear requirements
for human oversight.
Want to build stronger AI skills for M&E?
Explore practical AI resources, tools, tutorials and training designed
specifically for Monitoring & Evaluation professionals.
Primary source:
Martha Powell, Cochrane (2026),
Cochrane launches innovative study to assess AI tools for evidence synthesis
Educational note:
This EvalCommunity case study adapts the documented Cochrane study for
Monitoring & Evaluation learning. The M&E scenarios, questions and examples
are illustrative and are not presented as Cochrane M&E activities or results.
