TypeSafe AI for Evaluation Evidence Screening
How to Use TypeSafe AI for Evaluation Evidence Screening
A practical beginner’s tutorial for screening abstracts, reports, findings, and evidence summaries using structured AI decisions.
A structured screening workflow that checks whether an evidence record is relevant to an evaluation and identifies records that require human review.
1. The problem
Imagine an evaluation team has collected 5,000 abstracts, reports, or evidence summaries.
Each record needs to be screened against predefined inclusion criteria.
Instead of asking AI one large question such as:
break the task into smaller decisions.
This is consistent with TypeSafe’s current workflow approach: decompose a task into narrow intelligent judgments and use ordinary programmatic rules wherever possible.
2. The screening workflow
| Decision | Question |
|---|---|
| 1 | Is the evidence relevant to the evaluation question? |
| 2 | Is the population relevant? |
| 3 | Is the intervention relevant? |
| 4 | Should an evaluator review the record? |
The final decision should be based on your predefined screening protocol, not on an AI-generated interpretation of what your evaluation “probably” needs.
3. Define the evaluation criteria first
Before using TypeSafe, define your inclusion criteria.
For example:
What evidence exists on the effectiveness of vocational training programs for unemployed young adults in low- and middle-income countries?
Possible screening criteria:
- The evidence concerns vocational or employment-oriented training.
- The population includes unemployed or economically inactive young adults.
- The evidence addresses outcomes, effectiveness, implementation, or impact.
Write these criteria down before configuring the AI. This gives both the evaluator and the AI a defined basis for screening.
4. Build a human validation sample
Do not begin with 5,000 records.
Start with perhaps 20–50 records and have an evaluator screen them manually.
Record at least:
- Record ID
- Abstract or evidence summary
- Human relevance decision
- Human population decision
- Human intervention decision
- Reason for inclusion or exclusion
This becomes your reference set for testing the AI workflow.
5. Example evidence record
Population: Unemployed young adults aged 18–29
Intervention: Six-month vocational skills training
Summary: The study examines whether vocational training improved employment outcomes among unemployed young adults in three low-income countries.
A human evaluator might classify this record as relevant to all three screening criteria.
6. Decision 1 — Evidence relevance
Create a narrow decision:
Is this evidence relevant to the evaluation question?
Define what “relevant” means using your protocol.
For example, the evidence should:
- address vocational or employment-oriented training; and
- provide information relevant to outcomes, effectiveness, implementation, or impact.
Use a structured YES / NO output.
TypeSafe currently describes its yes/no decision primitive as Noul. It also supports Choice and Score decision types.
7. Decision 2 — Population relevance
Is the population described in this evidence relevant to the evaluation?
For our example, consider:
- Are young adults included?
- Are they unemployed, economically inactive, or seeking employment?
- Is the population sufficiently comparable to the evaluation population?
Return YES / NO.
8. Decision 3 — Intervention relevance
Is the intervention described in this record relevant to the evaluation?
Your definition might include vocational training, technical skills training, job-readiness training, or other interventions explicitly covered by your protocol.
Do not automatically classify every education program as relevant.
9. Decision 4 — Human review
Create a separate decision focused on workflow handling:
Should an evaluator review this record before the final screening decision?
Possible outputs:
- YES — send to human review.
- NO — the record meets the conditions for automated handling under your validated protocol.
For ambiguous or consequential cases, human review can be built into the workflow rather than forcing the system to make a final inclusion/exclusion decision.
10. Use code for rules, AI for judgment
This distinction is important.
Suppose your protocol says:
THEN the record is excluded, unless your protocol requires human confirmation.
That is a deterministic rule. Ordinary code can handle it.
The AI should focus on the judgment required to determine whether the population is relevant.
This separation is central to TypeSafe’s published workflow methodology.
11. Test a clearly relevant record
Run the example record through the individual decisions.
| Decision | Illustrative result |
|---|---|
| Evidence relevant? | YES |
| Population relevant? | YES |
| Intervention relevant? | YES |
| Human review? | Depends on your validated workflow |
The results above are illustrative. Inspect the actual model output rather than assuming it will produce these results.
12. Test an ambiguous record
Population: Adults in rural communities
Intervention: Community education workshops
Summary: The program provides general education and community activities but does not clearly describe vocational training or employment outcomes.
This is deliberately ambiguous.
The record mentions education, but that does not necessarily make it relevant to an evaluation of vocational training.
A sensible workflow could route this record to human review if the evidence is insufficient to make a reliable automated screening decision.
13. Build your validation spreadsheet
Compare the AI results with the human-coded reference sample.
| Column | Purpose |
|---|---|
| Record ID | Unique identifier |
| Human relevance | Reference decision |
| AI relevance | AI decision |
| Human population decision | Reference decision |
| AI population decision | AI decision |
| Human intervention decision | Reference decision |
| AI intervention decision | AI decision |
| AI probability/confidence | Model output, when available |
| Reason for disagreement | Why human and AI decisions differ |
14. Investigate disagreements
Do not reduce validation to a single accuracy percentage.
When human and AI decisions disagree, ask why.
- Was the screening criterion ambiguous?
- Was information missing from the abstract?
- Did the AI misunderstand the population?
- Did human evaluators disagree?
- Was the intervention borderline?
- Does the screening protocol need clarification?
These disagreements can help improve the screening criteria as well as the AI workflow.
15. Use confidence carefully
TypeSafe says Jev returns structured decisions together with probabilities and confidence information.
Do not assume that a high confidence value automatically means the decision is correct for your evidence-review task.
First compare confidence with actual performance on your human-coded validation sample.
Only then consider using a confidence threshold to route records to human review.
16. Test difficult cases
Your validation sample should contain more than easy examples.
- Very short abstracts
- Ambiguous populations
- Multiple interventions
- Similar but different interventions
- Missing information
- Conflicting information
- Indirect outcomes
- Unusual terminology
- Records that appear relevant but fail an inclusion criterion
17. Important: screening is not evidence appraisal
Evidence appraisal asks: “How credible, rigorous, relevant, or useful is this evidence for the evaluation?”
These are different tasks.
A document can be highly relevant to an evaluation but still have methodological limitations.
Do not treat an automated screening result as a judgment about the quality, validity, or credibility of the underlying research.
18. Scale gradually
Check the results at each stage before increasing the volume.
Do not move directly from a few examples to thousands of records without validation.
19. The final architecture
↓
Relevance judgment
↓
Population judgment
↓
Intervention judgment
↓
Programmatic screening rules
↓
Human review when required
↓
Final screening dataset
This separates three things:
- AI judgment — interpreting unstructured evidence.
- Programmatic logic — applying explicit screening rules.
- Human judgment — resolving ambiguous or consequential cases.
20. Privacy and responsible use
Before sending real evaluation documents to an external AI service, review your organization’s privacy requirements, consent arrangements, contractual obligations, applicable laws, and the provider’s current terms and data-handling practices.
For your first experiment, use public, fictional, or appropriately anonymized evidence records.
21. Beginner checklist
- ☐ Define the evaluation question.
- ☐ Define inclusion and exclusion criteria.
- ☐ Create a human-coded validation sample.
- ☐ Separate relevance, population, and intervention decisions.
- ☐ Create narrow structured decisions.
- ☐ Use programmatic rules where possible.
- ☐ Test clear and ambiguous records.
- ☐ Compare AI decisions with human decisions.
- ☐ Investigate disagreements.
- ☐ Validate confidence before using thresholds.
- ☐ Define human-review rules.
- ☐ Scale gradually.
22. The key lesson
Break the task into smaller decisions, define the criteria first, validate against human judgments, use code for deterministic rules, and keep humans involved where uncertainty or consequences require it.
The practical formula is:
Criteria → Decisions → Test → Validate → Review → Scale
Official resources
