Calibration Exercise Simulator
Practice applying evaluation rubrics consistently. Rate evidence, compare with AI, measure your inter-rater agreement, and sharpen your evaluative judgment.
Configure Your Calibration Session
Choose a scenario, define your rubric criteria, and set calibration preferences before you begin rating evidence.
Select the type of program evaluation you want to practise calibrating.
Each round presents a new evidence vignette. Calibration improves with more rounds.
Include at least 2 criteria with all 4 performance levels for the best calibration experience.
Customise how the simulator behaves during your session.
Read the Evidence Vignette
Study this evidence carefully before rating. Do not scroll ahead to see the AI rating.
These are the criteria and performance levels you will apply to the evidence.
Apply the Rubric Independently
Rate each criterion based on the evidence you read. Make your judgment before seeing the AI rating.
AI Evaluator's Rating & Reasoning
The AI has independently applied the same rubric to the same evidence. Review its reasoning carefully.
Each rating includes the AI's reasoning, evidence citations, and confidence level.
Inter-Rater Agreement Analysis
See where you agreed with the AI evaluator, where you diverged, and what it means for calibration.
| Criterion | Your Rating | AI Rating | Match |
|---|
AI analysis of your rating patterns and recommendations to improve consistency.
Your Calibration Session Summary
A complete analysis of your calibration performance across all rounds, with personalised recommendations.
AI was used as a calibration reference evaluator in this exercise. Agreement scores measure alignment between human and AI judgment on the same rubric and evidence. These scores are indicators of rubric clarity and evaluator calibration — not a definitive measure of evaluation quality. All final evaluative judgments in real evaluations must be made by qualified human evaluators with contextual knowledge.