Human-in-the-Loop Validation Practice
Interactive Quality Assurance Training for AI-Coded Qualitative Data
M&E Professional Development📋 How This Works
- Review the thematic codebook in the sidebar
- Generate a 15% random sample from 200 interview excerpts
- Validate 5 representative items by comparing AI codes with your judgment
- Review agreement metrics and error rates
- Apply the 10-20% rule to make a quality assurance decision
📊 Step 1: Generate Sample
Our dataset contains 200 interview excerpts about community trust in health services. Generate a 15% random sample (30 items) to begin your validation practice.
200 Total Interviews
30 Sample Size (15%)
5 Items to Validate
Generated Sample (30 items)
✓ Step 2: Validate AI Codes
Compare the AI-assigned codes with your expert judgment. Code 5 representative items from your sample to assess agreement levels.
0/5 Items Validated
0 Agreements
0 Disagreements
📈 Step 3: Analysis Results
Validation Metrics
0 Agreements
0% Percent Agreement
0% Error Rate
📊 Understanding Your Results
Percent Agreement shows how often you and the AI assigned the same code. Error Rate is the inverse (100% - agreement). These metrics guide your quality assurance decision.
🎯 Step 4: Apply 10-20% Rule
Based on your error rate, select the appropriate action:
The 10-20% rule provides a systematic approach to quality assurance decisions. Choose the option that matches your error rate findings.