SHORTS: Which statistical metrics best detect bias drift over time
Kolmogorov-Smirnov (KS) test and Population Stability Index (PSI) stand out as the best statistical metrics for detecting bias drift over time in M&E AI models, capturing shifts in data distributions that amplify disparities across deployment cycles like program monitoring phases.arxiv+1
Distribution Shift Metrics
KS Statistic measures maximum distance between cumulative distribution functions of baseline (training) vs. current M&E data for sensitive attributes (e.g., region, gender); values >0.05 signal drift, with p<0.01 confirming significance for retraining triggers. PSI quantifies stability as ∑[(current% – baseline%) × ln(current%/baseline%)], flagging >0.1 as moderate drift in beneficiary outcome predictions.acceldata+2
Bias-Specific Drift Metrics
Equality of Opportunity Difference (EOD) Drift: Tracks Δ|TPR_groupA – TPR_groupB| over cycles; rising >0.1 indicates amplifying false negatives for minorities in impact forecasts [arxiv]. Wasserstein Distance gauges “earth mover’s” shift in prediction distributions, ideal for continuous M&E scores like risk indices (>0.05 threshold) [acceldata].
M&E Monitoring Table
| Metric | Use Case | Threshold | M&E Application [arxiv] |
|---|---|---|---|
| KS Test | Feature/Label drift | >0.05, p<0.01 | Survey response shifts |
| PSI | Categorical stability | >0.1 (moderate), >0.25 (high) | Demographic parity in logframes |
| EOD Drift | Fairness degradation | Δ>0.1 per cycle | Beneficiary targeting equity |
| Wasserstein | Prediction drift | >0.05 | Impact score disparities [evidentlyai] |
Combine with performance drops (e.g., AUC decline >5%) for automated alerts in routine M&E dashboards, enabling proactive interventions.linkedin+1
