SHORTS: Methods to attribute bias changes to data versus model drifting
To attribute bias changes to data versus model drifting in M&E AI systems, decompose observed fairness degradation (e.g., rising EOD gaps) into isolated components using controlled experiments and counterfactual analysis tied to program cycles.arxiv+1
Data Drift Attribution
Retrain the fixed baseline model on current production data while freezing parameters; compute Δfairness (e.g., DIR drop). If >80% of total bias shift occurs here, attribute to input drift—e.g., evolving survey demographics skewing rural representation. Use PSI/KS per feature to pinpoint culprits (>0.1 flags specific fields like beneficiary location).[evidentlyai]
Model Drift Attribution
Freeze input data at baseline; fine-tune or deploy model variants on held-out sets. Measure performance decay via group-wise AUC drops or amplification delta; if model-only retrain shows >50% of bias growth (e.g., overfitting to urban outcomes), blame algorithmic drift from concept shift.[arxiv]
Decomposition Workflow
Baseline Snapshot: Log initial model (M0) + data (D0) fairness KPIs at cycle start.
Test Data Drift: Train M0 on current data (D1) → Fairness_M0_D1; Δdata = Fairness_M0_D1 – Fairness_M0_D0.
Test Model Drift: Train current model (M1) on D0 → Fairness_M1_D0; Δmodel = Fairness_M1_D0 – Fairness_M0_D0.
Interaction Check: Full M1_D1 minus sum of isolates reveals synergistic drift.[substack]
M&E Attribution Table
| Test | Method | Bias Source if Primary | Example Threshold [arxiv] |
|---|---|---|---|
| M0 on D1 | Retrain fixed model | Data (e.g., survey non-response) | PSI >0.1 per attribute |
| M1 on D0 | Refit current model | Model (e.g., parameter decay) | EOD Δ >0.05 |
| Ablation: Remove features | Causal tracing | Feature-specific drift | Attribution >20% total shift |
Automate via M&E dashboards with lineage tracking; intervene on dominant source (rebalance data vs. retrain) within 48 hours of alerts.[relyance]
