Best fairness metrics for evaluating impacts on Indigenous groups
Best fairness metrics for evaluating AI impacts on Indigenous groups emphasize group-level equity, cultural context, and rights-based thresholds over individual predictions, adapting standard metrics with Indigenous-led criteria like those from OCAP® and impact severity scales.canada+1
Top Metrics
Prioritize these for M&E, calibrated to Indigenous data sovereignty and cumulative effects.
Disparate Impact Ratio: Measures outcome gaps between Indigenous and non-Indigenous groups; ideal threshold >0.8 to flag underrepresentation in SDG indicators (e.g., health access SDG 3).[optiblack]
Equalized Odds: Ensures similar true/false positive rates across groups, critical for vulnerability models affecting land rights (SDG 15).[canada]
Cultural Severity Score: Custom composite from Canadian guidance—likelihood × (geographic extent + cultural well-being + inequity); co-define thresholds with communities.[canada]
Application in M&E
Tailor to Indigenous governance via participatory baselines.
Baseline Establishment: Use community-defined historical “good conditions” for rights exercise as reference, comparing AI outputs against them.[canada]
Subgroup Testing: Stratify by intersections (e.g., Indigenous women + remote areas) with demographic parity; flag if <80% equity.[iapp]
Cumulative Impact: Weight frequency, reversibility, and spiritual health per rights-holding input.[canada]
Validation: Indigenous reviewers score explainability (SHAP) tied to relational values like reciprocity.[datacamp]
Comparison Table
| Metric | Indigenous Strength | M&E Threshold | SDG Fit [canada] |
|---|---|---|---|
| Disparate Impact | Representation equity | >0.8 ratio | SDG 10 inequality audits |
| Equalized Odds | Error fairness | 0.8-1.2 range | SDG 16 justice predictions |
| Severity Score | Rights-based (likelihood, cultural well-being) | Low/Mod/High per community | SDG 15 land impacts |
| Demographic Parity | Access equity | Balanced proportions | SDG 2 food security |
Integrate with CARE Principles for holistic, decolonial assessments enhancing SDG evaluation trust.[prism.sustainability-directory]
