AI Governance Toolkit: a step‑by‑step guide for Monitoring & Evaluation

1. Getting started

Navigate to https://www.evalcommunity.com/tools/ai-governance-toolkit/ – the toolkit loads immediately, no login required. All data stays in your browser’s local storage; you can close the tab and return later. Use the sidebar to switch between nine tools. Export any table to CSV or Excel for reporting.

🛡️ Tool 1: AI Security & Ethics Risk Checker

Answer eight questions about your AI use case (data type, identifiability, processing location, AI task, human oversight, etc.). The tool calculates a total risk score (low 8–12, moderate 13–25, high 26–40) and generates specific risk alerts plus recommended governance tools.

1. Click the 🛡️ Risk Checker in the sidebar.
2. For each question, select the option that best describes your situation – options are color‑coded by risk level (low/green, medium/orange, high/red).
3. Monitor the progress bar at the bottom; all 8 questions must be answered.
4. Click “Evaluate Risk” to receive your score, a list of key alerts, and tailored tool recommendations.
5. Click any recommended tool card to jump directly to that component.
6. Use “Reset” to clear answers and start a new assessment.

⚠️ Tool 2: AI Risk Register Builder

Log risks with impact (1–5) and likelihood (1–5); the system calculates risk score = impact × likelihood. Colour‑coding helps prioritise (low: green, medium: yellow, high: red).

1. Go to Risk Register and click “+ Add Risk”.
2. Fill in: AI System, Category, Description, Impact, Likelihood, Mitigation, Owner, Status.
3. Save – the new risk appears in the table; score is automatically computed.
4. Use the search bar and dropdown filters (Category, Status) to focus on specific risks.
5. Click column headers (e.g., Score) to sort.
6. Edit or delete any risk using the buttons in the Actions column.
7. Export filtered view via ↓ CSV or ↓ Excel for donor reports.

📋 Tool 3: AI Governance RACI Matrix Builder

R = Responsible, A = Accountable (only one per activity), C = Consulted, I = Informed. Pre‑loaded with 12 activities aligned to NIST functions (GOVERN, MAP, MEASURE, MANAGE).

1. Open RACI Matrix. Default roles and activities appear.
2. Click “+ Role” to add custom roles (e.g., “M&E Director”).
3. Click “+ Activity” to add new governance activities (name + function).
4. For each cell, select R/A/C/I from the dropdown – the background colour changes accordingly.
5. The system warns if any activity has multiple “A” (Accountable).
6. Use search and function filter to narrow activities.
7. Delete roles or activities using the ✕ buttons.
8. Export the matrix via CSV or Excel.

🔴 Tool 4: AI Red Teaming Simulation Planner

1. In Red Teaming Planner, click “+ Add Scenario”.
2. Enter: AI System, Attack Type (Prompt Injection, Data Poisoning, Bias Testing, etc.), Risk Category, Severity, Testing Method, Mitigation Plan, Status.
3. Save – the scenario appears in the table.
4. Filter by Severity or Attack Type to focus on high‑priority tests.
5. Track status: Planned, In Testing, Vulnerability Found, Mitigated, Closed.
6. Export the testing plan for security documentation.

⚖️ Tool 5: AI Fairness Metrics Evaluator

1. Open Fairness Evaluator and click “+ Add Assessment”.
2. Select AI System, Sensitive Attribute (Gender, Race/Ethnicity, Age, etc.), Fairness Metric (e.g., Statistical Parity Difference), Bias Risk Level, Evaluation Method, Mitigation, Status.
3. Save – entries can be filtered by attribute or bias level.
4. Update status as you implement mitigation (Bias Detected → Mitigation Implemented → Fairness Verified).
5. Export for inclusion in ethics reports.

🔬 Tool 6: Trustworthy AI Metrics Explorer

The most comprehensive open repository of AI metrics tailored for M&E. Filter by objective (Accuracy, Robustness, Fairness, Privacy, Security, Explainability), domain, lifecycle stage, and status.

1. Go to Metrics Explorer. Use the coloured pills to filter by trustworthiness objective.
2. Apply additional filters: Domain (Classification, NLP, Computer Vision…), Lifecycle Stage, Status.
3. Click “☆ Add” to include a metric in your personal catalogue; “★ In” removes it.
4. Use the status dropdown to track: Not Reviewed, Under Evaluation, Used in Project, Validated.
5. Click “+ Custom Metric” to add your own metrics.
6. Check “Catalogue only” to see only selected metrics.
7. Export the metrics database or your filtered view.

🧭 Tool 7: AI Ethics Risk Assessment

1. In Ethics Assessment, click “+ Add Assessment”.
2. Fill: AI System, Ethical Risk Type (Discrimination, Privacy Violation, Lack of Transparency…), Affected Stakeholders, Impact, Likelihood, Mitigation, Status.
3. Save and filter by risk type or impact level.
4. Document stakeholder engagement and update mitigation progress.

🏛️ Tool 8: AI Governance Framework Builder

1. Open Governance Framework and click “+ Add Policy”.
2. Enter Policy Name, Governance Domain (Risk Management, Data Governance, Human Oversight, etc.), Responsible Role, Implementation Stage (Planning, Pilot, Full Rollout…), Priority, Status.
3. Track active policies and filter by domain or status.
4. Export the framework for board presentations or donor reporting.

📊 Tool 9: AI Evaluation Monitoring Dashboard

1. Go to Monitoring Dashboard and click “+ Add Metric”.
2. Add: AI System, Metric, Metric Type, Current Value, Threshold (e.g., ≤ 0.10), Risk Level, Last Review Date.
3. Save – entries are colour‑coded by risk level.
4. Filter by Risk Level or AI System to focus on high‑risk items.
5. Update values regularly; keep a log for audit trails.

Best practices for M&E teams

Frequently asked questions

Additional resources

• NIST AI RMF
• OECD AI Principles
• EU AI Act
• EvalCommunity AI in M&E Framework
• AI in M&E Course
• Ethical risks in AI & M&E