Back

AI Misuse Red Flags Checklist for M&E

Monitoring & Evaluation · Responsible AI Toolkit

AI Misuse Red Flags Checklist for M&E
Full Evaluation Process Diagnostic Toolkit

The world’s first systematic diagnostic to detect, prevent, and mitigate AI misuse across the entire evaluation lifecycle — from scoping to reporting.

🏆 WORLD-FIRST TOOLKIT · LAUNCHED BY EVALCOMMUNITY

Why this toolkit is groundbreaking: As artificial intelligence increasingly shapes monitoring and evaluation (M&E) — from generating survey questions to synthesizing qualitative data — unprecedented risks emerge: algorithmic bias, context blindness, false precision, and erosion of evaluative reasoning. This diagnostic, the first globally available red-flag instrument, equips practitioners with a structured audit framework across 6 phases and 16 critical risk domains. By applying this checklist, evaluation teams can identify high-stakes misuse, protect credibility, and maintain human accountability.

📊 6 evaluation phases
⚠️ 40+ red flag indicators
📈 3-tier risk scoring
🧠 Human-in-the-loop gold standard

🔴 PHASE 1 — SCOPING AND EVALUATION DESIGN

critical entry point

1.1
Defining Purpose & Evaluation Questions
✅ AI can support
  • Generating draft evaluation questions, suggesting frameworks (OECD-DAC, utilization-focused), structuring hierarchies
⛔ Red flags:

  • AI-generated questions used without stakeholder validation
  • Generic, not decision-oriented, no alignment with users
  • Overly broad or unmeasurable
🚨 CRITICAL RISK SIGNAL: “We copied AI-generated evaluation questions directly into the design”
1.2
Theory of Change & Logic Model
✅ AI can support
  • Drafting ToC structures, identifying causal pathways, suggesting assumptions
⛔ Red flags:

  • AI defined causal pathways without expert validation
  • Assumptions not tested or contextualized
  • Linear logic applied to complex systems
🚨 CRITICAL RISK SIGNAL: “The Theory of Change was generated by AI and not validated”
1.3
Indicator Selection & Framework
✅ AI can support
  • Suggesting indicator options, structuring matrices, aligning with outcomes
⛔ Red flags:

  • Indicators not measurable in the field
  • No baseline or data source, not linked to causal logic
  • Overly complex or irrelevant
🚨 CRITICAL RISK SIGNAL: “AI-generated indicators were used without feasibility testing”

🔴 PHASE 2 — METHODOLOGY AND PLANNING

design realism

2.1
Evaluation Design (Methods)
✅ AI can support
  • Recommending qualitative/quantitative methods, mixed-method approaches, sampling drafts
⛔ Red flags:

  • Methods chosen ignoring context constraints
  • Textbook methodology not feasible
  • Ethical considerations missing
🚨 CRITICAL RISK SIGNAL: “We used AI-recommended methods without adapting to field realities”
2.2
Sampling Strategy
✅ AI can support
  • Explaining sampling techniques, sample size calculations, structuring plans
⛔ Red flags:

  • Sampling ignores marginalized groups
  • Unrealistic size, no clarity on representativeness
2.3
Data Collection Tools
✅ AI can support
  • Drafting questionnaires, interview guides, logical structuring
⛔ Red flags:

  • Culturally inappropriate or leading questions
  • Tools not piloted, language too complex
🚨 CRITICAL RISK SIGNAL: “AI-generated tools were used without field testing”

🔴 PHASE 3 — DATA COLLECTION

integrity risks

3.1
Field Data Collection
✅ AI can support
  • Transcription, real-time summaries, enumerator guidance
⛔ Red flags:

  • Over-reliance on AI summaries instead of raw data
  • Blindly following AI scripts, ethics ignored
3.2
Data Quality Assurance
✅ AI can support
  • Detecting inconsistencies, flagging missing data, validation checks
⛔ Red flags:

  • No human review of anomalies
  • Blind trust in AI-cleaned datasets
  • Missing documentation
🚨 CRITICAL RISK SIGNAL: “AI cleaned the dataset and we assumed it was accurate”

🔴 PHASE 4 — DATA ANALYSIS

⚠️ Full phase includes: Quantitative Analysis, Qualitative Coding & Thematic Analysis, Triangulation, Causal Analysis — each with AI support notes, red flags, and critical risk signals similar to phases above. Full checklist available in downloadable PDF.

🚨 Critical risk example: “AI replaced human interpretation in qualitative analysis”

⚠️ Cross-cutting risks — apply to all phases

  • Context loss: AI outputs used without local adaptation
  • Bias amplification: Dataset bias not assessed
  • Over-standardization: Generic frameworks used blindly
  • False precision: AI outputs appear more certain than reality
  • Responsibility gap: No clear ownership of conclusions

🧮 FINAL DIAGNOSTIC — RISK SCORING

Scoring protocol: 🚨 Critical risks = 3 points  |  🔴 High risks = 2 points  |  ⚠️ Warnings = 1 point

0–5
🟢 LOW RISK
Responsible AI use
6–12
🟡 MODERATE RISK
Needs stronger validation
13+
🔴 HIGH RISK
Credibility compromised

🟢 AI SHOULD:

  • Generate drafts & accelerate analysis
  • Suggest patterns & challenge assumptions
  • Structure repetitive workflows
❌ AI SHOULD NOT:

  • Define causality or replace human judgment
  • Validate itself or replace stakeholder voice
  • Supplant lived experience

🔁 SAFE AI WORKFLOW (GOLD STANDARD)

1. HUMAN DEFINES: Purpose, Theory of Change, evaluation logic
2. AI SUPPORTS: Drafting, pattern detection, structuring
3. HUMAN VALIDATES: Evidence, Context, Interpretation
4. HUMAN DECIDES: Conclusions, Recommendations, Accountability

Preferred citation: EvalCommunity. (2025). AI Misuse Red Flags Checklist for M&E: Full Evaluation Process Diagnostic Toolkit. Retrieved from https://www.evalcommunity.com/toolkits/ai-misuse-red-flags-toolkit/
🌍 First globally available systematic red-flag instrument for AI misuse in M&E. Free to use, adapt, share.
Academy

The courses and articles have been developed by an experienced team of evaluators and software developers under the guidance of Fation Luli. The EvalCommunity Academy combines practical expertise in Monitoring & Evaluation with cutting-edge AI technologies to provide high-quality, accessible learning experiences for professionals around the world.