AI Ally: Co-Designed Anti-Harassment AI for Girls, Young Women and Gender-Diverse Individuals

Analysis and critique by EvalCommunity: This case study presents an independent analysis of the AI Ally project, led by the University of Melbourne in partnership with Girl Geek Academy and funded by the eSafety Commissioner (Australian Government). It is based on publicly available information from ANROWS and University of Melbourne research publications, and reflects the perspectives of EvalCommunity on strengths, gaps, and recommendations for M&E practice in technology-facilitated gender-based violence prevention. EvalCommunity does not claim ownership of the original content. We encourage readers to consult the original sources for complete context.

In response to high rates of tech-based gendered violence in Australia, University of Melbourne researchers, in partnership with Girl Geek Academy and funded by the eSafety Commissioner, developed AI Ally — an AI-powered dashboard designed to help combat online abuse by empowering victims rather than simply punishing perpetrators.

The research team surveyed 230 Australian girls, young women, and gender-diverse individuals aged 14-25 about their experiences using social media and their opinions on AI moderators.

The survey findings revealed a critical gap: there is a lack of practical support tools offered to female gamers and young women navigating online spaces. AI Ally aims to fill that gap by reducing the burden on victims when navigating complex reporting processes.

AI Ally is an AI-powered dashboard that uses Natural Language Processing (NLP) to detect and document online harassment in real-time on social platforms. The initial focus is on Discord, a platform used by approximately 150 million people each month to communicate via voice, video, or text.

Key features:

• Real-time detection: Monitors user’s conversations and flags inappropriate or harmful interactions as they happen.
• Explanatory AI: Provides an explanation of why specific comments have been deemed toxic or inappropriate.
• Evidence logbook: Automatically generates a logbook that can be used to file reports with platforms or authorities like the eSafety Commissioner.
• Opt-in basis: Users have full control over whether to activate monitoring and how to use the documented evidence.
• Autonomy-focused: Designed to offer users the knowledge and resources to make informed decisions based on their own preferences and personal safety.

Why this approach is different: Existing AI moderation often works by detecting ‘toxic’ messages and ‘punishing’ the perpetrator with bans or suspensions. However, researchers say this is a flawed approach because language is nuanced and AI moderation can misinterpret human interactions. AI Ally instead prioritizes victim empowerment over perpetrator punishment.

The project adopts a user-centered, trauma-informed approach to AI system design. This means:

• Survey research: ~230 responses detailing experiences and preferences in relation to online gendered harassment.
• Iterative development: The AI tool is being developed through ongoing internal and external evaluations.
• Public hackathon: Planned to explore future deployments and broader application of the technology across various digital platforms.
• Co-design with end users: Young women and gender-diverse individuals aged 14-25 are actively involved in shaping the tool’s features and functionality.

Project leads:

• Dr Eduardo Oliveira, Senior Lecturer, School of Computing and Information Systems, University of Melbourne
• Dr Lucy Sparrow, School of Computing and Information Systems, University of Melbourne
• Dr Mahli-Ann Butt, Lecturer in Cultural Studies, University of Melbourne

This project raises awareness of the target cohorts’ experiences while also developing assistive tools to combat high rates of online harassment in this group in Australia. The tool provides users with an extra line of moderation support in what can be increasingly incendiary and unsafe online platforms.

Key outputs and dissemination:

• Presentation at the Australian and New Zealand Communication Association conference (November)
• Public hackathon to explore broader application across various digital platforms
• Peer-reviewed publications on co-design methodology and NLP detection approaches
• Open-source documentation for potential replication by other organizations

• Co-design as a quality indicator: Involving 230+ end users in development ensures the tool reflects actual needs. Evaluators should assess the depth and representativeness of co-design processes.
• Trauma-informed approach: AI Ally prioritizes user safety and autonomy over punitive measures. M&E frameworks for technology-facilitated support should measure psychological safety and user empowerment, not just technical outputs.
• Opt-in, user-controlled design: Users decide whether to activate monitoring and how to use evidence. Evaluators should track user control metrics and opt-in rates as indicators of trust.
• Explanability as a feature: The tool explains why content was flagged, building user understanding and trust. M&E should assess whether users find explanations helpful and actionable.
• From Discord to other platforms: The prototype’s success could lead to broader application. Evaluators should track platform expansion and cross-platform usability.
• Government funding as validation: The eSafety Commissioner’s investment signals policy priority. Evaluators can consider government funding as a process indicator of national relevance.

Key takeaways for M&E and development practitioners

AI Ally demonstrates that effective anti-harassment technology must prioritize victim empowerment, user control, and explanatory transparency — not just perpetrator punishment. For evaluators, the key indicators are not just detection accuracy but: user trust, opt-in rates, evidence utilization for reporting, and qualitative feedback on the trauma-informed experience. The co-design methodology offers a replicable framework for developing technology-facilitated gender-based violence interventions with and for vulnerable populations.

Advance your skills in AI-enhanced evaluation and gender safety

Learn how to integrate co-design methodologies, trauma-informed approaches, and gender-responsive AI standards into your M&E practice. Explore our specialized courses and certifications below.

Main references:
ANROWS (Australia’s National Research Organisation for Women’s Safety). AI Ally: Co-designing anti-harassment AI with girls, young women and gender-diverse individuals. Retrieved from anrows.org.au
University of Melbourne (2024-2025). AI Ally: AI-powered dashboard to combat online abuse. Research publications and conference presentations.
eSafety Commissioner (Australian Government). Preventing Tech-based Abuse of Women Grants Program.

Disclaimer: Analysis and critique by EvalCommunity. This case study is an independent analysis for educational and policy guidance purposes. EvalCommunity does not claim ownership of the AI Ally project, University of Melbourne research, or eSafety Commissioner funding. Readers are encouraged to consult the original sources for complete methodology and findings.

Published by EvalCommunity Academy. Based on ANROWS project page, University of Melbourne research, and eSafety Commissioner public information.