AI survey response analysis case study
EvalCommunity Case Study
How The Football Association Used AI to Analyze 3,300+ Open-Ended Survey Responses in Hours, Not Days
A real-world case study on using AI-powered text analysis to process large volumes of open-ended feedback, identify themes, segment responses, and support action-oriented reporting.
Case Study Summary
The Football Association, the governing body for football in England, needed to analyze more than 3,300 open-ended survey responses from grassroots football participants.
The feedback focused on discipline reporting, discriminatory abuse, physical misconduct, barriers to reporting, and advice for improving reporting processes. To manage the volume and complexity of the data, The FA used Caplena’s AI-powered text analysis platform.
Main Source
This EvalCommunity case study is based on Caplena’s original customer story about The Football Association. Readers should review the original source for full details.
Original source: The Football Association customer story by Caplena
At a Glance
3,300+
Open-ended survey responses analyzed
Hours
Instead of days of manual work
2
Key open-ended questions analyzed
AI + Human
AI supported human insight work
1. Why This AI Survey Analysis Case Study Matters
Open-ended survey responses are valuable because they capture people’s experiences, concerns, explanations, and recommendations in their own words. However, when organizations receive hundreds or thousands of responses, manual analysis can become slow and difficult to complete within reporting deadlines.
The Football Association case is useful for evaluators, M&E teams, researchers, and stakeholder engagement professionals because it shows how AI can support large-scale qualitative feedback analysis while still requiring human interpretation, validation, and ethical judgment.
2. The Context: Grassroots Football Feedback
The Football Association oversees football in England and plays an important role in grassroots football. The survey feedback came from people involved in grassroots football and focused on discipline reporting and behavior.
The topic was sensitive because the responses addressed discriminatory abuse, physical misconduct, reporting behavior, confidence in reporting processes, and suggestions for improvement.
Important Note for Evaluators
When AI is used to analyze sensitive feedback, it should not replace ethical judgment, confidentiality safeguards, source checking, or human review of high-risk findings.
3. The Challenge: Thousands of Open-Ended Responses
Large Feedback Volume
The FA needed to analyze more than 3,300 open-ended responses. Reviewing and coding this volume manually would require significant time and resources.
Complex and Sensitive Themes
The responses covered abuse, misconduct, discrimination, reporting barriers, confidence, support, and recommendations for improvement.
Need for Segmentation
The FA needed to compare responses across different roles and experience levels to understand how different groups viewed the reporting process.
4. The Survey Questions Analyzed
The original case study identifies two key open-ended questions used in the analysis.
| Survey Question | Purpose | Responses |
|---|---|---|
| Are you more or less likely to report abuse now compared to 12 months ago? Why? | Understand reporting behavior and reasons behind changes in willingness to report abuse. | 3,331 |
| What further advice do you have for The FA and Kick It Out? | Collect practical recommendations for improving reporting and support processes. | 1,458 |
5. The AI-Supported Approach
To manage the volume and complexity of the responses, The FA used Caplena’s AI-powered text analysis platform. The approach helped identify key themes, compare audience segments, and reduce the time required for open-ended survey analysis from days to hours.
What AI Helped With
- Processing thousands of open-ended survey responses.
- Identifying recurring themes and patterns.
- Comparing feedback across audience segments.
- Supporting faster first-pass synthesis.
- Helping move from raw feedback to action-oriented insights.
6. Key Findings and Action Areas
The original case study reports that the AI-supported analysis helped The FA uncover key themes, understand differences across respondent groups, and identify practical improvements.
| Area | What the Analysis Supported | Why It Matters |
|---|---|---|
| Theme discovery | Identifying recurring issues across thousands of responses. | Large feedback datasets become easier to interpret and act on. |
| Segmentation | Comparing themes across roles and experience levels. | Different stakeholder groups may experience the same system differently. |
| Reporting barriers | Understanding why some people may hesitate to report abuse or misconduct. | Barrier analysis helps improve trust, access, and responsiveness. |
| Action planning | Identifying improvements to reporting systems, awareness, and support. | Feedback analysis should support decisions, not only produce summaries. |
7. What Evaluators and M&E Teams Can Learn
Use AI for First-Pass Synthesis
AI can help organize large volumes of text quickly, but the first output should be treated as a draft for human review.
Analyze by Stakeholder Group
Segmentation helps reveal differences between roles, experience levels, regions, or respondent groups.
Validate Sensitive Themes
Themes related to abuse, discrimination, misconduct, retaliation, safeguarding, or trust require careful human review.
8. Example Workflow for AI-Assisted Survey Analysis
Step 1
Prepare Data
Clean responses and preserve response IDs.
Step 2
Protect Data
Remove unnecessary personal identifiers.
Step 3
Identify Themes
Use AI for first-pass theme discovery.
Step 4
Segment
Compare patterns by role or group.
Step 5
Validate
Check sensitive findings manually.
Step 6
Report
Share evidence-backed findings.
9. Responsible AI Lessons
- Use AI for scale: AI can help structure thousands of open-ended responses faster than manual review alone.
- Keep humans responsible: Human analysts must review, interpret, and validate the findings.
- Protect sensitive data: Remove unnecessary personal information before using AI tools.
- Preserve traceability: Keep response IDs linked to quotes, themes, and coded outputs.
- Review sensitive themes: Abuse, discrimination, misconduct, retaliation, and safeguarding themes require additional human review.
- Avoid overclaiming: AI-assisted findings should be described as first-pass analysis until validated.
10. Suggested AI Use Statement
AI was used to support the first-pass analysis of open-ended survey responses, including theme identification, draft coding, segmentation support, and preparation of initial summaries. Human analysts reviewed the AI-generated outputs, checked findings against source responses, revised the coding where necessary, and made final decisions about interpretation and reporting. Sensitive themes were subject to additional human review.
11. Quality Checklist for AI Survey Analysis
- Was the survey question clearly defined?
- Were all open-ended responses included?
- Were response IDs preserved?
- Were personal identifiers removed or protected?
- Were themes checked against source responses?
- Were differences across respondent groups interpreted carefully?
- Were low-frequency but high-risk issues reviewed?
- Were sensitive findings manually validated?
- Were quotes checked and anonymized before reporting?
- Were limitations documented?
- Was AI use disclosed transparently?
Conclusion
The Football Association case shows how AI can support large-scale open-ended survey analysis in a practical and action-oriented way.
By using AI-powered text analysis, The FA was able to process more than 3,300 responses related to grassroots football discipline reporting, discrimination, abuse, and misconduct. The process helped identify themes, compare audience segments, and reduce analysis time from days to hours.
For evaluators and M&E professionals, the lesson is clear: AI can accelerate first-pass survey analysis, but trustworthy findings still depend on human validation, ethical safeguards, source checking, and transparent reporting.
