NLP and AI to Evaluate Climate Adaptation in Bangladesh – Case Study
Bridging Qualitative Depth and Quantitative Scale: Using NLP and AI to Evaluate Climate Adaptation in Bangladesh
What is this case study about?
This case study documents an innovative mixed-methods evaluation conducted in cyclone-prone communities in Southern Bangladesh to understand how households adapt to repeated climate shocks—specifically cyclones, floods, and droughts—and the role of beliefs, expectations, and local knowledge in shaping their responses. The study was led by Prabhmeet Kaur Matta (PhD student, UC San Diego), Rocco Zizzamia (University of Oxford), and Anindita Bhattacharjee (BRACT Institute of Governance and Development, BIGD), in partnership with BRAC.
What makes this evaluation distinctive is its methodological integration of AI tools to process and analyze a large volume of qualitative data—something traditionally difficult to scale. By embedding open-ended questions into a structured survey with 860 households, the team collected 645 hours of audio recordings. They then used AI for transcription, translation, and NLP-based analysis to identify patterns in climate adaptation strategies that quantitative methods alone could not capture.
Why was this approach necessary?
In climate adaptation research, quantitative surveys can measure the frequency of certain behaviors (e.g., "how many households evacuated?"), but they often miss the why and how—the narratives, trade-offs, and contextual factors that shape decisions. Open-ended questions capture this depth, but analyzing them at scale is traditionally resource-intensive and time-consuming. As the research team noted: productive adaptation is largely overlooked in quantitative surveys, and migration—a key adaptation strategy—is often mischaracterized.
Study context and sample stratification
The study was conducted in cyclone-prone communities in Southern Bangladesh, a region highly vulnerable to climate-driven disasters. The sample of 860 households was strategically stratified across three groups to enable comparative analysis:
- BRAC Ultra Poor Graduation Programme (UPGP) recipients: Poorer, female-headed households, less likely to own land, fewer cows, low savings
- Microfinance clients: Approximately 1/3 male-headed, own land, high savings but also high debt
- Non-BRAC/non-microfinance households: Wealthier, own land and cows, approximately half male-headed
This stratification allowed the team to examine how social protection programs and financial inclusion shape climate adaptation behaviors.
What AI methods and tools were used?
🎙️ AI Transcription & Translation
Initial fully automated transcription using voice.bangla.bd failed, so the team used computer-assisted transcription followed by manual transcription, with automated translation using Gemini.
📊 Topic Modeling
Two approaches: Latent Dirichlet Allocation (LDA) and Structural Topic Modeling (STM) to identify prevalent themes across interviews.
💬 Sentiment Analysis
Combination of TextBlob (lexicon-based) and ClimateBERT (transformer model), with manual validation to ensure cultural accuracy.
🧑💻 Human-in-the-Loop Validation
AI outputs were systematically reviewed by researchers to ensure cultural and contextual accuracy, with bias mitigation through validation against human coders.
AI governance and ethical considerations
The team implemented robust safeguards for AI use:
- Privacy: De-identification of data before AI processing; use of institutional licenses preventing training data reuse
- Transparency: Full documentation of AI's role in processing
- Bias mitigation: Validation against human coders; transparent reporting of where AI failed
Key findings: What did the AI analysis reveal?
The combination of NLP techniques allowed the team to identify patterns in qualitative data that would have been difficult—if not impossible—to detect through traditional quantitative methods alone.
The adaptation puzzle: Quantitative vs. qualitative disconnect
If the team had relied solely on the quantitative survey, they would have concluded that 64% of households reported doing nothing to adapt, with only 15.5% investing in flood-proofing and 6.2% evacuating their residence. However, NLP analysis of qualitative responses revealed a very different picture: 70.3% of respondents who answered "no" to adaptation measures in the quantitative survey actually described adaptation measures in their open-ended responses.
This disconnect highlights a critical lesson for evaluators: quantitative surveys systematically underestimate adaptation behaviors because they often capture only formal, protective measures while missing productive, livelihood-based adaptations.
Productive vs. protective adaptation
The NLP analysis enabled the team to distinguish between two distinct types of adaptation:
🛡️ Protective adaptation
Non-productive, purely protective measures (e.g., flood-proofing, evacuation). Costs are borne up-front, liquidity constraints bind. These are typically captured in quantitative surveys.
🌾 Productive adaptation
Changes in the productive elements of a household's livelihood portfolio. Costs come as risk-reward trade-offs. These are multi-causal, may emerge through "natural selection" rather than design, and are picked up primarily through qualitative methods.
Migration as a key productive adaptation strategy
NLP topic modeling and sentiment analysis revealed critical insights about migration that quantitative surveys missed:
- Permanent household migration: No respondents spontaneously mentioned permanent migration; few aspirations for permanent migration, though some openness to permanent migration of children. For landed households, reliance on land-based livelihoods disincentivizes migration; for unlanded households, liquidity constraints reduce migration.
- Seasonal circular migration: This emerged as a core livelihood strategy, with prime-aged (mostly) men migrating seasonally for work. This represents portfolio and spatial diversification—a productive adaptation strategy overlooked in quantitative surveys.
The analysis also revealed an important policy insight: place-based social protection can alleviate liquidity constraints but may introduce moral hazard for household migration (while not affecting temporary migration).
Cognitive clustering: Survey structure ≠ cognitive structure
Using cosine dissimilarity analysis, the team discovered that respondents' mental models do not align with survey structures:
- Insurance cluster: All nine insurance questions clustered together, isolated from adaptation thinking
- Climate cluster: Experiences, worries, and information formed the core mental model
- Livelihoods + migration cluster: Migration emerged as an economic strategy, tightly linked to livelihoods
This finding has profound implications for survey design: respondents do not organize their thinking around thematic sections but rather around integrated mental models of their lives.
Uniform exposure to climate shocks
The sentiment and topic analysis revealed near-universal experiences across all household types:
- 97% believe climate will change significantly by 2050
- 86% see climate change as a very serious threat
- 395 responses explicitly mentioned flooding without prompting
- Floods and cyclones rank among "life's biggest challenges," described as "devastating"
- Livelihoods are extremely vulnerable to disruption, with destruction of property and accumulation of debt as recurring themes
What were the challenges and adaptations?
The research team was transparent about the difficulties encountered, offering important lessons for evaluators considering similar approaches:
🔊 Transcription accuracy
Fully automated transcription using voice.bangla.bd failed. The team pivoted to computer-assisted transcription followed by manual transcription—a critical reminder that AI tools may fail in specific linguistic contexts.
🌍 Contextual sensitivity
Sentiment analysis models trained on Western data risked misinterpreting culturally specific expressions. Researchers recalibrated interpretations through manual validation and local knowledge.
⚙️ Translation complexity
Automated translation using Gemini worked, but required human review to ensure accuracy of nuanced concepts like "adaptation" and "resilience" in Bengali.
📚 Balancing scale and depth
While AI enabled breadth, the team ensured that deep dives into specific transcripts still occurred to preserve qualitative richness and validate AI findings.
Next steps: Prosodic analysis
The team is now extending their methodological innovation by exploring prosodics—the emotional and paralinguistic features of speech:
- Feature extraction including pitch contours (F0), speech rate (syllables/second), pause frequency and duration, intensity (loudness), and spectral features (voice quality)
- Correlation analysis with topics (migration, climate worry, adaptation)
- Comparison across household types
- Emotional state classification to capture the affective dimensions of climate adaptation
This innovative approach could capture the emotional texture of survey responses that semantic or quantitative components might miss.
Key lessons for evaluators and M&E practitioners
✅ AI enables new data sources
Open-ended questions can now be systematically analyzed at scale, enriching traditional surveys with narrative depth and revealing productive adaptation strategies overlooked by quantitative methods.
🧠 Human review is non-negotiable
AI accelerates processing but cannot replace contextual understanding, ethical judgment, and validation by skilled evaluators. Automated transcription may fail; human oversight remains essential.
🌐 Language barriers are falling
Advances in multilingual AI tools (like Gemini) make it possible to conduct rigorous mixed-methods evaluations in diverse linguistic contexts—but local language models still require development.
⚖️ Survey structure ≠ cognitive structure
Cosine dissimilarity analysis revealed that respondents organize knowledge differently than survey instruments. This has profound implications for questionnaire design.
📊 Quantitative surveys systematically underestimate adaptation
64% of respondents reported "no adaptation" in quantitative questions, yet 70% described adaptation measures in qualitative responses. Mixed methods are essential for accurate assessment.
🤝 Interdisciplinary collaboration is key
Success required expertise in evaluation, climate adaptation, AI/ML, and local cultural knowledge—highlighting the value of diverse, cross-institutional teams.
Frequently asked questions
How did the team use AI to analyze qualitative interviews?
What were the main benefits of using NLP in this evaluation?
Did AI replace human evaluators in this study?
What challenges did the team face with AI-based analysis?
What is the main takeaway for M&E practitioners?
Main reference & original sources
📘 This case study is based on the presentation by Prabhmeet Kaur Matta, Rocco Zizzamia, and Anindita Bhattacharjee, as documented in:
1. WFP Evaluation (2026, March 23). Four examples of how AI, machine learning and data innovation are reshaping evaluation and research. Medium. https://wfp-evaluation.medium.com/four-examples-of-how-ai-machine-learning-and-data-innovation-are-reshaping-evaluation-and-research-f4d1b3cf4e74
2. Zizzamia, R., Matta, P. K., & Bhattacharjee, A. (2025, December). Adaptation to Climate Change in Cyclone-prone Bangladesh. Presentation slides. WFP Global Impact Evaluation Forum 2025. Download full presentation (PDF)
Resources for further learning
- WFP Evaluation – Medium Blog
- EvalCommunity – AI in M&E Course
- Download: Adaptation to Climate Change in Cyclone-prone Bangladesh (PDF)
- EvalCommunity Services & Resources
Advance your skills in AI-enhanced evaluation
Learn how to integrate NLP, sentiment analysis, and mixed-methods AI tools into your own M&E practice. Access courses, templates, and expert guidance.
