Open-ended responses can contain some of the most useful evidence in an evaluation. They can also produce a lot of work.

Survey comments, beneficiary feedback, interview transcripts and reflection questions can quickly add up to thousands of lines of text. For an evaluation team working under time and resource constraints, analysing all of it carefully is not always easy.

That was the starting point for Roy and Rambo-Hernandez’s study, published in the American Journal of Evaluation in 2021. Their question was straightforward: could sentiment analysis help evaluators make sense of a large amount of open text when time and analytical capacity were limited?

The answer was yes — with some important qualifications.

The original case

The study examined student reflections connected to an engineering education intervention. The activities addressed issues including diversity, teamwork and culture.

The researchers looked at three activities:

• Algorithmic Justice League — an activity about diversity, implicit bias and technology.
• Neuroplasticity — an activity about learning, persistence and different approaches to teamwork.
• Wage gap — an activity dealing with the gender wage gap in engineering.

The same reflection questions were used across the activities, which made the resulting open-text data suitable for comparison. :contentReference[oaicite:5]{index=5}

The wider programme generated a very large amount of potential text. The authors estimated that the university data could eventually contain around 72,000 individual responses. The particular analysis reported in the paper was based on 118 open-ended response sets from participating students. :contentReference[oaicite:6]{index=6}

Why use sentiment analysis?

The problem was not that qualitative analysis was considered unnecessary. Quite the opposite.

The authors recognised qualitative analysis as the established way of making sense of open-ended text. Their argument was more practical: when there is a lot of text, limited time and a need for relatively quick feedback, a computational approach can help an evaluator get through the material. :contentReference[oaicite:7]{index=7}

In the original analysis, the researchers used R together with text-analysis packages including tidytext, tokenizer and quanteda. They used tokenisation, lemmatisation, stop-word filtering and a bag-of-words representation, followed by TF-IDF and sentiment lexicons. :contentReference[oaicite:8]{index=8}

Three sentiment lexicons were used:

• AFINN
• Bing
• NRC Emotion

The researchers also used bigrams and trigrams when individual words were not enough to describe the sentiment reliably. :contentReference[oaicite:9]{index=9}

What did the analysis actually tell them?

This is where the case becomes more interesting than a simple positive/negative classification exercise.

For the Algorithmic Justice League activity, for example, the word bias appeared prominently in the responses from female students but not among the most common terms for male students. The pattern suggested a difference worth investigating and helped the researchers make recommendations about the activity. :contentReference[oaicite:10]{index=10}

The wage-gap activity produced another set of patterns. Terms related to discrimination and inequality were associated with negative emotions, while different groups showed somewhat different associations with the issue. The researchers used these findings to suggest changes to the activity and its reflection questions. :contentReference[oaicite:11]{index=11}

That is an important distinction.

The analysis was useful because it helped the researchers decide where to look more closely. It was not useful because a computer had somehow discovered the final meaning of the students’ responses.

The time-saving argument

The researchers reported that around 10 hours were spent developing the R analysis. Once the code existed, running the analysis and producing the visualisations took less than a minute. :contentReference[oaicite:12]{index=12}

The bigger advantage was repeatability.

If the same questions were asked again in another semester, the analytical workflow could be reused. That makes computational text analysis particularly attractive for monitoring systems that collect similar feedback repeatedly.

For an M&E team, this is still a useful idea in 2026.

Instead of thinking only about whether AI can analyse one dataset, it is often more useful to ask:

Can we build a reliable process that can be repeated every time new feedback arrives?

What has changed since 2021?

The basic problem has not changed. The tools have.

In 2021, the case focused on lexicons, bag-of-words methods, TF-IDF and R-based NLP. Those approaches are still useful, particularly when transparency and reproducibility matter.

Since then, sentiment analysis has developed in several directions.

Recent research shows that aspect-based sentiment analysis has become a major area of development, including work with transformer models and LLMs. A 2026 systematic review describes the field’s movement from traditional machine-learning approaches toward transformer and LLM-based methods, while also highlighting continuing challenges such as implicit aspects, sentiment drift and multilingual data. :contentReference[oaicite:13]{index=13}

A useful development: aspect-based sentiment analysis

For many M&E applications, knowing that a response is “negative” is not enough.

An evaluator usually wants to know what the participant is negative about.

Consider this response:

“The health workers were very helpful, but the waiting time was too long and the medicines were sometimes unavailable.”

A basic sentiment analysis might classify the overall response as mixed or negative.

An aspect-based analysis can provide something more useful:

• Health workers: positive
• Waiting time: negative
• Medicine availability: negative

That is much closer to the questions an evaluation team may actually need to answer.

Recent research continues to develop ABSA for multilingual and cross-lingual settings. A 2025 survey found that the field still faces difficulties with limited labelled data and language-specific expressions, particularly when transferring models from resource-rich languages to lower-resource languages. :contentReference[oaicite:14]{index=14}

The multilingual problem

This matters particularly in international development and humanitarian work.

Beneficiary feedback may be collected in several languages. A model that performs well in English should not automatically be assumed to perform equally well in another language.

The problem is not only translation. Slang, cultural references, expressions and local ways of communicating emotion can all affect classification.

Cross-lingual ABSA research is therefore looking at multilingual pretrained models, cross-lingual transfer and other ways of working with limited labelled data. But the evidence also suggests that this remains an active research problem rather than a solved one. :contentReference[oaicite:15]{index=15}

Where do LLMs fit?

LLMs make this type of analysis much easier to try.

An evaluator can provide a set of responses and ask an LLM to classify sentiment, identify emotions, extract themes or identify the programme components being discussed.

That is a significant change from the 2021 workflow, which required programming and carefully selected lexicons.

But easier does not necessarily mean more reliable.

LLM results can vary depending on the model, instructions, examples and analytical design. For evaluation work, this means that an LLM should be treated as part of the method, not as the method itself.

Recent research illustrates the point. A 2026 study of ABSA in resource-scarce settings found strong results from fine-tuned open-source LLMs, while few-shot prompting was promising but generally less effective than fine-tuning. :contentReference[oaicite:16]{index=16}

For an evaluator, the lesson is fairly simple: there is no single “best AI” for sentiment analysis. The appropriate approach depends on the question, data, language, scale and consequences of the decision.

The problem of context has not gone away

This is probably the most important point to carry from the original study into today’s AI environment.

The authors acknowledged that some terms could be classified incorrectly because they were not always assessed in context. They also recognised that an unsupervised system did not understand the reasoning behind a participant’s sentiment. :contentReference[oaicite:17]{index=17}

Consider a simple example:

“The training was challenging, but that is exactly what made it useful.”

A word-based approach might associate “challenging” with negative sentiment.

A human reader can see that the overall judgement is positive.

Modern language models are much better at considering context, but they can still misunderstand sarcasm, mixed emotions, cultural references or indirect criticism.

Better language models reduce some problems. They do not remove the need for validation.

A better workflow for M&E in 2026

For an evaluation team today, I would not recommend simply asking an AI tool to “analyse the sentiment” of the entire dataset.

A more defensible workflow would look like this:

1. Start with the evaluation question. Decide what you actually need to learn from the text.
2. Prepare the data. Remove unnecessary personal information and establish appropriate data-protection safeguards.
3. Create a human-coded sample. Have evaluators classify a subset of responses before relying heavily on automation.
4. Choose the method. This might be a lexicon, machine-learning model, transformer, ABSA model or LLM.
5. Compare results with the human sample. Pay particular attention to disagreements.
6. Review difficult cases. Check mixed, ambiguous and unexpected responses.
7. Triangulate. Compare the findings with themes, quantitative indicators, interviews or other evidence.
8. Interpret the findings. Decide what the evidence means for the evaluation question and the programme.

What should evaluators validate?

• Context: Does the classification reflect the meaning of the whole response?
• Consistency: Are similar responses treated in similar ways?
• Language: Does performance change between languages?
• Groups: Could apparent differences reflect language or response style rather than genuine differences?
• Human judgement: Have important classifications been checked?
• Reproducibility: Could another evaluator understand how the analysis was conducted?

Sentiment is not the same as meaning

This distinction is easy to lose when working with AI.

A participant can express positive feelings about a programme without demonstrating that the programme produced its intended outcomes.

Likewise, a negative comment does not automatically mean that a programme failed. A participant may be unhappy with one aspect of a service while still experiencing positive outcomes overall.

Sentiment analysis can help answer questions about how people express their views.

It does not, by itself, answer questions about why outcomes occurred, whether change was attributable to the intervention, or whether the programme was effective.

What this means for evaluators

The original 2021 study remains useful because it identified a problem that has not disappeared: evaluators often have more text than they have time to analyse.

What has changed is the range of tools available to help.

Today, an evaluator can choose among relatively transparent lexicon-based approaches, supervised machine learning, transformer models, aspect-based methods, multilingual models and LLM-assisted workflows.

The choice should be driven by the evaluation question rather than by the novelty of the technology.

And the central principle from the original case still holds: computational analysis can help identify patterns, but interpretation and judgement remain part of the evaluator’s job. The authors themselves recommended treating automated results as indicators and using additional qualitative or quantitative analysis when the decisions involved are more consequential. :contentReference[oaicite:18]{index=18}

Questions to discuss with your M&E team

1. How much open-text data does your current M&E system collect but rarely analyse?
2. Which evaluation questions could benefit from sentiment analysis?
3. Would sentiment alone be enough, or would you also need themes and aspects?
4. How would you check whether an AI classification is correct?
5. Would the same approach work across all languages in your programme?
6. What would you do when the AI interpretation conflicts with the evaluator’s interpretation?

Continue learning with EvalCommunity Academy

If you want to move from individual AI techniques to practical AI-supported M&E workflows, the EvalCommunity Academy flagship certificates provide a useful next step.

Main reference

Roy, A., & Rambo-Hernandez, K. E. (2021). There’s So Much to Do and Not Enough Time to Do It! A Case for Sentiment Analysis to Derive Meaning From Open Text Using Student Reflections of Engineering Activities. American Journal of Evaluation, 42(4).

Read the original article

Selected contemporary references

• Šmíd, J., & Král, P. (2025). Cross-lingual Aspect-Based Sentiment Analysis: A Survey on Tasks, Approaches, and Challenges. Information Fusion, 120, 103073. :contentReference[oaicite:19]{index=19}
• Shukla, P., Kumar, R., Dwivedi, V. K., & Singh, A. K. (2026). Aspect based sentiment analysis: A systematic review, taxonomy, applications, and future research directions. Computers & Security. :contentReference[oaicite:20]{index=20}
• Fehle, J., Kruschwitz, U., Hellwig, N. C., & Wolff, C. (2026). Leveraging fine-tuning of large language models for aspect-based sentiment analysis in resource-scarce environments. Knowledge-Based Systems, 336, 115277. :contentReference[oaicite:21]{index=21}