EvalCommunity Academy · Case Study

When AI Classifies the Enquiry: What Should Evaluators Measure?

A documented case of AI-powered helpdesk ticket assignment — and what it teaches evaluators about accuracy, coverage, exceptions, human oversight and changing performance.

1. The Case at a Glance

The documented case concerns an end-to-end system that automatically assigns incoming helpdesk email tickets to appropriate resolver groups. The system was developed for large-scale service environments where manual dispatch can affect how quickly tickets are resolved.

The authors report that the system was deployed in production across six major service providers in diverse service domains. It was assigning approximately 100,000 emails per month, with accuracy close to 90% and coverage of at least 90% of email tickets. They also report net savings of more than 50,000 hours of effort per year and more than two million tickets served in production.

2. The Problem: Getting the Ticket to the Right Team

Service requests can arrive through different channels, including email. In the system described in the paper, email tickets were generated and then assigned to an appropriate service team and resolver group.

The authors identify ticket dispatch as an important step because incorrect assignment can increase turnaround time. They also describe several challenges: organizations may have hundreds of resolver groups, groups may have overlapping responsibilities, email language is heterogeneous and informal, and service organizations change over time.

In other words, what looks like a simple classification problem becomes much more complicated when it has to operate continuously in a large organization.

3. What the AI System Did

The system used a combination of machine-learning classifiers, a rule engine and retraining strategies.

• An ensemble classification system predicted the resolver group from unstructured email text.
• Additional classifiers predicted mandatory ticket fields such as category and priority or severity.
• A separate strategy addressed frequently occurring and sparse resolver groups.
• A few-shot learning approach was used for rarely occurring classes.
• A rule engine captured domain-specific knowledge needed for disambiguation.
• A retraining strategy was used to respond to changes in email language and resolver-group organization.

The paper therefore presents AI as an end-to-end operational system, rather than simply as one machine-learning model.

4. The Long-Tail Problem

Large service organizations may have many resolver groups, but the groups do not necessarily receive the same number of tickets. Some are frequent; others have relatively few examples.

The authors divide the problem into a frequent short head and a sparse long tail. Their system uses an ensemble classifier for the short head and a few-shot learning strategy based on Siamese networks for the long tail.

This matters for evaluation because an overall performance figure can conceal differences between common and uncommon categories.

5. What Happened in the Documented Deployment?

The paper reports production deployment across six major service providers and approximately 100,000 emails assigned per month on average. The authors report accuracy close to 90%, coverage of at least 90%, and more than two million tickets served in production.

The experimental results also use three real customer datasets. The largest dataset contained more than 700,000 emails and as many as 428 resolver groups. The authors report human-level accuracy with more than 90% coverage across the datasets.

Evidence note: The paper uses several related performance concepts, including classifier accuracy, classifier coverage, assignment-engine accuracy and assignment-engine coverage. The headline figures should therefore be read in the context of the definitions provided by the authors.

6. What Happens When the AI Is Not Confident?

This is an especially important part of the documented system.

When the classification system produces a confidence score above a configured threshold, the ticket is routed to the predicted resolver group. When the confidence score does not meet the threshold, the ticket is sent to a manual queue for inspection by a human agent.

This is highly relevant to M&E: a responsible AI workflow can define conditions under which the system stops, escalates or requires human review.

7. Is 90% Accuracy Enough?

For an AI developer, accuracy is an important performance measure. For an evaluator, it is only one part of the story.

A 90% accuracy figure does not, by itself, tell us:

• whether errors are concentrated in particular categories;
• whether rare cases perform differently from frequent cases;
• how costly an incorrect assignment is;
• how long incorrectly assigned tickets take to resolve;
• whether particular categories require more human intervention; or
• whether performance changes as the operating environment changes.

These are EvalCommunity evaluation questions inspired by the documented case. They are not claims that the original paper answered all of them.

8. Accuracy, Coverage and Outcomes Are Different Things

The case makes an important distinction between whether the system is correct and how much of the workload it can handle automatically.

For an evaluator, three questions should therefore be kept separate:

1. Accuracy: When the system makes a classification, how often is it correct?

2. Coverage: What proportion of cases can the system assign automatically?

3. Outcome: Does using the system improve the service outcome that matters?

9. AI Performance Changes Over Time

The authors explicitly identify change over time as a challenge. Resolver groups can be split, merged or renamed, while the problems assigned to groups can also change. New terminology can therefore reduce the performance of a previously trained model.

The system addresses this through retraining, including a sliding-window approach combined with active learning, as well as automatic rule mining to help identify changes in resolver groups.

This produces a broader lesson for evaluators:

AI performance should be considered over time, not only at the moment of initial validation.

10. What Should an Evaluator Measure?

This framework is an EvalCommunity Academy educational adaptation. It is not presented as an evaluation framework in the original paper.

11. Applying the Case to M&E and International Development

Now consider an organization receiving thousands of enquiries from programme participants, partners or communities.

AI could potentially classify enquiries into categories such as information requests, programme-access problems, administrative issues, complaints or technical problems.

But classification would not itself be the programme outcome.

The evaluator would need to examine what happens after classification:

• Does the enquiry reach the appropriate team?
• Does the response happen faster?
• Are errors detected and corrected?
• Are unusual cases identified?
• Can people obtain human assistance when needed?
• Does the process produce better outcomes for users?

This is the difference between evaluating an AI component and evaluating an AI-enabled service process.

12. The AI Classification Evaluation Chain

Enquiry → Classification → Routing → Response → Resolution → User Outcome

An evaluator can examine each stage of this chain. This prevents the AI model from becoming the sole focus of the evaluation.

For example, an AI classification can be technically correct while a later organizational process still causes delays. Conversely, a classification error may be corrected quickly by a human and have little effect on the final outcome.

13. What Could Go Wrong?

Rare cases may perform poorly. Overall performance can hide weaknesses in low-frequency categories.

Incorrect routing can create delays. A classification error can affect the subsequent service process.

The environment can change. New terminology, new problems and organizational changes can reduce model performance.

Errors can have unequal consequences. A wrong classification may be relatively minor in one category but much more consequential in another.

Automation can become over-trusted. This is why monitoring, review and clear responsibility matter.

The risks above are evaluation implications and applications of the documented case, not additional findings claimed by the authors.

14. A Practical Evaluation Checklist

Before deployment

• Define the service problem and intended outcome.
• Establish a baseline.
• Identify important and high-consequence categories.
• Define acceptable performance levels.
• Define when human review is required.

During implementation

• Monitor accuracy and coverage.
• Track incorrect assignments.
• Examine rare and emerging categories.
• Monitor human review and correction.
• Measure downstream service outcomes.

Over time

• Check for performance changes.
• Review changes in the operating context.
• Update training data and models when appropriate.
• Review new categories and terminology.
• Document lessons and corrective actions.

15. Connecting the Case to Responsible AI in M&E

Several lessons from this documented case connect closely with EvalCommunity’s responsible-AI guidance for M&E.

Principles for AI Use in Monitoring & Evaluation →

The EvalCommunity framework emphasizes purpose-driven AI use, data quality, transparency, equity, human oversight, continuous learning, accountability and appropriateness.

The case provides a useful real-world example of why these principles matter: the documented system was designed around accuracy and coverage while also incorporating domain-specific rules, a manual queue for low-confidence tickets and retraining for changing conditions.

16. Practice Exercise

Scenario: Your organization receives 20,000 programme-participant enquiries each year. Management wants to introduce AI to classify and route them automatically.

Before approving the system, develop an evaluation plan addressing:

1. What is the baseline?
2. How will AI accuracy be measured?
3. How will coverage be measured?
4. Which cases require human review?
5. How will rare or unusual cases be monitored?
6. What service outcomes should improve?
7. How will performance be monitored over time?
8. What happens when the system is wrong?

The objective is not to prove that AI is good or bad. It is to determine whether, where and under what conditions the AI-supported process creates value.

17. What This Case Teaches Evaluators

1. Measure more than accuracy. Accuracy is important, but it does not describe the entire service process.

2. Look at coverage. Understand how much of the workload is handled automatically.

3. Examine the long tail. Frequent cases can dominate aggregate statistics.

4. Evaluate the workflow. Classification is one stage in a larger service pathway.

5. Expect change. AI performance can be affected by changes in language, categories and organizational structures.

6. Design for appropriate human intervention. The documented system sends low-confidence tickets to a manual queue rather than automatically assigning every case.

18. The Bigger M&E Lesson

The question is not simply whether AI can classify enquiries.

The documented case demonstrates that AI-supported ticket assignment can operate at substantial scale. But the evaluator’s responsibility is broader.

We need to understand whether the AI-enabled process produces better outcomes, for whom, under what conditions, at what cost and with what risks.

19. Continue Learning with EvalCommunity Academy

How to Use AI in Evaluation Practice →
Practical guidance on using AI across evaluation work while maintaining human review, evidence traceability and professional judgment.

Principles for AI Use in Monitoring & Evaluation →
Guidance covering purpose, transparency, data quality, equity, human oversight, adaptation and accountability.

AI Standards for Monitoring & Evaluation →
A broader framework for responsible and accountable AI implementation in M&E.

AI in Monitoring & Evaluation Certificate →
A practical course covering AI-supported evaluation design, analysis, reporting, ethics, bias, transparency and human oversight.

20. Original Source

Evidence boundary: The documented facts about the system, production deployment, datasets, classification approach, accuracy, coverage, manual queue, retraining and reported efficiency results come from Agarwal et al. (2020). The M&E questions, indicators, development-sector applications and evaluation recommendations are EvalCommunity Academy’s educational interpretation and should not be attributed to the original authors.