EvalCommunity Academy · Case Study

From Manual Triage to AI-Powered Service

The University of South Florida case — and what evaluators can learn about automation, cost, performance, explainability and organizational change.

1. The Case at a Glance

The University of South Florida developed an AI-Powered IT Service Desk to automate the classification and routing of IT service tickets.

EAB reports that student workers previously manually sorted approximately 100,000 IT service tickets annually. The AI system classifies incoming tickets according to status, service team, issue type and priority. EAB identifies the application as Deployed.

The original USF presentation was delivered by Jason Hair at the 2023 EDUCAUSE Annual Conference under the title “Revolutionizing the IT Service Desk: GPT-Powered Ticket Classification at USF.” EDUCAUSE describes the system as using GPT models to classify tickets and improve resource allocation and efficiency.

2. The Problem

USF faced a high-volume, repetitive classification task. Student workers manually sorted a large number of incoming IT tickets before those requests could be directed to the appropriate service team.

The operational question was therefore simple:

EAB describes the intended benefit as speeding response and eliminating the manual effort involved in sorting IT help desk tickets.

3. What the AI Does

The system classifies incoming tickets according to four documented dimensions:

• Status
• Service team
• Issue type
• Priority

EAB also reports that the system provides a detailed explanation of how it classified a ticket, allowing service teams to examine the classification. The documented technology stack includes Azure OpenAI, ChatGPT-3.5 Turbo, OpenAI Embeddings, a vector database and Microsoft Semantic Kernel components.

The important point for evaluators is that the intervention is not simply “using ChatGPT.” It combines AI + organizational data + infrastructure + workflow integration + people.

4. The Business Case

EAB reports that USF estimated the annual cost of the AI-powered Service Desk at approximately $21,800. The documented estimate included prompt maintenance, Azure OpenAI-related costs and the WebServices virtual machine.

EAB also reports an estimate that the operating cost could fall from approximately $90,000 to $21,800 annually.

But this is an estimate, not a measured post-deployment saving.

This distinction is particularly important for evaluators. A projected financial benefit becomes an evaluation finding only after the relevant costs and outcomes have actually been measured.

5. The Evaluator’s First Question

If management says:

“The AI system will save $68,200 per year.”

the evaluator should ask:

• How was the baseline cost calculated?
• Which costs are included?
• Which costs are excluded?
• What assumptions were made about ticket volume?
• What human labor is actually being replaced or reassigned?
• What will the AI system actually cost after deployment?
• Will service quality change?
• Will users experience an improvement?

This turns a business-case claim into an evaluation question.

6. From Efficiency to Outcomes

The business case naturally focuses on efficiency:

AI classification → Less manual work → Potential cost reduction

An evaluator should extend the chain:

AI classification → Routing → Service response → Resolution → User outcome

The first part is the documented purpose of the USF system. The later stages are evaluation questions that should be tested rather than assumed.

7. How Evaluators Can Apply This: A Practical Example

Imagine that an international development organization receives 100,000 programme-support enquiries every year. Management proposes an AI system similar to the USF model to classify and route them.

Instead of beginning with the question “Which AI tool should we buy?”, the evaluator can work through the following steps.

Step 1 — Define the problem

Document the existing process.

Example: Programme staff spend substantial time manually reading, categorizing and forwarding participant enquiries.

Measure the baseline:

• number of enquiries;
• staff hours spent on classification;
• classification cost;
• response time;
• resolution time; and
• current routing errors.

Step 2 — Define what AI is actually supposed to do

Do not define the intervention as simply “use AI.” Specify the task.

Example: AI classifies each enquiry into information request, payment issue, programme access, complaint, technical problem or other.

Also define what the AI will not decide.

Step 3 — Establish performance measures

Before deployment, define how success will be measured.

• classification accuracy;
• correct routing;
• priority accuracy;
• percentage of cases requiring correction;
• response time;
• resolution time; and
• actual cost per enquiry.

Step 4 — Establish a human-review process

Decide in advance what happens when the AI is uncertain or when a case is sensitive.

Example: complaints involving safeguarding, serious harm or other high-consequence issues are automatically referred for human review rather than being handled solely through automated classification.

This is an EvalCommunity application example, not a claim about how USF handles such cases.

Step 5 — Compare AI with the baseline

After implementation, compare the new process with the original one.

Step 6 — Examine the exceptions

Do not evaluate only the average.

Look specifically at:

• rare enquiries;
• complex cases;
• high-priority cases;
• cases incorrectly routed;
• cases requiring human correction; and
• categories with systematically lower performance.

Step 7 — Evaluate the outcome

The final question is not “Did the AI classify tickets?”

Step 8 — Continue monitoring

Do not stop the evaluation when the system goes live.

Continue monitoring:

• accuracy;
• cost;
• error patterns;
• new types of enquiries;
• human workload;
• service outcomes; and
• user experience.

8. What Should Be Measured?

This is an EvalCommunity Academy evaluation framework developed from the case. It is not presented as a framework used by USF.

9. Explainability: Can People Understand the AI Decision?

The USF system is documented as providing detailed explanations of how tickets were classified.

For evaluators, the existence of an explanation is only the starting point.

Ask:

• Can staff understand the explanation?
• Does it help them detect errors?
• Can they challenge the classification?
• Does it support quality assurance?
• Does it improve appropriate human oversight?

Explanation available ≠ explanation effective.

10. What Could Go Wrong?

Misclassification: A ticket reaches the wrong team.

Incorrect priority: An important request receives an inappropriate priority.

Hidden workload: Manual sorting declines while monitoring and correction work increases.

Cost uncertainty: Actual AI costs differ from the original estimate.

Outdated information: Historical data no longer reflects current service structures.

Automation bias: Staff become overly reliant on AI classifications.

These are evaluation risks and questions derived from the case, not claims that USF experienced each of these problems.

11. The Bigger M&E Lesson

A business case tells you what an organization expects AI to achieve. Evaluation tells you what actually happened.

The USF example is valuable because it connects a real AI deployment with a concrete operational problem and a documented financial estimate.

For evaluators, the next step is to test the full chain:

Investment → Implementation → Output → Efficiency → Service → Outcome

The evaluator’s role is to determine where the evidence is strong, where assumptions remain, and what still needs to be measured.

12. Continue Learning with EvalCommunity Academy

How to Use AI in Evaluation Practice →

Principles for AI Use in Monitoring & Evaluation →

AI in Monitoring & Evaluation Certificate →

13. Sources