AI for customer service

Most enterprises have tried AI in customer support at least once. A chatbot went live, handled the easy questions, and was quietly deprecated when ticket volume did not move. The failure was not the technology. It was the architecture: what the system was asked to do, where it sat in the workflow, and whether there was any feedback loop connecting output quality to deployment decisions.

This post covers what it takes to build AI into support that actually reduces ticket volume and improves resolution time, rather than routing complexity around in circles.

Why most support chatbots fail

The dominant failure mode is not that AI cannot answer support questions. It is that the system is built to answer the easy questions, and those are not the questions driving ticket volume. The genuinely expensive tickets are ambiguous, require accessing multiple systems, involve policy exceptions, or need a customer to be made whole emotionally, not just procedurally. A chatbot that deflects the simple twenty percent and routes the complex eighty percent has not solved the problem.

The other failure mode is measuring deflection without measuring resolution quality. A system that closes tickets without actually resolving them inflates its own metrics while degrading customer trust. Customers who feel dismissed by an AI are more expensive to retain than customers who waited for a human.

The third failure mode is treating this as a point solution. AI in support has to connect to ticketing systems, knowledge bases, CRM records, and in some cases product backends. A system that cannot look up an account, check an order status, or verify a policy term is limited to questions that do not require any of that. Those are rarely the bottleneck.

The right starting point: ticket classification

Before building anything that generates responses, build something that classifies inbound tickets by type, urgency, and required action. This single step produces immediate value: it routes tickets to the right team without manual sorting, it surfaces the actual distribution of ticket types which is usually surprising, and it creates the labeled dataset you need to evaluate any generative system you build later.

Classification is also a lower-risk starting point than response generation. If the classifier makes an error, a human sees the ticket and routes it manually. If a generative system makes an error, the customer may have already acted on wrong information. Building trust in the AI layer starts with tasks where errors are recoverable.

Once classification is running in production, you have a real picture of what the ticket mix actually looks like. In most enterprise support contexts, the top five ticket types account for 40 to 60 percent of volume. Those are the candidates for automation in the next phase, and the classification data tells you which ones are formulaic enough to automate safely.

Retrieval-based vs generation-based responses

For most support use cases, the best architecture retrieves a relevant help article or policy document and surfaces it to the customer, rather than generating a novel response. Retrieval is safer because the output is a document you wrote and approved, not something the model composed from patterns. Generation is appropriate when the question requires synthesizing information from multiple sources or when you have enough volume and evaluation infrastructure to catch failures. We covered the underlying architecture in RAG.

The practical distinction matters for how you handle errors. A retrieval system that returns the wrong document is wrong in a way that is easy to audit and fix: you find the document, update it, and the answer improves. A generation system that hallucinates a policy term requires identifying that a hallucination occurred, understanding what triggered it, and either improving the prompt, the grounding data, or adding an explicit guardrail. The failure loop is longer and harder to close systematically.

Start with retrieval for questions that have documented answers. Move to generation only for question types where retrieval consistently fails to surface the right material and where you have the evaluation tooling to catch generation errors before customers see them.

The handoff problem

Every AI support system eventually hands a conversation to a human. How that handoff works determines whether customers feel the AI helped or obstructed them. The failure mode is transferring the conversation without the context: the human agent starts from scratch, the customer has to repeat themselves, and the frustration compounds. The AI should hand off a structured summary of what was already established, what was tried, and why escalation was triggered. The agent experience matters as much as the customer experience.

This requires designing the handoff explicitly rather than treating it as a fallback. What information does a human agent need to resolve this ticket without asking the customer to repeat themselves? How should that information be formatted so an agent can scan it in ten seconds? What escalation reasons should trigger a warm transfer versus an asynchronous queue? These decisions are product decisions, not engineering decisions, and they need to be made before the system is built, not after the first escalation goes wrong.

The AI should also know when to stop trying. A system that attempts three different approaches before escalating is worse than one that escalates after the first failure if the customer is clearly frustrated. Sentiment detection and explicit customer requests to speak to a human need to take priority over the system's confidence in its next response attempt.

What production looks like after six months

At launch, the system handles the most common, well-documented question types. Over the first two months, the evaluation feedback loop surfaces gaps in the knowledge base, cases where the AI gave a confidently wrong answer, and ticket types where customers systematically reject the AI response and request a human. Over months three through six, the team works through those gaps systematically. The deflection rate that is realistic after six months of this work is 30 to 50 percent of inbound volume, not the 80 percent in vendor demos.

The gap between vendor demo numbers and production numbers is almost entirely explained by two things. First, demo environments use curated question sets, while production receives the full distribution of how customers actually phrase their problems, including ambiguous phrasing, mixed-language inputs, and questions that span multiple topics in a single message. Second, demo environments do not include the long tail of edge cases that a live knowledge base has not yet covered. Every undocumented policy, every product variant that was not in the training data, every edge case that support agents handle by asking a colleague is a ticket the AI system cannot handle until someone documents it.

A realistic roadmap acknowledges this. The first six months are not just deployment. They are an ongoing process of identifying gaps, closing them, and measuring whether each closure moved the metrics.

Metrics that matter

Deflection rate measures how often the AI resolves a ticket without human involvement. But deflection is only valuable if the resolution is correct. Track CSAT on AI-handled tickets separately from human-handled tickets. Track escalation rates and the reasons for escalation. Track first-contact resolution. A system with 60 percent deflection and poor CSAT on those tickets is creating more problems than it solves. We covered the evaluation approach in LLM evals.

Time-to-resolution is more informative than deflection rate for understanding business impact. A ticket that the AI handles in three minutes but incorrectly, requiring a follow-up human interaction, has a longer true resolution time than a ticket a human handled in twelve minutes the first time. Measure the full resolution loop, not just the AI's portion of it.

Track the reasons customers request a human. If a large fraction of escalations happen immediately, before the AI has attempted a resolution, that signals trust failure. Customers who have been burned by AI support before will try to bypass it. Rebuilding that trust requires demonstrating accuracy on simpler questions before being given the chance to answer harder ones. The escalation data tells you how much trust you are starting with.

Guardrails for support contexts

Support AI operates in a context where errors directly damage customer relationships. The guardrail requirements are tighter than for internal tools. The system should refuse to answer questions it has low confidence on and offer escalation. It should never fabricate policy details, pricing, or account-specific information. It should not attempt to retain customers who want to cancel by misrepresenting terms. These constraints need to be evaluated explicitly and regularly, not assumed. We covered the broader topic in AI guardrails.

Hallucination risk in support contexts is particularly acute because the model has access to product and policy information that changes over time. A pricing change, a policy update, or a product discontinuation that is not reflected in the knowledge base creates a category of error the model cannot detect: it answers confidently from outdated information. Knowledge base freshness is not an AI problem, but it becomes the AI's problem when the system is generating responses based on stale data. Audit trails that show when each document was last updated, and alerts when documents have not been reviewed in a defined period, are part of the production infrastructure, not an afterthought.