AI latency in production

The short answer

Users abandon slow flows regardless of how good the answer would have been. A correct response that arrives after someone left is worth nothing, which makes latency a product constraint rather than an infrastructure detail.

Set the budget from the flow, then design inside it. Doing the reverse produces a system that is accurate in testing and unused in production.

Where the time actually goes

Rarely one place. Retrieval, including the embedding call for the query. Time to first token from the model. Generation, which scales with output length. Tool calls, which are network hops into systems you do not control. And in multi step flows, all of that repeated.

Without per stage timing you will optimise the wrong thing, usually the model, because it is the visible part. The tracing needed to see this is in observability.

Streaming changes perception

Time to first token often matters more than total time. A response that begins in half a second and takes five feels faster than one that appears complete after three.

So stream wherever the interface allows it, and design the first sentence to be useful rather than preamble. This is one of the highest ratio improvements available, because it costs almost nothing and changes how the product feels.

Retrieval latency

Similarity search over a large index is not free, and the embedding call for the query adds a round trip before the search even starts.

Filtering before searching, by tenant or date or type, shrinks the candidate set and speeds it up. Caching embeddings for repeated queries helps. And an index tuned for recall at any cost will be slower than one tuned for the accuracy you actually need, a tradeoff described in RAG.

Model choice is a latency decision

Bigger models are slower. For steps where a smaller model performs acceptably, classification, extraction, routing, the latency gain compounds because those steps often sit on the critical path before the visible answer.

Measure quality when you do this. Trading accuracy for speed silently is how a fast, subtly worse system ships without anyone deciding to.

Parallelise what you can

Many pipelines are sequential out of habit rather than necessity. Retrieval from two sources, several independent tool calls, or fetching user context while embedding the query can often run concurrently.

The constraint is genuine dependency: only serialise where a step needs the previous result. Auditing a slow flow for false sequencing frequently finds seconds nobody knew were there.

Timeouts, fallbacks and degradation

Everything in this stack can hang. Model providers have slow days, tool calls stall, retrieval blocks. Without timeouts, one slow dependency becomes a slow product.

Decide what happens on timeout before it happens: a smaller model, a cached answer, a graceful message, a human handoff. Degrading deliberately is a feature. Hanging is not.

Multi step flows multiply everything

In agentic systems, latency is the sum of the loop. Twelve steps at eight hundred milliseconds is ten seconds before anything reaches the user.

If the flow is user facing, cap the steps and show progress. If it genuinely needs many steps, move it off the synchronous path entirely and notify when done, rather than making someone watch a spinner.

Setting a latency budget

Write the number down from the product, not from the infrastructure. Inline assistance during typing is a different budget from a search result, which is different from a report generated on request.

Then allocate it across stages and measure against it, the same way you would measure quality. A budget nobody tracks is a preference. That discipline is part of how we build production AI, end to end.