What is AI observability?

The short answer

AI observability is being able to reconstruct, after the fact, exactly why a system produced a specific answer to a specific request. Not whether it responded, why it responded that way.

In conventional software, a request either worked or threw an error, and the error usually points at the cause. Here, the most expensive failures return HTTP 200 with a confident, well formatted, wrong answer. Nothing in your existing monitoring will notice.

Why standard APM is not enough

Your existing tooling tracks latency, error rate and throughput, and it should keep doing that. But all three can be perfectly healthy while the system quietly becomes useless.

Retrieval starts returning the wrong documents: latency unchanged, no errors. A prompt change makes answers vaguer: no error. The model provider ships an update and behaviour shifts: no error.

The signal you need is about content, not availability, and no generic dashboard is going to produce it for you.

What a trace has to contain

For any single request you should be able to see the full input as assembled, not the template but the actual final text with everything interpolated. The retrieved passages and their scores. The model and version. The parameters. The raw output before any post processing.

Then every tool call: which one, with what arguments, what came back, how long it took. And for multi step runs, the whole sequence with the decision at each point.

If any of that is missing, debugging becomes reconstruction from memory, and memory is not evidence.

Tracing a multi step request

One request can become a dozen model calls and a handful of tool invocations. Without a trace that ties them to a single identifier, you have a pile of unrelated log lines and no way to know which belong together.

Propagate a request id through everything, including the tool calls that leave your process. Record the parent and child relationships so the sequence is reconstructable in order.

This becomes non negotiable once you introduce indirection, which is one of the tradeoffs of adopting MCP: capability gets cleaner and debugging spans more systems.

Cost and token observability

Log token counts per request, broken down by what consumed them: instructions, retrieved context, history, tool definitions, output. Aggregate by feature and by customer.

Two things fall out immediately. You find out which feature is actually expensive, which is almost never the one people assume. And you catch runaway loops, because a task that normally takes four steps and suddenly takes forty shows up as a cost spike long before anyone reports a problem.

Without this breakdown, cost optimisation is guesswork. The budget framing is in what it costs.

Quality signals in production

Direct feedback is the cheapest and the most biased: thumbs, corrections, retries. A user who rephrases the same question twice is telling you something even though they never clicked anything.

Better are the implicit signals. Escalation rate to a human. Abandonment mid conversation. How often the system refused. How often the answer cited no source when it should have.

And sample real traffic for scored review against the same rubric your reference set uses, which connects production monitoring back to LLM evals. Passing your test set says you handle the cases you knew about. Only sampling tells you about the ones arriving now.

Alerting on the right things

Alert on rates and trends, not individual bad answers, because in a probabilistic system a single bad answer is expected and paging on it trains everyone to ignore the pager.

The alerts that earn their place: grounding rate dropping below a threshold, refusal rate moving sharply in either direction, escalation rate climbing, p95 token usage jumping, retrieval returning empty or low scoring results more often than usual.

Each of those describes a system changing character, which is the thing you actually want to be woken up for.

Debugging a bad answer, in order

Start at retrieval. Was the correct material in what was assembled? If not, the generation was never going to be right and the model is not the problem.

If the material was there, look at the assembled context. Was the right passage buried under nine irrelevant ones, or pushed past the point where it carried weight?

Only then look at the prompt and the model. This order matters because teams instinctively start at the end, tuning instructions to fix a problem that originated three steps earlier, and the fix appears to work until it does not.

What to build first

Full request tracing, before anything else. If you can pull up one request and see everything that happened, you can debug. Everything else is refinement on top of that.

Then token and cost breakdown, because it is cheap to add and immediately changes decisions. Then production sampling with scoring. Then alerting on the derived rates.

Teams routinely do this in reverse, buying a dashboard before they can reconstruct a single request. Observability is not a tool you purchase, it is a property you build in, and it is part of what makes production AI, end to end something you can actually operate.