Hallucination in production

The short answer

A model does not know it is wrong. It produces the most plausible continuation given what it was shown, and when it was not shown enough, plausible and correct come apart.

So hallucination is not a defect you patch. It is the default behaviour of the technology, and the job is to constrain the conditions under which it happens and to detect it when it does.

What it actually is

The model is not retrieving a fact and getting it wrong. It is generating text that fits the pattern of a correct answer. Fluency and accuracy are produced by the same mechanism, which is why a hallucinated answer reads exactly as confidently as a true one.

This is the property that makes it dangerous in business use. A system that failed obviously would be safe. A system that fails beautifully requires someone to check.

The three causes

No grounding. The model was asked something it has no source for, so it filled the gap. The most common cause and the most fixable.

Bad retrieval. Material was provided, but the wrong material, so the model reasoned correctly over an irrelevant document. This gets diagnosed as a model problem constantly, and it is not.

Pressure to answer. The system was built with no acceptable way to say it does not know, so it produces something. Refusal has to be designed in, or the model will always prefer a confident guess. The retrieval half of this is covered in RAG.

Why it gets worse in production

In testing, questions come from people who know what the system is for. In production they come from everyone, including people asking about things it was never meant to cover.

Meanwhile the corpus drifts. Documents are added, policies change, and old versions stay in the index. The model is now grounded in material that contradicts itself, and it will confidently pick one.

None of this shows up as an error. Latency is fine, no exception is thrown, and the only signal is that answers are quietly less right than they were.

Grounding is the primary fix

Give the model the material and require the answer to come from it. That single change removes most hallucination in most business systems, because most of it was the model filling a gap that did not need to exist.

The discipline is to make grounding enforceable rather than requested. Check that the claims in the output are supported by the passages provided, and treat an unsupported answer as a failure even when it happens to be true, because next time it will not be.

Citations and verification

Requiring the system to cite which passage supports each claim does two things. It gives a human a fast way to verify, and it makes the failure mode visible: an answer with no citation is a flag rather than a silent risk.

Be aware that citations can be fabricated too. Verify that a cited passage exists and actually contains what is claimed, rather than trusting the reference because it looks like one.

Refusal is a feature

A system that says it does not have the information is more valuable than one that always answers, and most teams build the second one by accident because refusals feel like failures in a demo.

Design the refusal path deliberately: what triggers it, what the user sees, where it escalates. Then measure the refusal rate, because both directions are informative. Too low means it is guessing. Too high means retrieval is failing. This is part of the same layering as guardrails.

Measure the rate

Stop arguing about whether the system hallucinates and produce a number. Take a reference set with known answers, run it, and score how often output is unsupported by the source. Then sample production traffic and score the same way. The method is in LLM evals and the plumbing in observability.

Once there is a number and a trend, the conversation changes from opinion to engineering, and you can tell whether last week's prompt change helped or hurt.

When it reaches a customer

Have the path decided before it happens: who is told, how the answer is retrieved and corrected, and how the case enters the evaluation set so it cannot recur silently.

The organisational failure is treating each incident as a one off apology rather than as data. Every hallucination that reaches a customer is a test case you did not have, and adding it is the cheapest improvement available. That loop is central to how we build production AI, end to end.