Model drift monitoring

The short answer

Systems degrade without anyone touching them. Your code is identical, your prompts are identical, and the answers are worse than they were in March.

This is normal and expected. The failure is not that it drifts, it is that most teams have no way to detect it and find out from a complaint.

Three kinds of drift

Provider drift. The model behind the API is updated, deprecated or subtly retuned. Behaviour changes and you were not consulted.

Data drift. Your corpus grows, policies change, old versions linger in the index. The system is now grounded in material that contradicts itself.

Requirement drift. Your product changed, your users ask different things, a competitor launched. The system still answers the old questions well and the new ones badly.

Why nothing errors

None of these produce an exception, a failed health check or a latency spike. Uptime stays perfect while quality declines, which is why standard monitoring provides false comfort.

The only signal is content, and content signals have to be built deliberately. That is the argument in observability.

Version pinning and its tradeoff

Pin the model version where the provider allows it. It converts a surprise into a scheduled decision, which is worth a great deal.

But pinning is not permanent safety. Versions get deprecated, and a system pinned for two years accumulates an upgrade that becomes a project rather than a change. Pin deliberately, and plan the upgrade rather than deferring it until forced.

The regression suite is the safety net

A reference set of real cases with known good answers, run automatically, is what turns drift from a discovery into an alert. Without it there is no baseline, and without a baseline there is no drift, only opinions.

Run it on a schedule, not only on deploys, because provider drift happens without a deploy. That is the difference between the suite described in LLM evals and one that actually catches this.

What to monitor continuously

Grounding rate, so you catch answers drifting away from the source. Refusal rate, in both directions. Escalation rate, which is often the earliest human signal. Output length distribution, which shifts noticeably when a model changes. And retrieval scores, which fall as the corpus grows.

Watch the trend, not the point. A single bad day is noise. Three weeks of decline is a system changing character.

Staged rollout for model changes

Treat a model upgrade like a deploy, because it is one. Run the new version against the regression suite first. Then route a percentage of live traffic to it and compare on real inputs.

Keep the ability to roll back to the previous version quickly. Teams that upgrade wholesale on a provider deadline discover the regressions in production with no way back, which is an avoidable position.

Alerting without noise

Alert on rates crossing thresholds and on sustained trend changes, never on individual answers. In a probabilistic system a bad answer is expected, and paging on it teaches everyone to ignore the page.

Route the quality alerts to whoever owns the outcome, not only to engineering. A drop in grounding rate is a business signal before it is a technical one.

A practical cadence

Regression suite on every change and weekly regardless. Sampled production review weekly. A deliberate look at provider release notes and deprecation schedules monthly. And a full reference set refresh quarterly, because a set that describes last year's product stops protecting this year's.

None of this is exotic. It is the ordinary discipline of operating software, applied to a component that changes underneath you, and it is part of what makes production AI, end to end something you can run rather than something you launched.