What are LLM evals?

The short answer

An eval is a test for software that does not produce the same output twice. Instead of asserting that a function returns 4, you check whether an answer meets a standard: is it grounded in the source, does it follow the format, does it refuse when it should.

That is the whole idea. The difficulty is not conceptual, it is that you have to write down what good means before you can measure it, and most teams never do.

Why "it looks good" is not a test

Every team starts the same way. Someone tries fifteen prompts by hand, the answers look impressive, and the feature ships. This works right up until it does not, because a human spot check has three properties that make it useless as a safety net.

It is not repeatable, so you cannot tell whether today is better or worse than last Tuesday. It is biased toward the cases you thought of, which are never the cases that break. And it does not scale, so the moment you change a prompt you either re-check everything by hand or you check nothing.

The result is a system that degrades silently. Nobody changed anything, and yet the answers got worse, and the first signal is a customer complaint.

The three kinds that matter

Reference set evals. A fixed collection of real inputs with known good outputs. You run them on every change, like a test suite. This is the backbone and it is the one most teams skip.

Rubric evals. For open ended output where there is no single right answer, you score against criteria: is it grounded, is it complete, is the tone right, did it stay in scope. A model can apply the rubric, but a human has to write it.

Production evals. Sampling real traffic after the fact and scoring it. This is the only one that catches the inputs you never imagined, which in practice is most of them.

Building your first reference set

Fifty to two hundred real cases is enough to start, and starting is worth more than getting it perfect. Pull them from actual usage rather than inventing them, because invented cases carry your assumptions about what users do, and those assumptions are exactly what is wrong.

Weight it toward the cases that hurt. Include the ambiguous ones, the ones where the correct answer is to refuse, the ones where the data is missing, and the handful that already caused an incident. A set made only of easy cases will pass forever and tell you nothing.

Write the expected output for each. This is slow and it is the actual work. If nobody on the team can say what the right answer is, that is not an eval problem, that is a product definition problem wearing a disguise.

Who writes the rubric

Not the engineer, and this is the part that gets skipped. The rubric encodes what a good answer means in your business, which means it belongs to whoever owns the outcome: the credit analyst, the support lead, the person who would be doing this by hand.

Engineering owns running the evals. The domain expert owns what passing means. When those two collapse into one person, the rubric drifts toward what is easy to measure instead of what matters, and you end up optimizing a number that stopped representing anything.

Run them in CI, not in a spreadsheet

An eval you run manually is an eval you run twice and then abandon. The set has to execute on every prompt change, every model version, every retrieval tweak, and it has to fail loudly.

In practice that means a command in your pipeline that runs the set, reports pass rates per category, and blocks the merge when a rate drops below the line. The moment a model provider ships an update, you want the failure to arrive as a red build rather than as a support ticket three weeks later.

This is the same discipline as any other production system. It is why we treat evals as part of shipping rather than as a research activity, in every engagement where we build production AI, end to end.

Evals in production are monitoring, not testing

Passing your reference set means the system handles the cases you knew about. It says nothing about the ones arriving right now. Real inputs drift: your product changes, your customers ask new things, a competitor launches and the questions shift.

So you sample live traffic, score it against the same rubric, and watch the trend. Not every request, a percentage. What you are looking for is not a single bad answer, it is the slope: the quiet decline that means retrieval is degrading or a prompt change had a consequence nobody predicted.

What good looks like numerically

There is no universal number, and anyone quoting one is selling something. The right target comes from the cost of being wrong. A system suggesting related articles can live at eighty percent. A system participating in a credit decision cannot.

What matters more than the absolute figure is that you have a baseline and a direction. Know the number today, know what it was last month, and know which category is dragging. A team that can say "grounding is at 94 and dropped two points after the retrieval change" is in control. A team quoting a single overall accuracy figure usually is not.

Also track refusals and escalations separately. A system that answers everything confidently is not better than one that hands off when it should, it is just less honest about its limits.

Where teams get this wrong

Three failures, in order of frequency. Building the eval set after launch, when the incentive is to make it pass rather than to find problems. Measuring the model instead of the system, so retrieval quality and prompt structure hide behind a model score. And letting evals go stale, so the set describes a product that no longer exists.

The fourth, less common but more expensive: evaluating a generation problem when the real failure is retrieval. If the right passage never reaches the model, no amount of prompt tuning fixes it. We covered that split in what RAG is, and it is the single most common misdiagnosis in this work.

If you are earlier than this and still sizing the work, what an AI agent actually costs covers where evaluation sits in a real budget.