What is RAG?

The short answer

RAG stands for retrieval-augmented generation. It means: before the model answers, you go find the relevant information and hand it to the model along with the question. The model then answers using that material instead of relying only on what it absorbed during training.

That is the entire idea. Everything else is engineering detail about how you find the right material fast enough and reliably enough.

The problem RAG solves

A language model knows what was in its training data, up to a cutoff date, in general terms. It does not know your pricing table, your policy document from last Tuesday, or the state of a specific customer account. Ask it anyway and it will produce something plausible, which is worse than producing nothing.

There are only three ways to give a model knowledge it does not have. You can put it in the prompt, which does not scale past a certain size. You can bake it into the weights through fine-tuning, which is slow and goes stale. Or you can retrieve it at question time, which is RAG.

For anything that changes, retrieval wins. Your policy document changes; the model weights should not have to.

How the pipeline actually works

Five steps, and every one of them is a place things go wrong. First, ingestion: you take your documents and split them into pieces. Second, embedding: each piece is converted into a vector, a list of numbers that represents its meaning. Third, storage: those vectors go into a database built to search by similarity.

Fourth, retrieval: when a question arrives, it is embedded the same way, and the database returns the pieces whose vectors sit closest to it. Fifth, generation: those pieces are inserted into the prompt and the model writes an answer grounded in them.

Notice that the model is only involved in the last step. Four fifths of a RAG system is a search problem wearing an AI costume.

Chunking and embeddings, without the jargon

Chunking is deciding how to cut your documents. Cut too small and you get fragments with no context, so the model sees a sentence about an exception without the rule it is an exception to. Cut too large and you dilute the signal, so a page about twelve topics matches weakly against a question about one of them.

Embeddings are how similarity gets measured. Two texts that mean similar things end up with vectors that are close together, even when they share no words. That is why a good system can match the question "can I get my money back" to a document titled "refund policy".

Most teams tune the model first and the chunking last. It should be the other way around.

Where RAG breaks in production

Almost always in retrieval, almost never in generation. If the right passage reaches the model, modern models summarize it accurately. If the wrong passage reaches the model, it will summarize the wrong passage with total confidence.

The failures we see repeatedly: the corpus contains three versions of the same policy and nobody marked which is current. Permissions were not carried into retrieval, so the system surfaces a document the asking user should never see. Updates are ingested nightly but the business assumes real time. And the long tail, where the questions that matter most are the rare ones, and the rare ones retrieve worst.

One more that is easy to miss: retrieval that works fine at a hundred documents degrades quietly at a hundred thousand. It does not throw an error. It just gets less right, and nobody notices without evaluation.

RAG or fine-tuning: different problems

This is presented as a versus and it is not. Fine-tuning changes how the model behaves: its format, its tone, its adherence to a structure, its handling of a specialized task. RAG changes what the model knows at the moment of answering.

If your complaint is "it does not know our products", that is RAG. If your complaint is "it knows, but it answers in the wrong shape", that is fine-tuning or better prompting. Teams routinely spend a quarter fine-tuning a knowledge problem and get nowhere.

Plenty of mature systems use both. Retrieval for the facts, tuning for the behaviour.

How to evaluate a RAG system

Evaluate the two halves separately, or you will not know which one is failing. For retrieval, take real questions and check whether the correct passage appears in what came back. For generation, check whether the answer is actually supported by the passages provided, not just whether it sounds right.

That second check is the one people skip. An answer can be factually correct and still be unsupported by your documents, which means the model filled the gap from training data. In a regulated context, that is a real problem, because you cannot show where the answer came from.

Build a reference set of fifty to two hundred real questions with known good answers, and run it on every change. Without that, every improvement is a guess.

What it costs to run

Three lines. Embedding your corpus, which is a one time cost per document plus a small ongoing cost for changes. Storing and searching the vectors, which scales with corpus size and query volume. And the generation itself, which is usually the largest line because RAG prompts are long by design.

The counterintuitive part: RAG makes each request more expensive than a bare model call, because you are paying to send retrieved context every time. It is still almost always cheaper than the alternative, which is a wrong answer that a human has to catch and fix.

When you do not need RAG

If the knowledge fits comfortably in the prompt and rarely changes, just put it in the prompt. If your questions are answered by a database query with a known shape, write the query; a model does not need to guess at something SQL answers exactly. And if the task is a decision rather than a lookup, what you need is closer to an agent than to retrieval.

Choosing the wrong shape is the expensive mistake, not choosing the wrong vendor. We wrote about the distinction in what an AI agent actually is, and it is the same principle: the architecture should follow the problem, which is why we build production AI, end to end rather than leading with one technique.