Vector databases, explained

The short answer

A vector database stores text as lists of numbers that represent meaning, and finds the entries closest to your query in that numeric space. It is search by similarity rather than by exact words.

That is why it can match "can I get my money back" to a document titled "refund policy" with no words in common. It is also the whole reason it exists, and everything else is implementation.

Why normal search falls short

Keyword search matches strings. It is fast, exact, and completely literal, which is fine when the user knows your vocabulary and useless when they do not. Your customers do not say "remittance", they say "the money I sent".

Traditional search fixes some of this with synonyms and stemming, all of it maintained by hand and all of it brittle. Similarity search sidesteps the problem by comparing meaning directly.

Which does not make keyword search obsolete. Exact identifiers, order numbers, SKUs, names: those are still better served by an exact match, and the best retrieval systems use both.

What an embedding actually is

An embedding model reads a piece of text and outputs a fixed list of numbers, commonly several hundred to a couple of thousand of them. Texts with similar meanings land close together in that space.

You do not need to understand the geometry to use it. What you do need to know is that the same embedding model must be used for storing and for querying, that changing the model means re-embedding everything, and that embedding quality bounds retrieval quality no matter how good the database is.

How the search works

Your query is embedded the same way your documents were. The database then finds the nearest stored vectors, using an index that trades a little accuracy for a lot of speed, because comparing against every record exactly does not scale.

You get back the closest matches with a similarity score. The critical detail: it always returns something. There is no such thing as no results. If nothing relevant exists, you receive the least irrelevant thing, which is how systems end up confidently answering from the wrong document.

Do you need a dedicated one

Often not. If you already run Postgres, pgvector handles a substantial corpus perfectly well and saves you a system, a deployment, and a synchronisation problem between your records and your vectors.

A dedicated database earns its place at genuine scale, with heavy concurrent query load, or when you need features it specialises in. Adopting one early is a common way to add operational surface for a problem you do not have yet.

The question to ask is not which is better. It is whether the operational cost of a second datastore is smaller than the pain you currently have.

The part that decides quality

Not the database. Chunking, metadata and filtering decide whether retrieval is any good, and they are where the work actually is.

Chunk too small and you strip context, so the model sees an exception without the rule. Chunk too large and you dilute the signal. Attach metadata for everything you will need to filter on, especially permissions, because retrieval that ignores who is asking will eventually surface a document to someone who should never see it.

Filter before you search when you can. Restricting the candidate set by tenant, date or document type improves both speed and relevance more reliably than swapping the database.

What breaks at scale

Quality degrades quietly. A system that works with a thousand documents gets subtly worse with a hundred thousand, and it never throws an error. Nearest neighbours become less meaningful as the space gets crowded, and near duplicates start crowding out the one correct answer.

Then the operational reality: re-embedding a large corpus after a model change is expensive and slow, and keeping vectors in sync with a source of truth that changes is a pipeline, not a script.

None of this is visible without measurement, which is why retrieval needs its own evaluation separate from generation. We covered that split in what RAG is and how to measure it in LLM evals.

What it costs

Three lines. Embedding the corpus, which is a one time cost per document plus a small ongoing cost for changes. Storage and query, which scales with corpus size and traffic. And the re-embedding you will eventually pay when you change models.

For most mid-market corpora these numbers are smaller than teams expect. The expensive part is the engineering around it, not the storage.

When you do not need one

If your knowledge fits in a prompt and rarely changes, put it in the prompt. If your questions are answered by a structured query with a known shape, write the query, because a similarity search over records is a worse version of SQL.

And if the task is a decision rather than a lookup, retrieval is not the shape you need. Choosing the wrong shape is the expensive mistake, which is why we start from the decision when we build production AI, end to end.