What is context engineering?

The short answer

Context engineering is deciding what information the model sees on any given request, how it is arranged, and what gets left out. Prompt engineering is about how you ask. Context engineering is about what you hand over.

The term appeared because teams noticed the bottleneck had moved. Once a prompt is competent, the difference between a system that works and one that does not is almost entirely about whether the right material was in front of the model at the moment it answered.

Why the term appeared

Early on, the model was the constraint and clever phrasing produced real gains. As models improved, that stopped being where the leverage was. A better instruction cannot compensate for a missing document.

At the same time, systems got more complicated. A production request now assembles system instructions, retrieved passages, conversation history, user attributes, tool definitions and prior tool results. Somebody has to decide what makes the cut.

That decision is engineering, not writing. It has budgets, tradeoffs, failure modes and it belongs in code review.

The context window is a budget, not a container

Large context windows encouraged a bad habit: put everything in, let the model sort it out. That fails in three ways at once.

It costs money on every request, and long prompts at volume dominate the bill. It costs latency, because more input means slower responses. And it costs accuracy, because relevant material competes with noise, and the model's attention is finite even when the window is not.

Treat it as a budget you spend deliberately. Every token you add should earn its place by making a specific class of answer better.

The four sources

Instructions. Who the system is, what it does, what it never does. Mostly static, and the part teams over invest in.

Retrieved knowledge. The passages pulled for this specific question. Usually the highest value per token and the hardest to get right.

State. Conversation history, what the user already told you, what has already been tried. Grows without bound if nobody manages it.

Capability. Tool definitions and the results of calls already made. Easy to underestimate, because ten tool descriptions can be larger than the actual question.

Retrieval is context engineering

Most of what people call a RAG problem is a context problem wearing a different name. The model did not fail to reason. It reasoned correctly over the wrong material, which is what happens when retrieval returns something plausible instead of something relevant.

This is why retrieval and generation have to be evaluated separately, a split we covered in RAG and vector databases. If the right passage never arrives, no amount of prompt work fixes the answer.

The practical implication is that improving context usually means improving what gets selected, not writing more instructions about how to use it.

Order and position matter

Where something sits in the context changes how much it influences the answer. Material at the very beginning and very end tends to carry more weight than material buried in the middle, and this effect gets stronger as the input grows.

So put the instruction that must not be ignored at the boundaries, not in the centre of a wall of retrieved text. Put the most relevant passage first rather than trusting the model to rank ten of them.

This is unglamorous and it is measurable. Reordering the same content, with nothing else changed, moves accuracy on a reference set often enough that it is worth testing explicitly.

Compaction, summarisation and memory

Any conversation of real length eventually exceeds what you want to send. You have three options and each one loses something.

Truncate and you lose the beginning, which is often where the actual requirement was stated. Summarise and you lose detail, and the summary itself can hallucinate. Retrieve selectively from history and you lose continuity when the selection misses.

There is no clean answer. What works in practice is protecting a small set of facts explicitly, the ones that must survive every compaction, and treating everything else as recoverable. Decide what those facts are on purpose rather than discovering the answer during an incident.

How to know whether context is working

Measure retrieval separately from generation. For retrieval, check whether the correct material appeared in what was assembled. For generation, check whether the answer is supported by what was actually provided. The methods are in LLM evals.

Then instrument the budget itself. Log how many tokens each source consumed on each request. Teams are routinely surprised to find that tool definitions or conversation history are eating most of the window while the retrieved passages, the part that carries the answer, get a fraction.

Without those numbers, context tuning is guesswork, and it also hides where the money goes. We covered the cost side in what it costs.

Where teams waste it

Four patterns, in order of frequency. Enormous system instructions that restate the same rule five ways. Full conversation history when the last two turns would do. Ten tool definitions when the request could only ever need two. And retrieved chunks that are too large, so a page about twelve topics is included for one paragraph.

The fix in every case is the same: be deliberate about what earns a place. That discipline is a large part of why some AI features hold up under real traffic and others only work in the demo, and it is central to how we build production AI, end to end.