Prompt injection security

The short answer

Prompt injection is when text a model reads contains instructions and the model follows them. It exists because models cannot reliably distinguish between content they were given to process and instructions they were given to obey.

That is not a bug in a particular model. It is a property of how these systems work, and it means the defence has to be architectural. We touched it as one layer in guardrails; this is the full picture.

Direct versus indirect

Direct injection is a user typing something adversarial into your interface. It is the version everyone tests for, and it is the less dangerous one, because the attacker only reaches what that user could already reach.

Indirect injection is the real problem. The attacker never touches your product. They put instructions in something your system will later read: a document, a support ticket, a web page, an email, a calendar invite, a filename, a code comment.

The instruction then arrives with the authority of your own pipeline, in a context where nobody is watching for it.

Why no prompt prevents it

The standard mitigation is a line in the system prompt saying to ignore instructions found in retrieved content. It helps. It is not a control.

You are asking a probabilistic system to reliably classify text as data rather than instruction, using the same channel for both. Every phrasing you add is another thing that can be outweighed by a sufficiently emphatic or cleverly positioned instruction downstream.

Treat prompt level defences as friction, not as a boundary. The boundary has to be somewhere the model cannot argue with.

Inventory your attack surface

Write down every source of text that reaches the model. In most systems the list is longer than the team expects: retrieved documents, user profile fields, previous conversation turns, tool call results, error messages from other systems, file names, metadata.

Anything on that list authored by someone other than you is untrusted. That includes content from inside your company, because an internal document is still text someone can edit. This is also the reason MCP servers and tool descriptions belong on the list.

Tool calling is what makes it consequential

An injected instruction in a system that only produces text is a content problem: an embarrassing answer, a leaked snippet.

The same instruction in a system that can call functions is an action problem. Now it can attempt to read records, send messages, modify data or exfiltrate what it retrieved by embedding it in a request.

So the severity of injection is determined by what the model is permitted to do, not by how good your prompt is. Reducing capability reduces the impact of every future attack you have not thought of.

Mitigations that actually work

Scope permissions to the requesting user, always, so a successful injection accomplishes only what that user could do anyway. Keep tool sets minimal per task. Make destructive actions require explicit confirmation outside the model's control.

Separate data from instructions structurally where the API allows it, and clearly delimit retrieved content so the model has a signal about provenance. Validate outputs before acting on them rather than trusting them because they parsed.

And treat exfiltration paths as part of the surface: if the model can put text into a URL, an image reference, or an outbound request, it can leak what it read.

Human in the loop, placed properly

For irreversible or high value actions, a person confirms. The confirmation has to show what will actually happen, in plain terms, not a summary the model wrote about itself.

A confirmation dialogue that says the model would like to proceed is theatre. One that says this will send this message to this address is a control.

How to test for it

Build an adversarial suite and run it in CI. Include documents with embedded instructions, attempts to access another user's data, requests to exceed tool scope, and content designed to trigger exfiltration.

Run it on every prompt change and every model update, because the behaviour underneath changes without your involvement. The discipline is the same as LLM evals, and the suite should grow every time someone finds something new.

After an incident

Assume the first one will happen. What matters is whether you can answer three questions quickly: what did the model read, what did it do, and what did it have access to. If your logging cannot answer those, the incident becomes an investigation with no evidence.

Then convert it into a test case so it cannot recur silently. That loop, incident to test to regression suite, is what turns security from a document into a practice, and it is part of how we build production AI, end to end.