LLM cost optimization

The short answer

For most production systems the bill is dominated by input tokens, not output. You are paying to send context on every single request, and that is where the leverage is.

Which means the first optimisation is almost never a cheaper model. It is sending less, less often.

Measure before you optimise

Log tokens per request broken down by source: instructions, retrieved context, history, tool definitions, output. Aggregate by feature.

Teams are consistently wrong about which feature is expensive. Without the breakdown you will optimise the visible thing rather than the costly one. The instrumentation is described in observability.

The biggest lever is context size

Long prompts are charged on every call. Trimming retrieved passages from ten to three, or cutting a system instruction that restates the same rule five ways, changes the bill more than switching model tiers usually does.

This is why cost work and quality work converge: the same discipline that improves answers, sending only what earns its place, also reduces spend. That is the subject of context engineering.

Caching

Two kinds and they are different. Prompt caching reuses the processed form of a repeated prefix, so a large stable system instruction is not reprocessed on every call. Cheap to adopt, often a large win, and it rewards putting the stable part first.

Semantic caching returns a previous answer for a sufficiently similar question. Powerful for repetitive workloads and dangerous where answers depend on user, permissions or freshness. Use it where the answer is genuinely the same for everyone, and nowhere else.

Model routing

Not every step needs your best model. Classification, extraction, routing and formatting frequently run on a smaller model at a fraction of the cost with no measurable quality loss.

Do this with measurement, not intuition. Route the step, run your reference set, compare. Without LLM evals you are trading quality for cost blindly and will not notice the trade until users do.

Loop control

In agentic systems, cost scales with iterations and the input grows as history accumulates. A task that normally takes four steps and occasionally takes forty is where budgets break.

Cap iterations hard. Budget tokens per task. Log both and alert on the tail rather than the average, because the average looks fine while the tail eats the margin.

Batch and asynchronous work

If a response does not have to be immediate, it should not be priced as if it does. Many providers offer substantially cheaper asynchronous processing, and plenty of production work, enrichment, classification, summarisation of yesterday, tolerates it.

The design question is which parts of your product genuinely need to be synchronous. Usually fewer than assumed.

What not to optimise

Do not optimise before you measure, do not optimise a feature nobody uses, and do not trade accuracy for cost on a decision where being wrong is expensive.

Also resist rebuilding infrastructure to save a small monthly amount. Engineering time is more expensive than most inference bills at mid-market volume, and a rewrite that saves a little and costs a quarter is a bad trade dressed as discipline.

Set a budget per decision

The useful framing is cost per decision rather than total spend. If a decision saves more than it costs to make, volume is good news. If nobody has computed that ratio, cost conversations stay abstract.

Once you have it, you can decide where a more expensive model is worth it and where it is not. That framing sits alongside the full breakdown in what an AI agent costs and is part of how we build production AI, end to end.