Cost Control for LLM Apps: Caching, Batching, Model Tiers

You shipped an AI feature and it works. Users are happy, product is happy, and then finance forwards you the Azure invoice. The number is not what you planned for. This happens to most teams on their first serious LLM deployment, not because they wasted tokens, but because none of the cost controls are on by default.

Here are the levers that actually move the bill, ordered by impact. Most teams can cut Azure OpenAI spend 40-60% without touching product logic.

Model tier is the biggest lever

One frontier model for every request is where the money goes, and if that model is still gpt-4o you are paying the wrong default twice over. Prices as of writing, Global Standard, per 1M tokens:

Two things jump out. First, gpt-4o now costs twice gpt-5 on input for the same output price, so defaulting to it is pure waste; switch the frontier tier to gpt-5. Second, gpt-5-mini costs nearly 10x less on input than gpt-5. Classification, entity extraction, intent detection, template filling: route all of it to a mini tier. In most apps I have audited, 30-50% of calls are simple enough for it. Run that audit before anything else.

Prompt caching is nearly free money

If your system prompt is over 1,024 tokens and identical across requests (true in almost every production app), you pay full price for it every call unless you structure prompts for Azure OpenAI’s prefix cache. The rule: everything stable at the top, dynamic content at the bottom.

Cached reads cost roughly 50% less on Standard deployments and up to 100% less on Provisioned. It is on by default on GPT-4o and every newer model, gpt-5 included, so the frontier tier you just picked already supports it. You just keep the first 1,024 tokens identical across requests.

Two things the docs bury. First, the cache lives only 5-10 minutes by default, which quietly kills hit rates on spiky traffic; the newest models keep it up to 24 hours, and you opt older ones in with prompt_cache_retention: "24h". Second, from gpt-5.6 onward cache writes are billed on top of the discounted reads, so the discount only lands if your prefix stays stable enough to hit reads rather than rewrite the cache every call. The stable-top rule buys you both.

The other three caches

Prompt caching is the built-in one and it only discounts input. Three more caches sit in your own code, and each saves something different:

• Exact-match first. Hash the full request, store the response with a TTL. No embeddings, catches identical repeats, cheapest cache to run. Add it before semantic caching, not after.
• Semantic for reworded repeats: embed the query (text-embedding-3-small is ~$0.02 per 1M tokens), vector-search Azure Cache for Redis at ~0.95 cosine similarity, return the cached answer on a hit. Bake the model name and a data-version hash into the key so stale content invalidates. Realistic hit rates run 15-40% for general assistants, 60-70% for FAQ-style workloads.
• Embedding cache is the one people forget, and it saves on the document side, not the query. Every RAG re-index embeds your corpus, and embedding a large corpus is a real bill (text-embedding-3-large is ~$0.13 per 1M). Store vectors keyed by a content hash so you never pay to embed the same unchanged text twice; re-embed only the chunks that changed. The same cache also covers the query embeddings your semantic cache generates.

Output discipline

Output tokens cost several times more than input, the price ratio nobody memorizes, and it widened when input got cheaper: output is 4x input on gpt-4o-mini and 8x on gpt-5. One sentence in your system prompt (“Respond concisely, under 200 words unless the user asks for more detail”) can halve output spend. For high-volume apps that single change cuts total cost 20-30%.

The reasoning-token trap

The gpt-5 and o4-mini tiers above are reasoning models, and they bill for tokens you never see. A reasoning model spends hidden “thinking” tokens before it answers, and those are charged as output at the full output rate.

The one real lever is reasoning_effort: minimal or low for routine work, high only for genuinely hard problems. On gpt-5 it defaults to a middle setting, so classification and extraction routed to a mini tier are often paying for reasoning they never needed. Set it explicitly, and pair it with the verbosity parameter to control length without a prompt sentence.

The rest, in order of when to bother

• Spending alerts first. Before optimizing, fence the spend so a retry loop or a ballooning prompt does not surprise you. The fastest reliable path is the portal: Cost Management → Budgets → Add, set a monthly amount and an 80% alert email. Five minutes. For the CLI, use az consumption budget create-with-rg with a --notifications block and a --time-period (start and end date). Note the gotcha: the plain az consumption budget create has no --notifications parameter and requires --start-date/--end-date, so a budget-with-alert has to go through create-with-rg.

az consumption budget create-with-rg \
  --resource-group your-rg --budget-name azure-openai-monthly \
  --category Cost --amount 500 --time-grain Monthly \
  --time-period start-date=2026-07-01 end-date=2028-07-01 \
  --notifications '{"actual-80":{"enabled":true,"operator":"GreaterThan","threshold":80,"contactEmails":["your-team@company.com"]}}'

• Context trimming once conversations get long: window to the last N turns, or summarize older turns with a cheap mini call. A 10-turn chat with a 500-token system prompt can hit 5,000+ input tokens before the user types.
• Batch non-realtime work. The Azure OpenAI Batch API runs at 50% less than synchronous Global Standard with a 24-hour target turnaround (a target, not a hard SLA; longer jobs are not killed). Document processing, report generation, nightly analysis: same model, half the cost, latency the only trade-off.

Where to start

Model tier and spending alerts first: those two alone usually cut bills 40-60% with zero quality hit. Then add prompt-cache structure (free, just move the stable prompt above dynamic content) and the output instruction, another two hours of work for most of the remaining gains. Add an exact-match cache next (cheap, no embeddings), then semantic and embedding caches once traffic is steady; context trimming and batching once conversations grow and jobs can wait.