Caching, Cost & Context Economics

Cost is an architecture problem

Teams usually discover LLM cost as a surprise, respond by switching to a cheaper model, and lose quality for a saving they never measured. The disciplined version runs in a fixed order, and the order matters more than any individual lever.

Step one: measure

Before touching anything, build a token profile. For each route, per request: input tokens, cached-read tokens, output tokens, reasoning tokens if applicable, and call volume.

Every provider returns usage on the response. Log it. If you are not logging usage per request with a route label attached, that is the first task, ahead of any optimisation.

What the profile almost always reveals:

• Spend concentrated in one or two routes nobody flagged.
• A large static prefix resent on every call, uncached.
• Output far longer than anyone asked for, because no limit was ever set.
• A retry loop quietly doubling the cost of a route that fails often.

Optimising before this is guesswork, and usually aimed at the wrong thing.

The cost model

$C = \frac{T_{\text{in}}^{\text{fresh}} \cdot p_{\text{in}} + T_{\text{in}}^{\text{cached}} \cdot p_{\text{in}} \cdot r_{\text{read}} + T_{\text{out}} \cdot p_{\text{out}}}{10^6}$

Two structural ratios hold broadly across providers and matter more than any specific price:

• Output is priced around 5× input. Generated tokens are the expensive half.
• Cached reads cost roughly a tenth of fresh input, against a write premium of about 1.25× to establish the entry.

That second ratio is the largest single lever available to most systems, so it goes first.

Prefix caching, properly understood

Recall from the architecture lesson: a provider caching your prompt is storing the KV cache for that prefix. This immediately explains the behaviour.

It is an exact prefix match. Not semantic, not fuzzy. One byte differs at position 400 and everything from position 400 onwards is recomputed. There is no partial credit.

Render order is fixed. Tools, then system, then messages. Your stable content must be first within that ordering, not merely first in your own mental model.

There is a minimum cacheable length. Short prefixes are not cached at all, and you are not told. Prefixes below roughly a thousand tokens frequently do nothing.

The silent invalidators

Every one of these has cost a real team real money, and none of them produces an error:

Verification is not optional. Read the cache-read token count on repeated requests. If it is zero, the cache is not working, no matter how carefully you arranged things. This is the single most common gap between an intended design and a deployed one.

Designing for it

tools           ← frozen, deterministic order
system prompt   ← frozen, versioned, no interpolation
few-shot block  ← frozen
────────────── cache breakpoint ──────────────
retrieved chunks
conversation history
user message    ← always last

Two structural notes. Per-user content belongs below the breakpoint, so all users share one cached prefix — a common design error is personalising the system prompt and destroying cross-user reuse. And where a provider supports mid-conversation operator instructions as a message rather than an edit to the system field, prefer it: editing the top-level system prompt mid-conversation invalidates everything after it.

The levers, in order

The order is deliberate. Everything above the line is free — no quality tradeoff. Do not touch anything below the line until the free wins are exhausted.

Free

1. Prefix caching. Covered above. Largest lever for most systems.

2. Input hygiene. Redundant restatements, a five-example block where three suffice, an entire document where a section was needed, conversation history resent in full rather than summarised. Every token is billed on every request.

3. Output discipline. Output costs ~5×. “Answer in three sentences” is a cost control. Set max_tokens to a real bound rather than a maximum — a runaway generation is billed in full.

4. Loop hygiene. In agent systems, every turn resends the full history. A loop that takes eight turns instead of five costs far more than 60% extra, because the resent context grows each turn. Reducing turn count is often worth more than reducing per-token price.

5. Batch processing. Where nothing is waiting on the result — nightly classification, bulk enrichment, backfills — batch endpoints typically cost around half. Requires tolerance for delayed completion.

Tradeoffs

6. Effort and reasoning depth. The first quality-trading lever. Tune per route, as covered in lesson 2.

7. Model selection. Only now. And measure the capable model at lower effort before dropping to a weaker model — it frequently wins on both quality and cost.

8. Cascades and routing. Genuine engineering, genuine complexity, and three costs people forget: lost cross-model cache reuse, two behavioural profiles to evaluate and maintain, and the failed first attempt still billed. Earn this one with data.

Semantic caching

Distinct from prefix caching: embed the incoming query, look for a previous query above a similarity threshold, return the stored answer.

Effective on workloads with heavy repetition — public FAQ endpoints, documentation search. Three things to get right before deploying it:

• The threshold is a correctness decision. Too low and you serve the answer to a different question. Test it against real query pairs, not intuition.
• Personalised or permission-scoped answers must never be cached across users. This is a data-leak vector, not a performance detail.
• Invalidation is yours. When the underlying documents change, stale answers persist until you clear them.

Latency, briefly

Cost and latency are not the same problem, and conflating them causes bad decisions.

• Streaming does not reduce cost at all, and transforms perceived latency. Use it in every user-facing path.
• Prefill scales with prompt length; decode does not. A long prompt hurts time-to-first-token specifically. Caching helps here too, since a cache hit skips prefill for that prefix.
• Long reasoning passes can exceed default HTTP timeouts. Streaming is the standard defence.

The workflow

1. Log usage per request with a route label
2. Build the token profile; rank routes by spend
3. Attack the top route with free levers, in order
4. Verify cache hits actually increased
5. Re-run evals — confirm quality held
6. Only now consider effort, then model, then cascades
7. Judge everything on cost per completed task

Step 5 is the one that gets skipped, and skipping it is how a cost project becomes a quality incident. Every lever below the free line changes output. Measure it.

Try this: Take your highest-volume route and log cache_read tokens for a hundred requests. If the figure is zero, walk the prefix top to bottom looking for anything that varies — a timestamp, an ID, a serialisation order. Fixing one of those is routinely a double-digit percentage of the bill, for an afternoon’s work and no quality cost at all.