Under the Hood: Tokens, Context & Cost

Tokens are the unit of everything

Cost, speed, context limits and rate limits are all denominated in tokens. Getting an intuition for them makes the rest of applied AI legible.

A tokeniser splits text into sub-word pieces chosen so that common sequences get one token and rare ones get several.

Working figures: ~0.75 words per token in English; ~4 characters per token. Code runs denser than prose because of punctuation and indentation.

The language asymmetry is not a footnote — the same meaning can cost two or three times more in some languages than in English. If you are building for a non-English audience, measure it rather than assuming.

Do not estimate when it matters. Every provider ships a token counter, and there is a counting endpoint on the major APIs. Use it before you ship anything with a budget attached.

The cost equation

Almost all pricing follows one shape:

$\text{cost} = \frac{T_{\text{in}}}{10^6} \cdot p_{\text{in}} + \frac{T_{\text{out}}}{10^6} \cdot p_{\text{out}}$

Two structural facts hold across current model families, and they matter more than any specific price:

• Output costs roughly 5× input. Across current tiers the ratio sits near five to one. Generated text is the expensive half.
• Tiers are separated by roughly an order of magnitude. A small fast model against a frontier model is typically a 5–10× difference per token.

Specific prices move constantly, so build the habit of reading them from the provider’s pricing page rather than memorising a table. As a reference point at the time of writing, a frontier model sits around $5 per million input tokens and $25 per million output; a small model is a few times cheaper again.

Where the money actually goes

Two consequences follow directly, and they surprise people:

Long outputs cost more than long inputs. A 500-token reply can outweigh a 2,000-token prompt. “Answer in three sentences” is a cost control, not just a style note.

Repeated prefixes are the biggest avoidable waste. A chat app that resends a 4,000-token system prompt on every turn pays for it every turn. Which brings us to caching.

Prompt caching

If a large prefix of your request is byte-identical to the last one, providers can serve it from cache. The economics are dramatic: cached reads cost roughly a tenth of normal input tokens, against a one-off write premium of about 1.25× to put it there.

The catch is that caching is a prefix match. Any byte that changes invalidates everything after it. So the architectural rule is:

Stable content first. Volatile content last.

Put the frozen system prompt and fixed tool definitions at the front; put timestamps, request IDs and the user’s actual question at the end. A datetime.now() at the top of a system prompt silently destroys the cache on every single request, and nothing in the response will tell you — you have to look at the cache-hit figures in the usage data.

The context window, and its two limits

Frontier models now reach around a million tokens; cheaper tiers commonly sit at 200K. That is a hard ceiling on everything visible at once — instructions, history, uploads, and the reply.

But the hard limit is rarely the real one. Two softer limits bite first:

Attention is uneven. Recall on a specific fact is strongest near the beginning and end of a long context and measurably weaker in the middle. Fill 800K tokens and you are not getting 800K tokens of reliable recall.

Latency and cost scale with what you send. Every token is processed on every request. A million-token prompt is slow and expensive whether or not the model needed all of it.

The practical consequence: a big window is not a substitute for retrieval. Selecting the right 5,000 tokens beats dumping 500,000 — it is faster, cheaper and more accurate. That is the entire argument for RAG, which is the next lesson but one.

Managing long conversations

Three approaches, in increasing sophistication:

• Truncate — drop the oldest turns. Trivial, and it loses things you needed.
• Summarise — periodically compress old history into a short digest. Cheap and effective; most production chat does this.
• Retrieve — store history externally and pull back only the relevant parts. Best quality, most machinery.

Practical checklist

• Measure tokens with a real counter before shipping anything with a budget.
• Constrain output length explicitly — it is the expensive half.
• Order prompts stable-first so caching can work, then verify cache hits in the usage data.
• Put critical instructions at the very start or very end of a long prompt.
• Route simple tasks to a small model; reserve the frontier tier for work that needs it.

Try this: Take a prompt you use often, count its tokens, and work out what it would cost at 1,000 requests a day. Then split it into a frozen prefix and a variable suffix and recompute with cached reads at a tenth of the price. The gap is usually large enough to change how you write prompts.