Reasoning Models & Effort Control

What actually changed

For a while, the way to get a model to reason was to ask: “think step by step.” That prompted trick worked because generated tokens condition subsequent generation — writing intermediate results puts them in the context where later steps can use them.

Reasoning models internalise this. Rather than relying on an instruction, they are trained to produce an extended internal reasoning pass before committing to an answer, and to allocate that pass according to difficulty. The mechanism is the same — tokens as working memory — but it is now trained behaviour rather than a prompt hack, and the amount of it is a parameter you control.

This changes three things in practice: you have a new cost/quality dial, latency is no longer proportional to output length, and the defaults differ between models in ways that will surprise you on upgrade.

The mental model: tokens as working memory

A transformer has no scratchpad. Every intermediate quantity it needs must either be recomputed in a single forward pass or written into the context where subsequent tokens can attend to it.

That is the whole reason reasoning tokens work. They are the model’s only mechanism for storing an intermediate result.

Which yields a genuinely useful prediction rule:

Reasoning helps in proportion to how much a task depends on intermediate results.

• Strong payoff — multi-step maths, debugging, planning under constraints, comparing many options against several criteria, long-horizon agentic work where step eight depends on step three.
• Weak or negative payoff — classification, extraction, translation, tone rewriting, formatting. Single-step transformations have nothing to store, so you pay tokens and latency for noise.

This is a structural property of the task. You can predict it before measuring, then confirm by measuring.

The effort dial

Current APIs expose reasoning depth as a setting — typically a scale from low through to maximum. Higher settings mean more reasoning tokens, more latency, more cost, and on the right tasks, better answers.

Three rules that hold across providers:

Tune per route, not globally. Your classification endpoint and your code-generation endpoint want different settings. A single global value either overspends on cheap work or underspends on hard work. Route-level configuration is the correct granularity.

Escalate only on measured headroom. The top of the range earns its cost on genuinely hard problems and wastes money everywhere else. Start in the middle, measure against your eval set, and raise the setting only where the numbers show room to improve.

Compare against the simpler alternative first. Before you build a multi-model cost cascade — cheap model first, escalate on failure — measure the capable model at lower effort on the same tasks. A strong model thinking less often matches a weaker model thinking hard, and it keeps everything in one cache namespace and one behavioural profile. Cascades are real engineering with real complexity; earn them.

Effort interacts with caching

Worth flagging because it catches people: changing the effort setting mid-conversation generally invalidates the message cache, and caches are model-scoped. A cascade that switches models between turns forfeits cache reuse across them. When you cost a cascade, count the lost cache hits, not just the cheaper per-token rate.

Reading the output honestly

Providers differ in what they return: some give a readable summary of the reasoning, some return nothing but still bill for it, and none expose the raw internal chain verbatim.

Two things follow.

Reasoning text is not a verified trace. It is generated by the same process as the answer. It is genuinely useful — for debugging, for spotting where a model went astray, for user-facing “showing its work” — but a model can produce impeccable-looking reasoning and a wrong conclusion. Never treat it as proof.

You are billed for reasoning whether or not you can see it. Hidden reasoning is not free reasoning. If your costs jumped after enabling it, that is why.

Handling it across turns

Two operational rules that cause real bugs when ignored:

• Echo reasoning blocks back unchanged when continuing on the same model. Stripping them can degrade multi-turn quality, because the model loses its own prior working.
• Do not carry them across models. Another model will not interpret them meaningfully, and typically ignores them.

Latency changes shape

Without reasoning, time-to-first-token is roughly a function of prompt length. With reasoning, the model may work for a long time before emitting anything a user can see.

For interactive products this matters more than the cost does. Three mitigations, in order of preference:

1. Stream a reasoning summary where the provider supports it, so the interface shows progress rather than a stalled spinner.
2. Lower the effort on interactive routes and reserve high settings for background work.
3. Move the work off the request path — queue it, and notify when done.

There is also a hard operational point: long reasoning passes can exceed default HTTP timeouts. Streaming is the standard defence, and on any high-effort route it should be the default rather than an optimisation.

The upgrade trap

This is the failure that catches teams, and it is worth stating explicitly.

Reasoning defaults are not consistent across model generations. Some models reason by default; some require it to be switched on; some reject the older configuration syntax outright. A model upgrade can therefore silently turn reasoning on — tripling latency and cost — or silently turn it off, quietly degrading quality on your hardest route.

Neither shows up as an error. Both show up as a metric moving for no apparent reason.

So: on any model change, re-baseline latency, cost per request and eval score before rolling out. Treat a model upgrade as a behavioural change, not a version bump.

Prompting reasoning models differs

Prompts tuned for older models are often over-specified for reasoning models, and this actively hurts.

• Stop prescribing the procedure. “First do X, then Y, then Z” constrains a model that would have found a better decomposition. State the goal and the constraints; let it plan.
• Drop “think step by step.” It is redundant at best. Instructions not to reason are worse — they interfere with trained behaviour and can produce leaked internal markers in the visible output.
• Give the whole task up front. Long-horizon work goes better with the full specification available than with drip-fed instructions, because the model plans against what it can see.

If you have prompts written eighteen months ago, they are probably worth auditing on exactly these three points.

A decision procedure

1. Classify the route: does it have dependent intermediate steps?
      no  -> lowest effort. Measure. Stop.
      yes -> continue.
2. Baseline at medium effort against the eval set.
3. Raise one step. Re-run. Compare score, latency, cost per completed task.
4. Keep the raise only if score improves materially.
5. Record the setting per route and re-baseline on any model change.

Step 4 deserves care about the denominator. Judge cost per completed task, not cost per request. A cheaper request that needs three retries or two extra agent turns is not cheaper.

Try this: Take one route in your system and run your eval set at low, medium and high effort, recording score, p95 latency and cost each time. Most teams find one of two things — a flat line, meaning they can drop to low and save immediately, or a sharp step, meaning they were quietly underspending on their hardest work. Either result pays for the afternoon.