AI in Production

Reliability, observability, failure isolation and rollout discipline — the engineering that turns a working prototype into a system you can operate.

What actually changes in production

An LLM is an unusual dependency: slow by service standards, expensive per call, non-deterministic, occasionally unavailable, and — the part that breaks conventional practice — capable of failing while returning a perfectly valid 200 response.

That last property drives most of what follows. Your existing reliability toolkit assumes failures announce themselves. Here the worst ones do not.

Failure taxonomy

Handling differs by category, so classify first.

Class	Examples	Handling
Transport	Timeout, 5xx, connection reset	Retry with backoff
Rate limit	429	Backoff with jitter; client-side throttling
Request	400, context overflow, bad schema	Fix — never retry
Refusal	Model declines the request	Distinct path; do not treat as an error
Semantic	Wrong, fabricated, off-format	Validation — invisible to error handling

The bottom row is where production AI systems actually fail. No amount of retry logic addresses it, because nothing failed in a way any monitor detects.

Semantic failure containment

Three layers, applied to output before it is used:

Structural validation. Schema conformance. Use constrained decoding so this cannot fail, rather than checking afterwards.
Business validation. Is the referenced order ID real? Is the total within a sane range? Is the cited document one you actually retrieved? Cheap, deterministic, and it catches a large share of real incidents.
Confidence routing. Where the model reports uncertainty or the retrieval score is low, route to a human rather than acting.

The framing worth internalising: you cannot make the model always correct, so make wrong output detectable and contained.

Retries and idempotency

Standard exponential backoff with jitter:

$\text{delay} = \min\left(d_0 \cdot 2^{a} + U(0, j),\; d_{\max}\right)$

Jitter is not optional at scale — synchronised retries after a provider blip produce a thundering herd that extends the outage.

Retry: timeouts, 5xx, connection failures, 429. Never retry: 400s, context overflow, validation errors. They will fail identically and you pay each time.

The idempotency problem is specific to agents and tool use. If a call failed after a tool sent an email or took a payment, retrying repeats the side effect. Requirements:

An idempotency key per logical operation, checked before acting.
A record of side effects already committed for this request.
Retry logic that resumes from the last committed step rather than restarting.

This is the bug that reaches customers, and it is architectural rather than incidental.

Timeouts and streaming

Timeouts need to reflect real behaviour, not a guessed default:

Time-to-first-token scales with prompt length — that is prefill.
Total duration scales with output length and reasoning depth.
Long reasoning passes can exceed default client timeouts entirely. On any high-effort route, streaming is the defence, not an optimisation.

Set per-route timeouts from measured p99, not one global value. And in user-facing paths, always stream: it does not reduce cost at all and it transforms perceived latency, because the user sees progress at a few hundred milliseconds rather than staring at nothing for eight seconds.

Observability

You cannot debug what you did not record, and aggregate metrics are insufficient here.

Per request

Request ID, correlating through your whole stack
Prompt version and model ID — the first question in any incident
Full input and output
Token usage: input, cached read, output, reasoning
Latency: time-to-first-token and total
Stop reason
Retrieved document IDs, for RAG
Tool calls and results, for agents
Cost

Prompt version and model ID are the two that get omitted and the two you always need. Without them, “it got worse last Tuesday” is unanswerable.

Aggregate

Latency percentiles per route (p50, p95, p99 — the mean hides everything), error rate by class, cache hit rate as a first-class metric since it is a large share of the bill, cost per route and per user, refusal rate, and validation failure rate.

Two alerts that catch real problems and are usually missing: cache hit rate falling — someone put a timestamp in the prefix — and output length drifting, which is both a cost and a behaviour signal.

Sampling for quality

Metrics tell you the system is up. They do not tell you it is good. Sample a small percentage of production traffic daily, grade it — with a calibrated model judge, spot-checked by humans — and track the trend. This is the only way to catch slow quality drift, which no infrastructure alert will ever show you.

Provider strategy

Depending on one provider is a real availability risk. Depending on many is real complexity. Choose deliberately rather than by default.

Abstract at the right layer. Wrap calls in your own interface so the provider is swappable, but do not pretend providers are identical — prompts, tool formats, refusal behaviour and reasoning configuration all differ.

Fallbacks must be tested. An untested fallback is a liability that fails on its first real use. Exercise it deliberately, on a schedule.

Evaluate each path separately. A prompt tuned for one model is not tuned for another. If a fallback serves meaningfully different quality, know the number before an outage tells you.

Some providers offer server-side fallback, retrying on another model within one call. Simpler than client-side routing where available.

Be honest about whether you need this. For many systems, a clear degraded mode — a cached response, a queued request, an honest error — is better engineering than a half-tested second provider.

Rollout

Prompts and models are deployed code with behavioural consequences. Treat them accordingly.

1. Eval set passes locally
2. Deploy behind a flag, 0% traffic
3. Shadow: run in parallel, log both, compare offline
4. 5% of traffic, watch quality and cost metrics
5. Ramp to 50%, then 100%
6. Keep the previous version deployable for instant revert

Shadow mode is underused and cheap. Run the new version on real traffic without serving its output, and compare. It surfaces distribution mismatch that no fixed eval set can represent, at the price of the extra inference.

A model upgrade is not a version bump. Reasoning defaults, tokenisation and refusal behaviour all differ between generations. Re-baseline quality, latency and cost before ramping — this is the trap lesson 2 describes, and it reaches production through exactly this door.

Security

Untrusted content and dangerous capabilities must not share a context. The architectural control from lesson 3, and it belongs in your deployment review.
Authorisation in code, per user, on every tool call. Never delegated to the model.
Output filtering for PII, credentials and system-prompt leakage before anything reaches a user.
Audit logs of every AI-influenced decision, retained per your compliance requirements.
Know where inference runs. Data residency is a contractual and regulatory question with real answers, and some providers let you pin it.
Rotate keys, scope them per environment, never in source.

Cost controls that hold

Everything from lesson 4, plus the operational guards:

Per-user and per-tenant rate limits, so one caller cannot consume the budget.
Spend alerts at a fraction of budget, not at budget.
Token caps per request, sized from measured p99 rather than the maximum.
A kill switch for expensive routes.

The failure worth naming: a retry loop around a route that fails often, quietly multiplying cost. Alert on retry rate, not only on error rate.

The readiness checklist

Before an AI feature carries real traffic:

The summary worth keeping

Production AI is a small amount of model selection and a large amount of ordinary engineering: validation, observability, cost control, rollout discipline, and failure isolation.

The model is the part that changes every few months. The engineering around it is the part that determines whether your system survives that.

Try this: Take your highest-traffic AI route and ask three questions. Which prompt version served the last thousand requests? What was the cache hit rate? What happens if the provider returns 503 for ten minutes? If any answer is a shrug, that is your next piece of work — and it is worth more than any prompt improvement.

What you'll be able to do