LLM gateway — routing, fallback, and cost attribution across providers

Intuition

The rate-limit handling covered on llmops — backoff, a circuit breaker, a fallback model — describes what to do when a provider fails. An LLM gateway is where that logic lives once it stops fitting comfortably inside application code: a single service every LLM call routes through, which knows every provider’s current health, every tenant’s spend, and which request should fail over to what, so that logic exists in one place instead of being duplicated (and inevitably drifting) across every service that calls a model.

Mechanics

The shape: one interface, many providers behind it

Application code
      |
      v
LLM Gateway  --- health, routing, cost tracking ---
      |
   +--+--+------+
   v     v      v
OpenAI Claude  Local Llama

Application code calls one interface — “generate a response for this prompt, this model class” — without knowing or caring which actual provider serves the request. This is the same abstraction-boundary argument made elsewhere on the site (a repository hiding a database engine, an interface hiding an implementation): the gateway absorbs provider-specific detail so callers don’t have to change when a provider is added, removed, or temporarily routed around.

Routing and fallback

async def generate(prompt, model_class="general"):
    for provider in ROUTING_TABLE[model_class]:
        if circuit_breaker.is_open(provider):
            continue
        try:
            return await provider.generate(prompt)
        except (RateLimitError, ProviderOutageError):
            circuit_breaker.record_failure(provider)
            continue
    raise AllProvidersUnavailable()

The routing table is ordered by preference (cost, latency, quality) per model class, and the gateway walks it, skipping any provider whose circuit breaker is open — the same circuit-breaker mechanism covered generally on cascading-failures, applied here to provider health specifically rather than a single downstream dependency. A provider outage becomes a routing decision made once, in one place, rather than a try/except block duplicated across every call site that happens to remember to add it.

Cost attribution: the metric a gateway makes free

Because every request passes through one place, tagging it with a tenant ID, a feature name, and the resulting token counts turns “what does this cost” from a manual reconciliation exercise into a query:

@dataclass
class GatewayRequestLog:
    tenant_id: str
    feature: str
    provider: str
    model: str
    input_tokens: int
    output_tokens: int
    cost_usd: float
    latency_ms: int

This is the natural extension of the per-request logging already covered on llmops — a gateway is where per-tenant cost attribution becomes practical, because it’s the one place that sees every tenant’s traffic across every feature and provider, rather than each service independently logging its own slice.

429 handling, centralized

if response.status == 429:
    retry_after = response.headers.get("Retry-After")
    backoff.schedule(provider, retry_after)
    circuit_breaker.record_failure(provider)

A 429 handled once, in the gateway, with the provider’s own Retry-After header respected, replaces N copies of similar-but-subtly-different retry logic scattered across every service that calls that provider — each of which is a separate opportunity for the retry-storm mistake covered on cascading-failures.

Cost & limits

A gateway is a new single point of failure, and its own latency and reliability now sit on the critical path of every LLM call in the system — a gateway that adds meaningful overhead per request, or that itself goes down, degrades everything routed through it, which is exactly the architectural tradeoff a shared component always makes: consolidating logic in one place also consolidates risk there.

Building and operating a gateway is real, ongoing engineering work, not a configuration toggle — routing tables, circuit breaker state, and cost attribution logic all need to be built, tested, and kept current as providers and models change, which is a cost that has to be justified by the number of call sites and providers actually involved.

When NOT to use it

Do not build a gateway for a single provider, single-service system. The entire value proposition is centralizing logic that would otherwise be duplicated across multiple call sites or multiple providers — with one provider and one caller, there’s nothing to centralize yet, and the gateway is pure overhead.

Do not route every request through the same fallback chain regardless of what the request actually needs. A request requiring a specific model’s particular capability (a very long context window, a specific fine-tune) shouldn’t silently fail over to a provider that can’t actually satisfy it — the routing table needs to encode real capability constraints, not just a preference order, or a “successful” fallback produces a response that doesn’t actually meet the request’s requirements.

Real-world usage

Any system calling more than one LLM provider, or one provider with more than a couple of call sites, converges on some version of a gateway — either a self-built thin routing service, or an off-the-shelf product built for exactly this purpose. The trigger is usually the second provider being added: at that point, provider-specific logic scattered across services becomes a maintenance burden the gateway pattern directly solves, and the per-tenant cost visibility becomes valuable enough on its own to justify the investment.

Failure modes

The gateway that became the outage. A gateway with no redundancy of its own, or with an internal bug (a routing table update that accidentally points every model class at a provider that’s down), turns a single point of failure into exactly that — every LLM call in the system fails at once, even though every actual provider might be healthy.

The fallback that silently served a worse response. A routing table that falls back to a smaller, cheaper model on provider failure, with no signal to the caller that a fallback occurred, means a customer-facing feature quietly degrades in quality with nothing in the logs flagging it as different from a normal, successful response — discovered only when someone notices response quality dropped and has to reconstruct, after the fact, which requests actually hit the fallback path.

The cost attribution that was wrong because tagging wasn’t enforced. A gateway that allows calls without a required tenant_id or feature tag accumulates a growing bucket of unattributed cost — the exact metric the gateway exists to make easy becomes unreliable because nothing enforced that every call site actually populated it.

Practice problems

1. A team has calls to OpenAI scattered across six different services, each with its own retry and backoff logic, subtly different from the others. What’s the argument for consolidating this into a gateway, beyond “it’s tidier”?

Six independent retry implementations are six independent opportunities for the retry-storm mistake — a naive, unbounded retry loop in even one of them can amplify load on a struggling provider regardless of how well the other five are implemented. Consolidating into a gateway means the backoff/circuit-breaker logic is correct in exactly one place, and fixing a bug in it fixes it everywhere at once, rather than requiring six separate patches applied consistently.

2. A gateway’s fallback chain routes a request that needed a 200k-token context window to a provider whose model only supports 32k tokens. The fallback “succeeds” — a response comes back. What’s actually wrong here?

The routing table treated provider preference as interchangeable without checking capability constraints — a response that comes back doesn’t mean the request was actually served correctly if the model silently truncated or couldn’t process the full context it needed. The routing table needs to encode hard capability requirements (context length, specific features) as constraints that rule out incompatible providers entirely, not just soft preferences ordered by cost or latency.

3. Cost attribution via the gateway shows a large “unattributed” bucket growing each month. What does this suggest, and how would you fix it?

Some call sites are making requests without populating the required tenant_id or feature tags — the gateway is only as good as the tagging discipline enforced at the call site. The fix is making those tags mandatory at the gateway’s API boundary (reject or flag untagged requests) rather than optional fields that are easy to forget, since the whole value of centralizing cost attribution depends on every request actually carrying the metadata needed to attribute it.

Check yourself

A team routes every LLM call in their system through a central gateway for provider fallback and cost tracking. What real tradeoff does this introduce?

• None -- centralizing is strictly better with no downside
• The gateway becomes a new single point of failure -- its own latency and reliability now sit on the critical path of every LLM call in the system
• It only affects cost tracking, not reliability
• Gateways eliminate the need for circuit breakers entirely

Interview answers

“Why would you build an LLM gateway instead of handling provider fallback in application code?” Once more than one provider or more than a couple of call sites are involved, provider-specific retry, circuit-breaking, and fallback logic duplicated across services becomes a maintenance and correctness risk — each duplicate is an independent chance to get the retry logic wrong, and a gateway consolidates it into one place, tested once. The caveat that shows real production experience: this isn’t free — the gateway becomes a new single point of failure and a genuine engineering investment, which is why it’s not worth building for a single-provider, single-call-site system.

“How would you track LLM cost per customer across multiple services and providers?” Route every call through a gateway that logs tenant ID, feature, provider, model, and token counts per request — this turns per-tenant cost attribution from a manual reconciliation exercise into a query, because the gateway is the one place that sees every tenant’s traffic across every feature and provider. The caveat: this only works if tagging is enforced, not optional — a gateway that allows untagged calls accumulates an unattributed cost bucket that undermines the entire point of centralizing the tracking.