What sits between your application and the model.

A route is a named policy: the model your application asks for, and the ordered list of upstream deployments allowed to answer it. Weights split steady-state traffic; order decides what happens when the preferred upstream is unavailable. Because the policy lives in the gateway, moving traffic between regions or providers does not require a deployment on your side.

Selection

Weighted choice among healthy primaries, then declared fallbacks in order.

Pinning

A route can be pinned to a region or a specific deployment for residency or contractual reasons.

Overrides

A single key, application or request header can be forced onto a specific upstream for testing or migration.

Change control

Route changes are versioned, and the version that served a request is recorded with it.

Provider incidents are not rare, and they rarely announce themselves cleanly. The gateway watches error rate, latency and rate-limit signalling per upstream, and treats a retryable failure as a routing decision rather than an application error. Attempts are bounded by a latency budget you set, so a bad upstream costs you a few hundred milliseconds instead of a timeout.

Retryable

Connection failures, timeouts, 429 rate limits and 5xx responses move to the next upstream.

Not retryable

Authentication failures, malformed requests and content-policy refusals return to you unchanged.

Budget

A per-route ceiling on total time. When the budget is spent, the last upstream error is returned honestly.

Streaming

Failover applies before the first token is emitted; once a stream has started it is not silently restarted.

Production traffic repeats more than most teams expect: the same classification, the same extraction, the same evaluation run twice. The cache keys on a normalized signature of the request, so a genuinely identical call is answered from storage — no provider tokens, no upstream latency, and no change to the response your client parses.

Signature

Model, messages, tools and the sampling parameters that affect the output. Nothing else is part of the key.

Scope

Cache entries are scoped to your organization. They are never shared between customers.

Control

TTL per route, plus a per-request bypass header for calls that must reach the provider.

Reporting

Hit rate is a first-class metric, so the saving is measurable rather than assumed.

The gateway is the one place that sees every call, so it is the right place to account for them. Each request produces a record with its route, upstream, model, latency, token counts and cache outcome. That record is what the analytics views read, what the API returns, and what your metered invoice is computed from — one number, not three that disagree.

Breakdown

By key, application, route, model and upstream, over any period in your retention window.

Latency

p50 and p95 measured at the gateway and at the upstream, so you can separate our time from theirs.

Chargeback

Per-key attribution makes internal cost allocation and end-customer billing straightforward.

Access

Queryable through the API and exportable for the billing period.

Two kinds of secret pass through the gateway: your RouteMind keys, and the upstream provider credentials you delegate to it. They are stored encrypted and are never returned by the API after creation. Request and response bodies are held only where a feature requires it — cache entries for their TTL, and nothing beyond that unless you turn on payload logging yourself.

Credentials

Upstream provider keys are encrypted at rest and write-only through the API.

Payloads

Not retained by default. Cache entries expire with their TTL; payload logging is opt-in and scoped per route.

Metadata

Usage records hold counts, timings and identifiers — not prompt or completion text.

Training

Your traffic is not used to train models. RouteMind does not build models.

A gateway that hides its own failures is worse than no gateway. Every routing decision is reported on the response, quota exhaustion returns a distinct error rather than a generic one, and configuration changes are versioned so a regression can be traced to the change that caused it.

Reporting

Provider, route, cache outcome and attempt count on every response header.

Errors

Gateway errors and upstream errors are distinguishable, with the upstream body preserved.

Limits

Per-key rate limits you set, enforced before an upstream is called.

Configuration

Routes are versioned; the version that served a request is stored with its usage record.