Routing and fallback

Routing is where LunarGate becomes useful. You can keep one stable client integration and decide at the gateway which upstream provider or model should serve a request.

Routing model

A route matches a request and produces a list of targets.

routing:
  default_strategy: weighted
  routes:
    - name: force-provider-openai
      match:
        path: /v1/chat/completions
        headers:
          x-lunargate-provider: openai
      targets:
        - provider: openai
          model: gpt-5-nano
          weight: 100
      fallback:
        - provider: deepseek
          model: deepseek-chat
          weight: 100

What can be matched

• Request path
• Request headers

A common pattern is to route by team, environment, complexity tier, or a forced provider header.

The request path is also the cleanest way to separate:

• chat traffic on /v1/chat/completions
• Responses traffic on /v1/responses
• embeddings traffic on /v1/embeddings

Load-balancing strategy

The default strategy is round-robin. Set routing.default_strategy: weighted to select eligible targets according to their positive weights. Every configured target still requires a positive weight; round-robin and random selection do not use it for distribution.

Fallback behavior

If the primary target ends with an eligible provider or transport failure after its retry policy is exhausted, LunarGate can continue through the fallback chain. Client/request validation errors, cancellation, and client deadlines are terminal and never trigger fallback.

Primary target -> retries -> fallback 1 -> fallback 2 -> error

Every attempt uses the timeout configured for the provider that serves that attempt. Client cancellation is terminal: it stops retry and fallback and is reported as client_cancelled, not as a provider failure. Stateful creation and lifecycle operations also disable retry or fallback where replay could create or mutate an object twice.

First-match policy and explicit targets

Routes are evaluated from top to bottom, and the first matching route is the policy boundary for the request. LunarGate never skips that route merely because a requested target is absent from it.

• an explicitly requested provider that is unavailable in the first matching route returns status 400 with type: invalid_request_error, param: provider, and code: provider_not_found
• an explicitly requested model that is unavailable in that route returns status 400 with type: invalid_request_error, param: model, and code: model_not_found

Both errors are returned before an upstream call. Put narrow rules before broad ones and include every target that the narrow policy is meant to allow.

Tool-aware routing

When a request has tools available and does not explicitly set tool_choice: none, LunarGate injects x-lunargate-requires-tools: true. This includes omitted and auto choices because the provider may still call a tool. Existing tool-call continuations also require a tool-capable model, even if a client sends none alongside the history.

Complexity-based routing

The config can score requests and emit headers such as:

• x-lunargate-complexity
• x-lunargate-complexity-score
• x-lunargate-skill

Those headers can be used as route match inputs, which makes autorouting configurable rather than hardcoded.

Things to keep in mind

Tip

Put the more specific routes first. Header-based force routes and tool-capability routes should appear before general default routes.

Tip

If chat and embeddings use different upstream models, create separate routes for each path instead of trying to force both through one generic route.

Warning

A matching route does not fall through to a later route when an explicit provider or model is unavailable. The request fails closed before any upstream call.