Architecture

System overview

LunarGate is a self-hosted AI gateway that keeps your applications on one stable API while routing traffic to multiple providers.

Need the shortest path first?
Start with the quickstart page if you want to run the gateway before diving into architecture.

Need the routing mental model?
Jump to routing and fallback once the high-level flow here makes sense.

Need the YAML map?
Use the configuration overview when you are ready to translate the architecture into real config.

High-level flow

Client apps -> LunarGate Gateway -> OpenAI / Anthropic / Gemini / Groq / Together / OpenRouter / Abacus / DeepSeek / Ollama
                         |
                         +-> Prometheus metrics
                         +-> Optional Dashboard collector for metrics and request logs
                         +-> Optional Dashboard remote-control channel

One-sentence summary: the gateway sits between your app and upstream LLMs, keeps the client protocol stable, and moves policy decisions into config instead of application code.

Core responsibilities

Request normalization

The public API is OpenAI-compatible. LunarGate keeps native OpenAI-compatible envelopes intact when a route uses the same upstream protocol. When the client and upstream protocols differ, a provider translator maps only fields it can preserve faithfully and rejects unsupported fields with a structured compatibility error.

Routing

Routing is config-driven. A route can match on request path and headers, then send the request to one or more targets using weighted balancing.

Resilience

Safe, stateless inference calls can use retry logic, per-provider circuit breakers, and a compatibility-filtered fallback chain. Stateful lifecycle, continuation, Conversation, cancellation, and compaction operations are single-hop: they remain pinned to the creating provider and are never replayed against another account or fallback target.

Observability

The gateway exposes Prometheus metrics locally. It can also send metrics-only or full request-log data to the LunarGate Dashboard on app.lunargate.ai, depending on the data_sharing settings.

Remote control

The same data_sharing section can also attach the gateway to an outbound control channel for the LunarGate Dashboard on app.lunargate.ai.

Today that channel is mainly used for sandbox features. In the longer term it is meant to support things like automated A/B tests, controlled experiments, and other remote operations against a connected gateway.

Request lifecycle

1. Request arrives
2. Optional inbound auth is validated (`security.provider: api_key`)
3. Rate limiting is applied
4. The API handler captures one immutable provider/routing/model-selection runtime generation
5. Cache eligibility and lookup are evaluated
6. Matching route is selected
7. Target is picked by balancing strategy
8. Gateway selects native pass-through or a compatible provider translation
9. Safe stateless calls may execute retries and fallback; stateful calls use one pinned target
10. Response is normalized to the OpenAI-compatible shape
11. Metrics are recorded
12. Optional collector payload is sent to the LunarGate Dashboard on `app.lunargate.ai`

Hot reload model

Configuration is watched on disk. When the YAML file changes, LunarGate reloads the parsed config and reconciles the running components in memory.

Provider definitions, routing, and model selection form one coherent runtime generation. LunarGate builds their replacement before publication and then swaps the generation atomically. A failed build leaves the previous generation active. Each HTTP API request retains the generation captured at handler entry for its full lifetime, including routing, translation, retry/fallback target resolution, model discovery, and cache namespacing. Requests already in flight finish on the old generation; later requests use the new one.

A Responses WebSocket connection retains the generation captured during its handshake for every response.create message on that connection. Reconnect after a reload when the client should use the new provider/routing generation.

This is deliberately not a single transaction across every configuration section. Retry, cache, rate limiting, inbound auth, logging, collector, remote control, and update-check settings are reconciled separately after a valid config change.

Today that means you can update all of the following without restarting the process:

• provider definitions used by translators and model discovery
• routing rules and load-balancing strategy
• retry policy
• cache settings
• rate limiting
• inbound auth settings
• model-selection behavior
• logging level
• data-sharing / collector behavior
• remote-control enablement and identity details
• automatic update-check behavior

What still needs a restart:

• listener address and port
• server read/write/idle timeouts
• logging output format

The important practical point is that hot reload now covers real upstream/provider changes, not only route weights and a few lightweight knobs.

Objects created before a provider account, endpoint, or credential change are not rebound to the new account. Follow-up stateful calls fail closed when the retained owner identity no longer matches the current provider configuration.

Changing a provider definition also rotates that provider's process-local cache namespace. An old in-flight request can finish and write its result, but the new generation cannot read that entry. Provider aliases whose effective configuration did not change keep their namespace; changing the caching section itself clears the cache.

Protocol paths

OpenAI-compatible client request
  |-- same native upstream protocol -> preserve request/response envelope
  |-- different supported protocol -> validate, translate, normalize response
  `-- stateful follow-up -> resolve retained owner, one upstream call

Chat Completions and Responses support regular HTTP and SSE streams. Responses also supports a WebSocket create/stream mode. Embeddings uses regular HTTP.

Current constraints

• Inbound client authentication currently supports config-defined API keys only; external auth backends are still a TODO.
• Cache and rate limiting are in-memory only.
• Server bind address and HTTP timeout changes still require a process restart.
• The gateway speaks HTTP, SSE, and Responses WebSocket; it does not expose gRPC.

What to read next

1. Read Routing and fallback if you want to understand how this architecture turns into request decisions.
2. Read Observability and data sharing if you want to understand what stays local and what can be exported.
3. Open Configuration overview when you are ready to map these concepts into YAML.