`caching`¶

The caching section enables a simple in-memory exact-match response cache.

Fields¶

Field	Type	Default	Notes
`enabled`	bool	`false`	master switch
`ttl`	duration	`1h`	cache entry lifetime
`max_size`	integer	`1000`	max number of cached entries
`max_entry_bytes`	integer	`16777216`	maximum retained size of one key and response (16 MiB)
`max_bytes`	integer	`67108864`	maximum retained size across all entries (64 MiB)

Example¶

caching:
  enabled: true
  ttl: 30m
  max_size: 5000
  max_entry_bytes: 16777216
  max_bytes: 67108864

What kind of cache this is¶

This is not semantic caching. It is a straightforward in-memory cache keyed from the complete canonical client JSON, its normalized request, the resolved provider, the target's effective model and upstream protocol, and selected upstream control headers. Requests that differ in output-affecting controls such as stop, response_format, penalties, logit_bias, seed, reasoning effort, or tool configuration do not share an entry. Provider feature and idempotency headers that can affect the result (OpenAI-Beta, Anthropic-Beta, and Idempotency-Key) also participate in the key.

The effective model is resolved in this order: the route target's model, the request model for that provider, and then the provider's default_model. LunarGate uses the same effective-model calculation for lookup and storage. If a fallback target succeeds with a different model, its response is retained only under that fallback model's key; it cannot satisfy a later lookup for the primary model.

Eligible requests¶

LunarGate caches only successful responses from operations that are safe to replay:

non-streaming POST /v1/chat/completions requests that do not explicitly include store
POST /v1/embeddings requests

The following always bypass response caching:

every POST /v1/responses create, including store: false
streaming requests
Chat Completions requests with an explicit store: true or store: false
stored Chat Completions, Responses lifecycle, Conversations, cancellation, compaction, input-token counting, and every other state-changing operation

An explicit storage policy is itself a side-effect contract. Serving store: true from cache would skip creation of the upstream object, while retaining a store: false result in the response cache would violate the caller's requested storage policy.

Use it when:

prompts repeat often
responses are deterministic enough for your use case
a single gateway instance handling repeated traffic is enough

Practical guidance¶

Keep it off if you are still validating routing behavior and want every request to hit the provider.
Use X-LunarGate-No-Cache: true for request-level cache bypass when needed.
Both the entry-count and total-byte limits apply at the same time. LunarGate evicts the oldest entries until a new response fits both limits.
Byte accounting includes the cache key and retained serialized response buffers; it intentionally excludes fixed Go object overhead. A response larger than max_entry_bytes is served normally but is not retained. max_entry_bytes must not exceed max_bytes.
Remember that this cache is process-local and disappears on restart.

Hot reload behavior¶

Reloading an effectively identical caching section preserves unexpired entries. Changing any cache setting replaces the cache configuration and clears all existing entries so values retained under different TTL or capacity semantics are not reused.

Provider reloads use a separate safeguard. If a provider definition changes, LunarGate rotates an opaque process-local cache namespace for that provider alias. Requests already in flight may finish with the previous provider generation, but their later cache writes cannot be read by requests using the new account, endpoint, credentials, translator settings, or model configuration. Unchanged provider aliases retain their namespace and can continue using otherwise eligible entries.

caching¶