Prompt Caching and Session Affinity

TL;DR

• Prompt caching discounts repeated input tokens, but only when a request reaches the same cache scope that stored the prefix: the same provider, organization or workspace, and on OpenAI, ideally the same machine.
• Load balancing across providers and API keys spreads one conversation across several cache scopes, so turns that should read the cache pay to write it again.
• Session affinity binds a session ID to the provider and key that last served it, so later turns go back to the warm cache while routing still decides what is allowed.
• Bifrost applies session affinity at the provider level and the key level, writes bindings only on served requests, and moves a binding when a fallback serves.
• Claude Code, Codex CLI, and OpenCode get session affinity through Bifrost with no configuration, because Bifrost adopts the session headers they already send.

Session affinity is the routing behavior that keeps every request in a session on the provider and API key that served the session before. For LLM traffic it matters because of prompt caching: providers store a processed prompt prefix and charge far less to reuse it, but only for requests that arrive where that prefix is stored. Bifrost, the open source AI gateway built by Maxim AI (GitHub), adds session affinity on top of its routing so that multi-turn conversations and agent runs keep landing on a warm cache.

How Prompt Caching Works and Why Placement Matters

Prompt caching stores the processed form of a prompt prefix (system prompt, tool definitions, earlier turns) so the next request that starts with the same bytes skips that work. The prompt caching guide covers the mechanics in depth; the part that matters here is the price difference between writing and reading.

On Anthropic, cache writes with the default 5-minute lifetime cost 1.25 times the base input price, 1-hour writes cost 2 times, and cache reads cost 0.1 times on most models (0.05 or 0.025 times on some newer ones), according to Anthropic's prompt caching documentation. OpenAI applies caching automatically and discounts reused input tokens by up to 95%, depending on the model. Reading a prefix costs a fraction of rewriting it.

The catch is placement. A cache is not global:

• Anthropic isolates caches by workspace on the Claude API, Claude Platform on AWS, and Microsoft Foundry, and by organization on Bedrock and Google Cloud. Different organizations never share a cache.
• OpenAI keeps cached state on individual machines. Its prompt caching guide says a request can reuse a prefix only if it reaches a machine holding a matching entry, and caches are not shared across organizations or regional processing boundaries.
• Different providers never share a cache. A prefix warmed on Azure OpenAI does nothing for a request sent to OpenAI directly, even for the same model.

Costs per model are in the Claude API pricing and OpenAI API pricing breakdowns.

Why Load Balancing Breaks Cache Hits

An LLM gateway usually spreads traffic for reliability and throughput. A virtual key might send 70% of gpt-4o traffic to OpenAI and 30% to Azure, and pick among several keys on each provider. Each request is placed independently, which suits single-shot calls but not conversations.

In a ten-turn agent session, each turn re-sends a long, growing prefix. If turn two is weighted to a different provider, or to a key in a different organization, it misses the cache turn one wrote and pays the write price again.

Figure 1: Provider caches are scoped to an organization, workspace, or deployment, so a turn that lands on a key in another scope writes the prefix again.

What Session Affinity Is

Session affinity, also called session stickiness or sticky sessions, routes every request that shares a session ID back to the destination that served that session before. For LLM traffic, the state being protected is the provider's prompt cache.

Bifrost's session affinity defines a session as a conversation, an agent run, or any group of requests sharing a session ID. Its three benefits are:

• Cache hits: the session keeps landing where its cache lives.
• Predictable rate limits: one session draws on one key's bucket.
• Consistency: the same provider serves every turn, so tool behavior and response style do not drift mid-run.

Affinity does not create or manage provider caches; it only changes which allowed destination is tried first.

How Bifrost Identifies a Session

A request joins a session through its session ID. Bifrost reads it from these sources, in order:

The request options reference lists the full header priority. Because these tools already send the headers, pointing Claude Code at Bifrost gives every session affinity with no configuration, and Claude Code subagents send their parent's session ID, so parallel subagents stay on one provider and one key. The same applies to other CLI coding agents.

Bindings are scoped to the virtual key, the user behind it, and the session ID. The same session ID under two virtual keys is two independent sessions, so one team cannot steer another team's routing.

The Two Levels of Session Affinity

Bifrost applies session affinity after its routing engines have built the list of allowed providers and keys. Routing rules, virtual key load balancing, the model catalog, and Adaptive Routing (Enterprise) decide what may be used; affinity only reorders that list.

Figure 2: Routing still decides which providers and keys are allowed; affinity only reorders them so the session lands where its cache lives.

Provider level. When a caller asks for a bare model such as gpt-4o and routing offers more than one provider, the provider and model that last served the session go first, as long as routing still offers that exact route. A request that names its provider (openai/gpt-4o) is never reordered. A provider that routing excluded for budget, rate limits, model access, or health is never brought back by a session.

Key level. Within the chosen provider, Bifrost reuses the bound key while it stays in the eligible pool. Without a binding, key selection is weighted random. If the bound key leaves the pool, the binding is deleted and a key is picked normally.

The Bifrost docs walkthrough shows the effect with a 70/30 OpenAI and Azure split. Turn one rolls Azure and binds key prod-eu-2. Turn two rolls OpenAI, but affinity moves Azure to the front, reuses prod-eu-2, and hits the prompt cache turn one warmed.

How Bindings Are Written, Refreshed, and Moved

Bifrost writes a binding only when a request is served. A failed request writes nothing, so a session is never bound to a provider or key that did not deliver. Each served request that followed the binding refreshes its time to live, which defaults to one hour.

Failures move bindings rather than trapping sessions. When the bound provider fails and a fallback serves, the fallback usually becomes the new binding at both levels and the old key binding is deleted.

Figure 3: Bindings are written only when a request is served, so a session never sticks to a provider or key that failed it.

Routing changes behave the same way. If a budget runs out on the bound provider and routing stops offering it, the binding is deleted and the session rebinds to whatever serves next. The guide to reliable fallback systems covers chain design.

Two behaviors are worth knowing before production:

• A bound key is reused on every retry. A session with a key binding does not rotate keys on a rate limit; the retry policy runs on the bound key. A key the provider rejects outright (bad credential, exhausted quota, no model access) moves the request to a sibling key, which becomes the binding.
• Weighted routing rules need fallbacks to stick. A rule that picks one weighted target and lists no fallbacks leaves nothing to reorder, so the weights are rolled on every request.

Controlling Session Affinity per Request

Session affinity is on by default for any request with a session ID. Three headers adjust it, all listed on the request headers page:

Match the binding TTL to the cache lifetime you pay for. Anthropic's 5-minute cache is refreshed free each time it is read, so an active session stays warm; a session idle for longer than the cache lifetime gains little from staying put, and a shorter TTL lets load balancing redistribute it. On OpenAI models before GPT-5.6, keep a stable prompt_cache_key per session as well, since that key helps OpenAI route related requests to the same machine inside one organization. Session affinity gets the request to the right organization and key; prompt_cache_key helps it find the right machine there.

Pairing Session Affinity with Automatic Prompt Caching

Affinity gets a request to the cache; something still has to mark what to cache. Anthropic and Bedrock need a cache marker, at block or request level, and many clients send none. Bifrost's automatic prompt caching (prompt_cache.auto_inject) adds a marker for clients that send none:

• Off by default, enabled per provider, because a cache write costs more than a plain input token.
• Caller markers always win: a request that already carries cache_control or prompt_cache_breakpoint is forwarded unchanged.
• Overridable per request with the x-bf-prompt-cache-auto-inject header, on providers that already have a prompt_cache block configured.
• Byte-stable: the marker goes on the first cacheable block, so the cached prefix stays identical across turns.

Together, the two features cover both halves: injection makes the prefix cacheable, and session affinity makes sure later turns reach it. The Bifrost LLM cost optimization post and the comparison of AI gateways with prompt caching support put these features alongside other cost levers.

Prompt caching differs from semantic caching, which answers a similar request from a stored response without calling the provider at all.

Session Affinity in Clusters and with Adaptive Routing

Bindings live in Bifrost's key-value store. The gateway always creates one; Go SDK users set KVStore themselves. In a Bifrost Enterprise cluster, the store replicates, so a session that reaches any node follows the same binding.

Without health signals, a session follows its binding until a request actually fails. Bifrost Enterprise adds health-aware invalidation: before following a binding, a node checks its adaptive routing load balancer and circuit breaker, and drops the binding if the bound provider or key is marked failed there. The closer look at Bifrost adaptive routing explains how those health scores are computed.

Observing Session Affinity in Logs and Traces

Every affinity decision appears in the request's routing trail under the session-affinity engine, such as "Session stays on azure/gpt-4o for gpt-4o; routing proposed openai/gpt-4o," so you can confirm a session is sticking and see why it moved.

For tracing, Bifrost's OpenTelemetry integration puts session.id on the root span, and group_traces_by_session groups a whole session into one trace. Combined with the cached-token counts providers return, this shows cache hit rate per session rather than per request, which is the view that session-level observability argues for.

Frequently Asked Questions

What is session affinity in an LLM gateway?

Session affinity in an LLM gateway routes every request that shares a session ID to the provider and API key that served the session before. It preserves prompt cache hits and rate-limit predictability across multi-turn conversations and agent runs. In Bifrost, routing still decides which providers and keys are allowed; affinity only decides which allowed option is tried first.

Does session affinity improve prompt cache hit rates?

Yes, when traffic is spread across providers, organizations, or workspaces. Provider caches are scoped to those boundaries, so a turn routed elsewhere misses the prefix the previous turn wrote. Session affinity sends later turns back to the same provider and key. It cannot help when all keys share one cache scope or when prefixes change between turns.

Is session affinity the same as sticky sessions?

Yes. Sticky sessions, session stickiness, and session affinity name the same idea: keeping a client's requests on the destination that holds its state. For LLM traffic that state is the provider's prompt cache and the key's rate-limit bucket, so the destination is a provider plus a key.

What happens to a session when the bound provider fails?

Retries reuse the bound key until the provider rejects it outright. If the provider still fails and a fallback serves the request, Bifrost moves the session's binding to the fallback provider and key, and later turns start there. A failed request never writes a binding, so a session is not stuck on a provider that stopped delivering.

How do Claude Code and Codex CLI get session affinity?

Both already send a session identifier on every request. When no x-bf-session-id header is present, Bifrost adopts the first coding agent session header it finds, such as x-claude-code-session-id from Claude Code or session-id from Codex CLI. Pointing either tool at Bifrost, as in the Claude Code setup guide, enables session affinity and session-grouped traces with no extra configuration.

How long does a session binding last?

A Bifrost session binding lasts one hour by default, and every served request that follows it restarts that timer. The x-bf-session-ttl header sets a different lifetime per request, as a duration such as 30m or a number of seconds.

Keep Sessions on a Warm Cache with Bifrost

Prompt caching saves most on agents, coding assistants, and long chats, and session affinity keeps those savings when traffic is load balanced across providers and keys. Bifrost combines session affinity, automatic prompt caching, fallbacks, and budgets in one self-hosted LLM gateway, and the complete guide to prompt caching covers how to structure prompts so there is a prefix worth keeping. To see sticky routing on your own traffic, book a Bifrost demo.