Understanding LLM Gateways: A Full Architecture Breakdown
A breakdown of LLM gateway architecture, covering the six internal layers and the eight stages a request passes through.
TL;DR
- An LLM gateway architecture has six processing layers: ingestion, routing, authentication, cache, provider communication, and observability. Each one sets a ceiling on throughput, latency, or reliability.
- A request passes through eight stages, from HTTP receipt to normalized response. The cache check runs once a provider has been selected, and a hit short-circuits the provider call itself.
- The language runtime is the most consequential decision. Python's global interpreter lock forces multi-process scaling for CPU-bound routing work, while Go multiplexes goroutines across cores in one process.
- Bifrost, the open-source AI gateway built in Go by Maxim AI, adds 11 microseconds of overhead per request at 5,000 requests per second on a t3.xlarge instance, with per-provider worker pools that stop one slow provider from blocking the rest.
- Cache placement matters at this scale: a gateway with 11 microseconds of overhead loses more to a 1-millisecond external cache round-trip than to its entire pipeline.
An LLM gateway is a dedicated infrastructure layer that sits between your applications and LLM providers, handling routing, authentication, caching, observability, and governance for every AI request. The architecture choices made at each layer (language runtime, concurrency model, caching backend, plugin system) directly determine throughput, latency overhead, and operational reliability in production. Bifrost, the open-source AI gateway built in Go by Maxim AI, is built around these constraints, adding 11 microseconds of overhead per request at 5,000 requests per second. This article goes deep on the internal architecture of an LLM gateway: what each layer does, how requests flow through the system, and why the implementation decisions matter at scale.
The Core Layers of an LLM Gateway Architecture
A production-grade LLM gateway architecture consists of six distinct processing layers. Each layer has a specific responsibility, and the design of each layer affects the overall system's behavior under load.
| Layer | Responsibility | What a poor design costs |
|---|---|---|
| Request ingestion | Parses the HTTP request, validates the schema, extracts credentials, manages streaming | Per-request parsing overhead on every call, and buffered streams that break time-to-first-token |
| Routing and provider selection | Applies model-to-provider mappings, key weights, fallback chains, and health exclusions | Requests sent to a provider that is already failing |
| Authentication and key management | Resolves the incoming credential to provider keys, budgets, rate limits, and model access | Quota overruns discovered only on the provider invoice |
| Cache | Checks exact-match and semantic hits before any provider call | Paid provider calls for answers the gateway already holds |
| Provider communication | Translates the internal request into each provider's API shape and normalizes the response back | A new provider integration becomes an application-code change |
| Observability and logging | Captures latency, token counts, cache hits, and errors, then emits metrics and logs | No way to attribute cost or diagnose a latency regression |
The table lists responsibilities, not execution order. The hub guide to LLM gateway architecture, features, and use cases covers what each layer is for, and this breakdown covers how each one is built.
Request Ingestion Layer
The ingestion layer receives incoming HTTP requests and translates them into the gateway's internal request schema. For gateways that expose an OpenAI-compatible API, this layer parses the JSON body, validates the schema, extracts authentication credentials from headers, and constructs an internal request object.
Streaming support is handled here too. The ingestion layer must manage chunked transfer encoding for streaming completions without buffering the entire response.
Bifrost's ingestion layer is built on FastHTTP, which adds approximately 2.1 microseconds for parsing typical requests. The drop-in replacement design means any OpenAI SDK, Anthropic SDK, LangChain, or LiteLLM-based application can point at Bifrost by changing only the base URL, with no changes to application code.
Routing and Provider Selection Layer
The routing layer decides which provider and which API key receives a given request. Provider selection evaluates routing rules: model-to-provider mappings, weighted distributions across API keys, fallback chains, and health-based exclusions. In a correctly implemented routing layer, provider selection is deterministic when health is known and probabilistic when distributing across multiple healthy keys.
Provider routing must also account for circuit breaker state: if a provider is returning 5xx errors or has exhausted its rate limit budget, the routing layer must route around it without requiring application code changes. Circuit-breaker handling belongs in the gateway, not in application code.
Authentication and Key Management Layer
The authentication layer resolves the incoming credential to a set of provider API keys and attaches any per-consumer policies. In gateways that support virtual keys, the incoming credential is a logical identifier that maps to one or more real provider API keys, plus associated budget limits, rate limits, and model access controls. The gateway resolves this mapping before any provider call is made.
The authentication layer is also where rate limits are enforced. Requests that exceed a virtual key's quota are rejected before they consume any provider tokens. Budget limits are tracked here as well, allowing per-consumer cost controls at the infrastructure layer.
Cache Layer
The cache layer intercepts requests before they reach a provider and checks for a matching cached response. Exact-match caching is the baseline: the request is hashed and looked up in a cache store. Semantic caching extends this by comparing the query embedding against cached embeddings and serving a response if the similarity score exceeds a configured threshold.
The cache layer's position in the pipeline matters. A cache hit must short-circuit the provider call itself and everything downstream of it, where the cost and latency sit. Cache writes happen asynchronously after a provider response is returned, so they add no latency to the first request. Bifrost's semantic caching uses a vector store backend supporting Redis/Valkey, Weaviate, Qdrant, and Pinecone.
Provider Communication Layer
The provider communication layer translates the normalized internal request into the provider's specific API format, sends it, and normalizes the response back into the gateway's common schema. Each provider has a distinct API contract: OpenAI, Anthropic, Bedrock, and Vertex all use different request and response shapes. The translation layer is what makes a single gateway usable across 25+ providers and 10,000+ models.
For streaming responses, this layer must handle chunked responses, translate streaming formats per provider, and pass the stream back to the ingestion layer for delivery to the client.
Observability and Logging Layer
The observability layer captures request metadata, provider responses, latency breakdowns, token counts, cache hits, and error codes. That telemetry feeds metrics systems through Prometheus or OpenTelemetry. The log store holds structured request logs that can be queried or exported.
Bifrost's observability layer exports OTLP traces that Grafana, New Relic, and Honeycomb consume, exposes Prometheus metrics directly, and ships a native Datadog connector. Logging writes are asynchronous, so they do not block the response path.
How a Request Flows Through an LLM Gateway
A request entering an LLM gateway is received and parsed, authenticated against a virtual key, routed to a provider and key, checked against the cache, translated into that provider's API format, forwarded with retry and fallback handling, logged asynchronously, and returned in the client's expected format. A cache hit ends the sequence before the provider is called.
The following sequence covers a standard non-cached request through a production LLM gateway. The eight stages below follow the Bifrost request pipeline:
- Receive: The HTTP transport layer accepts the incoming POST request (e.g.,
/v1/chat/completions), parses headers and body, and validates the JSON schema. - Authenticate: The virtual key in the request header is resolved to a set of provider API keys and associated policies (rate limits, budget limits, model access).
- Route: The routing layer selects a provider and API key based on routing rules, weights, health status, and fallback chains.
- Cache check: Once a provider is selected, the normalized request is checked against the cache in the per-attempt hook phase. If a direct hash match or a semantic similarity match is found above the configured threshold, the cached response is returned and the pipeline stops here.
- Translate: The internal request object is translated into the provider's specific API format.
- Forward: The request is dispatched to the provider. On 429 or 5xx responses, the retry and fallback logic triggers: the gateway rotates API keys or switches to a fallback provider before returning an error.
- Log: The response metadata is written asynchronously to the log store and metrics are emitted.
- Return: The normalized response is returned to the client in the expected format, with any streaming chunks passed through as they arrive.
This pipeline adds 11 microseconds of overhead at 5,000 RPS in Bifrost benchmarks; detailed results are published on the Bifrost performance benchmarks page.
Architecture Decisions That Affect Performance
Four implementation choices set the performance ceiling of an LLM gateway: the language runtime, the concurrency model, where the cache lives, and how plugins execute. Each one is difficult to reverse after deployment, because each shapes how the gateway behaves under concurrent load rather than what features it offers. The language runtime and concurrency model are the most consequential of the four.
Go vs. Python runtimes. Python's global interpreter lock prevents true thread-level parallelism for CPU-bound operations. LLM gateway routing logic, key selection, and plugin execution are CPU-bound. A Python-based gateway requires multiple processes to scale horizontally on a single machine, adding memory overhead and coordination complexity.
Go has no such lock: goroutines are multiplexed across available CPU cores by the Go scheduler, and thousands of concurrent goroutines add minimal memory overhead compared to OS threads.
Concurrent worker pools vs. thread pools. Thread pools in traditional languages allocate OS threads that are expensive to create and context-switch. Go's goroutine-based worker pools are cooperative and lightweight.
In Bifrost's concurrency model, each provider has an independent worker pool: OpenAI requests go to the OpenAI pool, Anthropic requests go to the Anthropic pool. A slowdown in one provider's response times does not block workers serving other providers.
In-process caching vs. external cache services. External cache services (Redis, Memcached) introduce a network round-trip per cache lookup. For a gateway adding 11 microseconds of total overhead, a 1-millisecond cache round-trip is significant. Bifrost's vector store integration is designed to minimize this: direct-only (hash-based) cache lookups are a single round-trip; semantic lookups add an embedding call before the vector search.
Plugin architectures. A gateway that extends behavior through a plugin pipeline must ensure that plugins cannot block the request path. Bifrost's plugin architecture uses a hook-based model with pre- and post-processing phases, with failure isolation that prevents a misbehaving plugin from crashing the core system.
The four decisions, and what each one buys:
| Decision | Failure mode it avoids | Bifrost's implementation |
|---|---|---|
| Language runtime | Multi-process scaling and its memory and coordination overhead | Go, with goroutines scheduled across cores in a single process |
| Concurrency model | One slow provider starving workers serving every other provider | An independent worker pool per provider, with channel-based hand-off |
| Cache placement | A millisecond network round-trip on a pipeline measured in microseconds | Single round-trip direct lookups, with embedding calls only on semantic search |
| Plugin execution | A custom plugin blocking or crashing the request path | Pre- and post-hook interfaces with failure isolation, in Go or WASM |
Teams comparing these choices across products will find the same dimensions in the reference architecture for scaling LLMs safely.
How Bifrost's Architecture Handles Production Scale
The Bifrost gateway implements each of the architecture layers above with Go concurrency primitives and a framework designed for predictable overhead. The concurrent worker pool architecture isolates providers: each provider has its own goroutine pool with channel-based communication between components. Object pools (sync.Pool) handle memory reuse for request and response objects, keeping garbage collection pressure low.
The config store holds the live configuration for providers, virtual keys, routing rules, and caching settings. The config store is the source of truth for the routing and authentication layers. The model catalog maps model identifiers to providers and capabilities, covering 10,000+ models across 25+ providers.
Plugin execution is managed by a central plugin manager. Plugins operate through well-defined pre- and post-hook interfaces. Go and WASM plugin formats are supported, allowing organizations to add custom business logic without modifying the core gateway binary.
Published benchmark runs show 11 microseconds of added latency per request at 5,000 sustained RPS. The weight calculation for adaptive load balancing, a Bifrost Enterprise feature, runs asynchronously every 5 seconds, so hot-path routing uses pre-computed weights with less than 10 microseconds of selection overhead.
MCP Integration in the Gateway Architecture
The MCP gateway layer extends the core LLM gateway architecture with a protocol translation layer for the Model Context Protocol. Bifrost connects to external MCP servers, manages authentication (including OAuth 2.0 with automatic token refresh), and exposes those tools to clients such as Claude Desktop. Tool calls are dispatched through the gateway rather than by the client, so they inherit the same governance and logging as inference traffic.
Code Mode is an architectural optimization specific to multi-server MCP deployments. Instead of exposing all tool definitions to the model on every request (which at 500+ tools consumes the majority of the model's context budget), Code Mode exposes four meta-tools and executes Starlark Python in a sandbox to orchestrate the underlying tools. At 508 tools across 16 MCP servers, this reduces input tokens by 92.8% and estimated cost by 92.2%. The MCP gateway resource page covers the full architecture, and MCP proxy server architecture covers how the protocol layer differs from the LLM request path.
Enterprise Architecture Considerations
For production deployments that need high availability, Bifrost Enterprise provides HA clustering with gossip-based synchronization across nodes and zero-downtime deployments. Cluster nodes share routing state and virtual key usage metrics, so load balancing weights remain consistent across instances.
In-VPC deployments allow the gateway to run inside a private cloud network with complete isolation inside the VPC and no external network dependencies. In-VPC deployment is the standard pattern for regulated industries where data must not leave the organization's network perimeter. Guardrails apply content safety policies at the gateway layer before requests reach providers. Audit logs record administrative activity as HMAC-signed, verifiable events covering who changed a provider, virtual key, or policy and when, with configurable retention and archiving to object storage for long-term review.
RBAC and SSO/OIDC integration (Okta, Entra, Keycloak) connect the gateway's access control layer to enterprise identity providers. For teams evaluating enterprise-grade LLM gateway options, the LLM Gateway Buyer's Guide provides a detailed capability matrix across the relevant dimensions, and the overview of what an LLM gateway is and where it is used sets the wider context.
Frequently Asked Questions
What is an LLM gateway?
An LLM gateway is an infrastructure layer between applications and model providers that handles routing, authentication, caching, observability, and governance for every AI request. Applications call one endpoint instead of several provider APIs, and policy that would otherwise live in application code moves into the gateway, where it applies to every caller at once.
What are the layers of an LLM gateway architecture?
Six: request ingestion, routing and provider selection, authentication and key management, cache, provider communication, and observability. They are a catalog of responsibilities rather than a strict sequence, and the execution order differs by implementation. Each layer sets an independent ceiling on throughput, latency, or reliability, so the weakest one determines production behavior.
How much latency should an LLM gateway add?
Gateway overhead should be a rounding error next to provider response time, which runs in hundreds of milliseconds to seconds. Bifrost adds 11 microseconds per request at 5,000 requests per second. At that scale the dominant risk is not the gateway pipeline but anything that introduces a network round-trip inside it, such as an external cache lookup.
Why does the programming language of an LLM gateway matter?
Routing logic, key selection, and plugin execution are CPU-bound, and a runtime with a global interpreter lock cannot run them in parallel across threads. Scaling then requires multiple processes on each machine, which adds memory overhead and coordination complexity. A runtime with lightweight scheduled concurrency, such as Go, handles the same load in one process.
Where does caching belong in the gateway pipeline?
Before the provider call, so a hit skips the paid request and the seconds of latency behind it. Exact-match lookups hash the normalized request; semantic lookups compare the query embedding against stored embeddings above a similarity threshold. Writes happen after the response is returned, so the first request never pays for populating the cache.
What is the difference between an LLM gateway and an MCP gateway?
An LLM gateway routes model inference requests to providers. An MCP gateway routes tool calls, connecting to Model Context Protocol servers, managing their authentication, and exposing their tools to clients. Bifrost runs the tool-routing layer inside the same gateway, so tool traffic and inference traffic share one set of governance controls.
Deploy and Configure Bifrost
The architecture above is what a deployment inherits by default; the work of standing one up is configuration rather than assembly. The quickstart guide covers getting Bifrost running, and provider configuration covers adding API keys for the providers you use.
Two decisions are worth making before the first production request rather than after. The first is whether the cache runs direct-only or adds semantic matching, because semantic mode requires an embedding-capable provider and pays an embedding round-trip on every direct miss. The second is how virtual keys map to teams and services, since that mapping determines what budget and rate-limit granularity is available later, and reshaping it after issuance means reissuing credentials.
The Bifrost Enterprise page covers clustering, VPC deployment, and compliance options for production deployments.
To see Bifrost's architecture in practice and discuss enterprise deployment options, book a demo with the Bifrost team.
