5 Best Open-Source LLM Gateways for Self-Hosted Deployments in 2026

TL;DR

• Self-hosting is the default for regulated and production AI in 2026: it answers data residency, audit trails for SOC 2 and HIPAA, fixed-cost economics at scale, and full inspectability of the routing layer, which managed services cannot.
• The deciding factors have converged on AI-native architecture, MCP readiness, and air-gapped deployment support.
• Five open-source gateways are compared: Bifrost, LiteLLM, Kong AI Gateway, Apache APISIX, and Envoy AI Gateway.
• Bifrost leads on performance (11 µs overhead at 5,000 RPS), native MCP support, and hierarchical governance, under Apache 2.0 and fully self-hostable.
• The others fit narrower niches: provider breadth (LiteLLM), existing Kong or APISIX estates, and Kubernetes-native stacks (Envoy); the buyer's guide compares the full set.

Enterprise AI teams running production workloads in 2026 face a common set of constraints that managed AI gateway services do not solve: data residency requirements for regulated industries, audit trails for SOC 2 and HIPAA, fixed-cost economics at scale, and the freedom to inspect every line of the routing layer. Open-source LLM gateways for self-hosted deployments have become the default answer, and the deciding factors now converge on AI-native architecture, MCP readiness, and air-gapped deployment support. Bifrost, the open-source AI gateway maintained on GitHub under the Apache 2.0 license and built by Maxim AI, is the best choice for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability, and leads this category on raw performance, MCP support, and governance depth. This guide compares five open-source LLM gateways worth evaluating for self-hosted deployments and the criteria that separate production-grade options from prototyping tools.

What Is a Self-Hosted Open-Source LLM Gateway

A self-hosted open-source LLM gateway is an infrastructure layer that runs inside an organization's own environment, sitting between AI applications and one or more LLM providers. It unifies provider APIs behind a single interface, enforces authentication and budgets, handles failover, and keeps prompt and response data within the perimeter. Because the source code is publicly licensed, teams can audit behavior, modify routing logic, and deploy in air-gapped or in-VPC environments without depending on a vendor's hosted control plane.

The shift in 2026 is that AI-native designs now treat MCP traffic, semantic caching, and per-consumer cost governance as first-class capabilities rather than plugins bolted onto a legacy proxy.

This guide covers the deployment and operations side specifically: how these gateways install, scale, and run inside your own infrastructure. For the category-level comparison of features and licensing, see the full comparison of open-source LLM gateways.

Why Self-Hosted Deployments Matter for Enterprise AI

Three pressures push enterprise AI teams toward self-hosted gateways, and each one independently rules out a managed service for the workloads it applies to. The in-VPC deployment guidance covers the first of them in more depth.

• Data sovereignty: regulators in healthcare, financial services, and the public sector require that prompt data, completions, and audit logs remain within national or organizational boundaries. Managed gateways route through the vendor's infrastructure by default, which limits deployment options for regulated workloads.
• Cost predictability: per-request gateway fees scale linearly with traffic, and AI workloads at scale produce request volumes that make hosted pricing models unattractive compared to fixed compute on owned infrastructure.
• Latency budgets: production AI applications often allocate single-digit milliseconds for the gateway hop. Self-hosted deployments run inside the same VPC or Kubernetes cluster as the calling application, eliminating cross-internet latency that managed services cannot avoid.

Key Criteria for Evaluating Open-Source LLM Gateways

The general capability criteria (provider coverage, MCP support, governance depth, caching, licensing) apply to any gateway and are covered in the category comparison and the LLM gateway buyer's guide. Four criteria matter specifically because you are the one running it:

• External state dependencies: what must exist before the gateway serves its first request. A gateway needing PostgreSQL and Redis turns one service into three, each with its own availability and backup plan.
• Gateway overhead at your instance size: latency added by the gateway itself, measured at sustained throughput on hardware comparable to yours. A figure quoted without instance size is not usable for capacity planning.
• Air-gapped viability: whether the gateway starts and runs with egress fully blocked, including pricing data and update checks. Test this rather than reading about it.
• Upgrade and scale mechanics: whether adding a replica requires coordination, and whether a version bump needs a maintenance window.

The 5 Best Open-Source LLM Gateways for Self-Hosted Deployments

The five gateways divide by origin. Bifrost and LiteLLM were built for LLM traffic; Kong AI Gateway, Apache APISIX, and Envoy AI Gateway are established proxies that added AI capabilities. For self-hosted deployments specifically, that difference shows up in external state dependencies, air-gapped viability, and whether AI governance sits in the free build or the commercial tier.

1. Bifrost

Bifrost is a high-performance open-source AI gateway built in Go by Maxim AI, designed from the ground up for production-scale self-hosted deployments.

In sustained benchmarks at 5,000 requests per second on a t3.xlarge instance, Bifrost adds 11 microseconds of overhead per request with a 100% request success rate. The same build adds 59 microseconds on a 2 vCPU t3.medium, so instance sizing is part of the number rather than separate from it. The full source code is available under the Apache 2.0 license, and the Bifrost LLM gateway GitHub repository is where releases, issues, and the benchmark suite live.

Core capabilities:

• Unified API across 25+ providers and 10,000+ models: a single OpenAI-compatible interface for OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, Azure, Mistral, Groq, Cohere, Ollama, vLLM, and more, with drop-in SDK compatibility.
• Native MCP gateway: functions as both an MCP client and server with Agent Mode for autonomous tool execution and Code Mode, which cut input tokens by 92.8% (75.1M to 5.4M) and ran roughly 40% faster in a benchmark spanning 508 tools across 16 servers. Full capabilities are documented on the MCP gateway resource page.
• Hierarchical governance: virtual keys are the primary governance entity, with per-consumer budgets, rate limits, and MCP tool allow-lists. Budgets can be set at virtual key, team, and customer levels.
• Reliability: automatic fallbacks and load balancing across providers and models, with no downtime when a provider returns errors.
• Semantic caching: dual-layer cache combining exact-match hashing and embedding-based similarity to cut cost and latency for repeated queries.
• Observability: built-in Prometheus metrics, OpenTelemetry tracing, and compatibility with Grafana, Datadog, New Relic, and Honeycomb.
• Enterprise deployment: in-VPC isolation, air-gapped deployments, clustering for high availability, HashiCorp Vault and AWS Secrets Manager support, HMAC-signed audit logs of administrative activity with configurable retention, and RBAC with SSO via Okta, Microsoft Entra, Keycloak, and Google Workspace. The Bifrost Enterprise tier covers the full feature set for regulated environments.

Bifrost deploys in under a minute via npx -y @maximhq/bifrost or a single Docker container, and integrates natively with CLI coding agents including Claude Code, Codex CLI, Gemini CLI, and Cursor.

Best for: Bifrost is built for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with robust security, policy enforcement, and governance capabilities.

2. LiteLLM

LiteLLM is a Python-based open-source LLM gateway that provides a unified OpenAI-compatible interface to a broad, community-maintained provider catalog. It is widely adopted in the open-source ecosystem, available both as a Python SDK and as a proxy server with virtual keys, spend tracking, and an admin UI in the open-source build.

Strengths: the largest provider catalog in the category, an active contributor community, and a low barrier to entry for Python-first teams. The proxy ships with budget controls, fallbacks, and basic observability hooks.

Considerations: the Python architecture introduces a measurable performance ceiling. The Global Interpreter Lock limits single-process throughput, which raises P95 latency at high concurrency. Running the proxy at scale requires maintaining the server process plus PostgreSQL and Redis for state. MCP is handled at the request layer rather than as a dedicated gateway that hosts servers and filters tools per virtual key. Teams evaluating a migration path can review Bifrost as a LiteLLM alternative for a feature-by-feature comparison.

Best for: Python-heavy engineering teams that need maximum provider compatibility for prototyping, internal tools, and development environments where throughput demands remain moderate.

3. Kong AI Gateway

Kong AI Gateway is an extension of Kong Gateway, the widely deployed open-source API gateway built on NGINX and OpenResty. The AI Proxy plugin and related AI plugins add LLM-specific routing on top of Kong's existing API management foundation, which appeals to teams already running Kong across their broader API estate.

Strengths: multi-LLM routing through the AI Proxy plugin with support for OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, and Mistral; mature plugin ecosystem covering OIDC, mTLS, rate limiting, and OpenTelemetry; and a managed SaaS option via Kong Konnect alongside the self-hosted Enterprise binary.

Considerations: the open-source version of Kong Gateway is limited. Advanced AI features including semantic caching, detailed analytics, and compliance tooling require Kong Enterprise, which is not free. AI Gateway plugins are an extension of an API management platform rather than an AI-native architecture, and Kong does not publish sustained-throughput overhead figures for the AI proxy path.

Best for: Enterprises already running Kong for non-AI API management that want to extend an existing gateway to handle LLM traffic without introducing a separate AI infrastructure layer.

4. Apache APISIX

Apache APISIX is a cloud-native API gateway from the Apache Software Foundation that has added a set of AI plugins to support LLM traffic management. As an Apache top-level project, it benefits from strong open-source governance and an active contributor base.

Strengths: the open-source AI plugin set includes the ai-proxy plugin for multi-LLM access, ai-rag for retrieval-augmented generation, token-based rate limiting, prompt decoration, and an mcp-bridge plugin that converts stdio-based MCP servers into HTTP SSE services. The same gateway can handle both traditional API traffic and AI workloads, reducing the number of proxy layers in a typical stack.

Considerations: AI-specific capabilities are delivered through plugins rather than as a native gateway architecture. The AI feature set is narrower than purpose-built options, with limited MCP gateway depth and limited AI-specific governance in the open-source build. Configuration complexity can be significant for teams unfamiliar with the APISIX ecosystem.

Best for: Teams already running APISIX for API management that want to extend their existing gateway to handle AI traffic without standing up a separate AI proxy.

5. Envoy AI Gateway

Envoy AI Gateway is an open-source project that extends Envoy Gateway to handle GenAI traffic, originating from the service mesh and Kubernetes Gateway API community. It targets Kubernetes-native AI traffic management with native integration into Envoy and Istio.

Strengths: recent releases add support for the Kubernetes Inference Gateway API, including an Endpoint Picker for inference routing to self-hosted models. Check the project's current release notes before relying on a specific version's capabilities, since the project is moving quickly. For teams already running Envoy or Istio, the gateway slots into an existing service mesh without introducing a separate proxy.

Considerations: Envoy AI Gateway is early stage, with provider support more limited than the mature options in this list, and no virtual key hierarchy or budget management in the current release. The Envoy xDS configuration model carries a steep learning curve for teams not already operating within the Envoy ecosystem.

Best for: Teams deeply invested in Kubernetes and the Envoy or Istio service mesh that want Kubernetes-native AI traffic management integrated with their existing infrastructure.

Open-Source LLM Gateways Compared at a Glance

The table compares license, caching, MCP support, measured overhead, and best-fit team across the five gateways. Overhead is listed as "Not published" where the project does not publish sustained-throughput benchmarks, which is itself a useful signal when the gateway will sit on your critical path.

Deploying a Self-Hosted LLM Gateway on Kubernetes

Most production self-hosted deployments land on Kubernetes, where the gateway's operational shape matters more than its feature list. Three properties decide how much work it is: whether the gateway needs external state to start, whether instances can be added without a coordination step, and whether a rolling upgrade drops requests.

Bifrost runs from a single container image and starts with zero configuration, so a first deployment is a Deployment and a Service with no database to provision:

# local, for evaluation
npx -y @maximhq/bifrost

# container, the same image used in cluster
docker run -p 8080:8080 maximhq/bifrost

Persistence is added when it is needed rather than required up front. A config store holds providers, virtual keys, and governance rules so configuration survives a pod restart; a log store backed by SQLite, Postgres, or ClickHouse holds request history.

For high availability, clustering uses peer-to-peer gossip so every instance is equal. Scaling out is adding replicas rather than promoting a leader, and rolling deployments do not drop in-flight requests. The Kubernetes deployment guide covers manifests and Helm.

The contrast that matters when comparing options: a gateway that requires PostgreSQL and Redis before it will serve a single request turns a one-service deployment into a three-service deployment, each of which now needs its own availability, backup, and upgrade plan.

Air-Gapped and In-VPC Installation

Air-gapped means no outbound network access at all, which quietly breaks assumptions most gateways make: pulling a pricing sheet, checking for updates, calling a hosted control plane, or resolving a license server. Test a candidate with egress actually blocked rather than reading its documentation.

Three requirements come up in every air-gapped deployment:

• Image and dependency mirroring. The container image and any Helm chart dependencies must be pullable from an internal registry. A gateway that fetches components at runtime cannot be mirrored this way.
• Pricing and catalog data. Cost calculation needs model pricing. Bifrost's Model Catalog downloads a pricing sheet and syncs it every 24 hours when a config store is present, so a disconnected deployment needs that data staged and refreshed deliberately rather than assumed.
• Model endpoints inside the boundary. With no external provider reachable, routing targets self-hosted inference: vLLM, SGLang, and Ollama are the usual endpoints, and a gateway that only speaks to hosted APIs is not usable here at all.

For in-VPC deployments, where egress is restricted rather than absent, the gateway runs inside private networking with no public ingress and reaches providers over controlled routes. Bifrost's security hardening guidance covers network and credential handling for both patterns, and the enterprise deployment overview covers AWS, GCP, Azure, and on-premise targets.

Sizing and Operating a Self-Hosted Gateway

Gateway overhead is a function of instance size as much as of software, which is why a single latency number quoted without hardware is not usable for capacity planning. Bifrost's published benchmarks make the relationship explicit:

Doubling vCPU cut overhead by 81% and queue wait by 96%, because queueing rather than processing is what dominates when the instance is undersized. The practical implication is that an underprovisioned gateway looks like a slow gateway, and the fix is capacity rather than a different product.

Four operational concerns tend to be underestimated when a gateway moves from pilot to production:

• Credential handling. Provider keys sit in the gateway, so they belong in HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, or Azure Key Vault rather than in a config file, with secrets detection catching credentials that appear in prompts.
• Log volume. Request and response payloads grow quickly. Log exports offload payloads to S3 or GCS while the database keeps searchable metadata, which keeps query performance stable as history accumulates.
• Observability wiring. The gateway is on the critical path, so its metrics belong in the existing stack: Prometheus for scraping and OpenTelemetry for traces into Grafana, Datadog, New Relic, or Honeycomb.
• Upgrade path. Check whether a version bump requires a maintenance window. Peer-to-peer clustering allows rolling upgrades; a leader-based design usually does not.

The sizing and redundancy guidance covers capacity planning in more depth.

How to Choose the Right Open-Source LLM Gateway

Match the gateway to the deployment profile and the production maturity of the workload. In practice the decision collapses to two questions: are you already operating a proxy that could absorb this traffic, and does the workload need agent and tool governance rather than just model routing. The five profiles below follow from those answers.

Teams still narrowing the field can work through how to choose an open-source LLM gateway or the enterprise self-hosted roundup. The governance resource page covers the access control model behind the first row.

For teams comparing several options on capability rather than deployment, the five open-source LLM gateways compared head to head covers licensing and feature depth, and the Bifrost AI gateway buyer's guide provides a structured capability matrix.

Frequently Asked Questions

What is a self-hosted open-source LLM gateway?

It is an LLM routing layer you run inside your own infrastructure, with source code you can inspect and modify. It sits between applications and providers, unifying access, failover, caching, and governance behind one API, while keeping every request inside your network. Bifrost is one such gateway, under Apache 2.0.

Why choose an open-source gateway over a managed service?

Four reasons drive it: data residency for regulated industries, audit trails for SOC 2 and HIPAA, fixed-cost economics instead of per-request fees at scale, and the ability to inspect and modify the routing layer. Managed services trade all four for zero operations, which suits smaller or less-regulated teams.

Which open-source LLM gateways support MCP?

MCP readiness is now a key differentiator. Bifrost has native MCP support, including tool governance and a Code Mode execution path. Kong and Apache APISIX add MCP capabilities through plugins, Envoy AI Gateway's support is emerging, and coverage across the category is uneven, so verify it against your agent stack.

How much hardware does a self-hosted LLM gateway need?

Less than most teams provision, but the sizing has to be deliberate. Bifrost's published benchmarks sustain 5,000 requests per second at 100% success on a 4 vCPU, 16 GB instance, adding 11 microseconds per request. On 2 vCPU the same load costs 59 microseconds, because queueing rather than processing dominates when the instance is undersized. Size for peak concurrency, not average throughput.

Which open-source LLM gateway is best for production at scale?

For production workloads with strict governance and multi-provider routing, Bifrost is the strongest fit: lowest overhead, native MCP, hierarchical virtual-key governance, and semantic caching in an Apache 2.0 core. The other four fit narrower profiles around existing infrastructure or provider breadth.

Get Started with Bifrost for Self-Hosted Deployments

The open-source LLM gateway category in 2026 has matured to the point where running an AI gateway inside an enterprise's own perimeter is no longer a research exercise. For teams that need production latency, compliance-grade governance, native MCP support, and Apache 2.0 transparency in a single self-hosted package, the open-source Bifrost AI gateway is the default recommendation.

Deploy Bifrost locally with npx -y @maximhq/bifrost, or run the official Docker image inside any Kubernetes cluster. The source code is Apache 2.0 licensed, and the documentation covers gateway setup, MCP configuration, virtual keys, and observability. To see Bifrost on your traffic with multi-provider routing, MCP controls, and semantic caching configured for your environment, book a Bifrost demo.