11 Best LLM Proxies: Open Source and Managed

TL;DR

• An LLM proxy is a service that sits between applications and model providers, forwards each request, and returns the response through one endpoint.
• Most teams that search for an LLM proxy need an AI gateway: a proxy that also routes between models, enforces budgets and access rules, and records every request.
• Bifrost is the top pick for enterprises: an open-source AI gateway in Go that adds about 11 microseconds of overhead per request at 5,000 RPS and runs inside your own network.
• Managed options such as Cloudflare AI Gateway, Vercel AI Gateway, Azure API Management, and Apigee suit teams that accept a hosted control plane or have standardized on one cloud.
• Data residency and the number of teams sharing the layer decide which category fits before any feature comparison does.

An LLM proxy is a service that sits between your applications and model providers, forwards every model request through one endpoint, and returns the response in a consistent format. Bifrost, the open-source AI gateway built in Go by Maxim AI, is the best choice for enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability, because it adds routing, failover, governance, and observability to that forwarding layer. This guide explains how an LLM proxy differs from an AI gateway and an LLM router, sets out the criteria that matter in production, and compares 11 open-source and managed options.

What Is an LLM Proxy?

An LLM proxy is a network service that accepts model requests from applications, forwards them to one or more LLM providers, and returns the responses through a single endpoint. Its core job is forwarding and format translation, so application code talks to one API instead of a separate SDK and credential per provider.

A minimal LLM proxy server does three things:

• Accepts one request format, usually the OpenAI Chat Completions shape, so existing SDKs work by changing the base URL.
• Translates and forwards the request to the provider that serves the requested model.
• Holds provider credentials centrally, so API keys stop living in application code and environment files.

Production teams soon need more: retries on 5xx or 429 errors, a second provider during outages, per-team spend limits, and a record of who called which model. Once those requirements arrive, the component is no longer a simple proxy. It is the control plane described in what an AI gateway is and how it governs LLM traffic, which is the scope the Bifrost documentation covers. In Bifrost, the switch itself is a base URL change in the OpenAI, Anthropic, or Google GenAI SDK.

LLM Proxy vs AI Gateway vs LLM Router

An LLM proxy forwards requests, an LLM router chooses which model or provider serves each request, and an AI gateway does both while also enforcing access policy and recording telemetry. The scope difference decides whether a tool can carry production traffic for several teams.

Figure 1: An LLM proxy covers the first stage and an LLM router the second; an AI gateway runs all four on every request.

As Figure 1 shows, the categories nest rather than compete:

The difference between an LLM gateway and a simple proxy matters most once more than one team shares the layer. The same distinction shows up for coding agents, where a lightweight router compared with a full AI gateway changes what an engineering manager can see and control. It also applies to tool traffic: the split between an MCP gateway, an MCP proxy, and an MCP server follows the same pattern.

Bifrost sits in the AI gateway row: it forwards, routes, governs, and records in one process.

Key Criteria for Evaluating an LLM Proxy

A production-ready proxy is judged on six criteria: added latency, API compatibility, failover behavior, caching, spend and access controls, and deployment model. The useful test is what happens to one request when a provider fails or a team exceeds its budget.

Figure 2: Cheap checks run first, and the fallback provider is only called when the primary returns an error.

Figure 2 is the request path these criteria describe. In Bifrost, retries use exponential backoff and fallbacks move to the next provider, each with its own retry budget. Weighted load balancing across API keys spreads traffic across keys, and semantic caching replays answers for identical or semantically similar requests so the provider is never called.

On overhead, Bifrost publishes benchmarks of about 11 microseconds per request at 5,000 RPS. Security teams should also map the proxy to the OWASP Top 10 for LLM applications, since prompt injection and sensitive data disclosure are both visible at this layer.

LLM Proxies Compared at a Glance

The 11 options fall into two groups: self-hosted gateways and proxies that run in your own infrastructure, and managed services operated by a cloud or platform vendor. "Not published" means the vendor page reviewed for this guide does not state it.

The LLM gateway buyer's guide turns these columns into vendor evaluation questions.

1. Bifrost: The Open-Source AI Gateway for Enterprise LLM Traffic

The open-source Bifrost gateway gives applications one OpenAI-compatible API to 10,000+ models across 25+ providers, and applies routing, failover, governance, and observability to every request. Teams looking for an LLM proxy get forwarding plus the controls shared production traffic needs.

Best for: Bifrost is built for enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with strong security, policy enforcement, and governance capabilities.

Figure 3: One OpenAI-compatible endpoint replaces a per-provider integration, and the same policies apply to model and tool calls.

Key capabilities:

• Provider coverage: one API to the supported providers, with responses returned in the OpenAI-compatible format.
• Routing: routing rules written as CEL expressions evaluate conditions at runtime; on Bifrost Enterprise, adaptive routing sends each request to the healthiest provider and key based on live traffic, without sending its own test calls.
• Governance: virtual keys control access, budgets, rate limits, and routing per consumer, and hierarchical budgets roll up through teams and customers.
• Observability: built-in request logging captures inputs, outputs, tokens, cost, and latency asynchronously, and OpenTelemetry export sends traces to existing collectors following the OpenTelemetry semantic conventions for generative AI.
• Guardrails: Bifrost Enterprise runs guardrails such as secrets detection and custom regex, and integrates external providers including AWS Bedrock Guardrails, Azure Content Safety, and Google Model Armor.
• Tool traffic: as an MCP gateway, Bifrost exposes connected tools to MCP clients such as Claude Desktop and Cursor under the same keys and logs, and its Code Mode cuts MCP token costs by up to 92% on large tool sets.

For high availability, Bifrost Enterprise clustering uses a peer-to-peer architecture with gossip-based state sync, so no single node is a point of failure. Bifrost's compliance coverage includes SOC 2 Type II, ISO 27001, HIPAA, and GDPR, and the enterprise deployment options keep prompts, responses, and telemetry inside the customer's infrastructure. See the Bifrost Enterprise page for details.

Open-Source and Self-Hosted LLM Proxies

Self-hosted LLM proxies run in infrastructure you control, which keeps prompts and responses inside your network and lets platform teams tune the layer to their own traffic. Most options below extend an existing API gateway or proxy rather than starting from AI traffic. For a broader comparison of this group, see how the leading open-source AI gateways compare.

2. Agent Router (formerly Envoy AI Gateway)

Agent Router is the renamed Envoy AI Gateway project: an Apache 2.0 control plane for AI and agent traffic, built on Envoy and Envoy Gateway for Kubernetes. It exposes one OpenAI-compatible API across hosted providers, self-hosted models, and MCP servers, and manages credentials, routing, quotas, global rate limiting, failover, and usage attribution. The project now sits under the Agentic AI Foundation; its CRDs, aigw CLI, and container images kept their names.

Best for: Kubernetes platform teams already standardized on Envoy who want AI routing as custom resources.

3. Kong AI Gateway

Kong AI Gateway governs LLM, MCP, and agent-to-agent traffic on Kong's API platform. Kong lists multi-LLM support, semantic caching and load balancing, PII sanitization with prompt guards, MCP governance, token quota management, and L7 observability on AI traffic, with sign-up through Kong Konnect.

Best for: organizations that already run Kong for REST APIs and want AI policies in the same control plane.

4. F5 AI Gateway with NGINX

F5 AI Gateway routes requests to an LLM backend and inspects both requests and responses. Its processors can modify, reject, or annotate traffic, for example detecting prompt injection or redacting credit card numbers. It runs on Kubernetes, handles authentication and authorization, stores auditable records in S3-compatible storage, exports telemetry through OpenTelemetry, and F5 recommends placing NGINX Ingress Controller in front of it.

Best for: security-led teams with an existing F5 or NGINX footprint.

5. Traefik Hub AI Gateway

Traefik Hub AI Gateway is a control layer on the Traefik Hub API Gateway for chat-completion traffic. It works with OpenAI-compatible providers including OpenAI, Anthropic, Gemini, Mistral, Ollama, and Amazon Bedrock, and adds Semantic Cache, Content Guard, token-based Rate Limit and Quota, and GenAI metrics and traces through OpenTelemetry. It is configured as Kubernetes Middleware resources, with MCP handled by a separate Traefik MCP Gateway.

Best for: Kubernetes teams already using Traefik as their ingress.

6. Docker Model Runner

Docker Model Runner runs models on a developer's machine and serves them through OpenAI- and Ollama-compatible APIs, loading each model into memory only when a request arrives. Docker documents no routing or governance features for it; tool traffic is a separate product, the open-source Docker MCP Gateway, which runs each MCP server in its own container behind one endpoint.

Best for: local development against open-weight models before traffic moves to a shared gateway.

Managed LLM Proxies and Cloud AI Gateways

Managed LLM proxies are operated by a platform or cloud vendor, so there is nothing to deploy, but prompts and responses pass through the vendor's control plane. They fit teams that accept that trade-off or already run most workloads on one cloud. The overlap with existing API management products is covered in how an AI gateway differs from an API gateway.

7. Cloudflare AI Gateway

Cloudflare AI Gateway adds observability and control in front of providers such as Workers AI, Anthropic, Google Gemini, and OpenAI. It lists caching, rate limiting, analytics, logging, request retries, model fallback, and routing with dynamic routes; guardrails and data loss prevention are marked beta, and its OpenAI-compatible unified API is marked deprecated in the documentation.

Best for: teams whose applications already run on Cloudflare.

8. Vercel AI Gateway

Vercel AI Gateway is a hosted API that applications on any cloud can call with an AI Gateway key. It accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages requests, routes across providers with ordered provider and model fallbacks, and records latency, tokens, and cost per request. Budgets apply to a team, project, API key, or member, and full-response caching is left to the application.

Best for: product teams building on Vercel and the AI SDK.

9. Azure API Management

Azure API Management adds AI gateway policies to its existing API gateway: llm-token-limit for token quotas, llm-emit-token-metric for usage metrics, llm-semantic-cache-store and llm-semantic-cache-lookup backed by Azure Managed Redis, backend load balancing with circuit breakers, and Azure AI Content Safety moderation. It supports OpenAI, Anthropic Messages, and Vertex AI API shapes, plus MCP servers and A2A agent APIs; feature availability varies by tier.

Best for: Microsoft-centric enterprises already publishing APIs through Azure API Management.

10. Apigee

Apigee, Google Cloud's API management platform, adds AI policies for LLM traffic: LLMTokenQuota enforces token quotas over time intervals, PromptTokenLimit limits token bursts, SanitizeUserPrompt and SanitizeModelResponse screen content, and SemanticCacheLookup with SemanticCachePopulate cache responses. These run on Apigee and on Apigee hybrid from version 1.16.1.

Best for: enterprises that already govern their APIs in Apigee on Google Cloud.

11. Amazon Bedrock

Amazon Bedrock is AWS's managed service for running inference on hundreds of foundation models, rather than a proxy in front of external providers. It adds Bedrock Guardrails for safety checks and Intelligent Prompt Routing, which AWS says can cut costs by up to 30% while maintaining quality. Its reach stops at the models Bedrock offers.

Best for: AWS-native teams whose approved models are all available in Bedrock.

How to Choose an LLM Proxy

Choose an LLM proxy by answering two questions before comparing features: must model traffic stay inside your network, and will several teams share the layer with their own budgets and keys? Those answers place you in one of four categories; compare features only within that category.

Figure 4: Data residency and the number of teams sharing the layer decide the category before any feature comparison.

Figure 4 reduces the decision to three questions:

1. Traffic must stay in your network and several teams share it: a self-hosted AI gateway. This is the case Bifrost is built for, with AI governance through virtual keys and budgets.
2. Traffic must stay in your network and one team owns it: a lightweight open-source proxy is enough until a second team arrives.
3. Hosted is acceptable and you have standardized on one cloud: that cloud's gateway, such as Azure API Management, Apigee, or Amazon Bedrock.
4. Hosted is acceptable and you are multi-cloud: a managed AI gateway such as Cloudflare AI Gateway or Vercel AI Gateway.

In any category, test failover before launch: automatic failover and load balancing for LLM apps only works once the fallback chain has been exercised. The AI gateway control plane overview is a good next read for teams moving from a proxy to a gateway, and the Bifrost governance resource shows how keys, budgets, and limits map to teams.

Frequently Asked Questions

What does an LLM proxy server do?

It accepts model requests from applications in one format, forwards them to the provider serving the requested model, and returns responses through a single endpoint. It centralizes provider credentials and translation. Production deployments usually add retries, fallbacks, caching, and per-team limits, at which point the component is an AI gateway such as the Bifrost AI gateway rather than a plain proxy.

What is an AI gateway?

An AI gateway is a control plane between applications and model providers that forwards requests, chooses the model or provider for each one, enforces access and spend policy, and records telemetry. It covers the job of an LLM proxy and an LLM router together; Bifrost also governs MCP tool calls and agent-to-agent traffic.

Is there an open-source LLM proxy?

Yes. Open-source options include Bifrost (Apache 2.0), Agent Router (formerly Envoy AI Gateway, Apache 2.0), and the open-source Docker MCP Gateway for tool traffic. Bifrost runs as a single binary through npx, Docker, Kubernetes, or the Go SDK, and its GitHub repository is the starting point for self-hosted deployments.

What is an LLM router?

An LLM router decides which model or provider should handle each request, based on rules, cost, latency, or request content. Routing alone does not enforce budgets or record who made each call. In Bifrost, routing runs inside the gateway through weighted provider routing, CEL-based routing rules, and, on Enterprise, adaptive routing driven by live traffic health.

What is the difference between an AI gateway and an API gateway?

An API gateway manages general HTTP APIs: authentication, rate limits by request count, and routing to services. An AI gateway adds model-specific behavior: token-based budgets, provider failover, response caching by semantic similarity, model-aware cost tracking, and content guardrails. Several API gateway vendors now add AI policies; Bifrost is built around model, tool, and agent traffic from the start.

How does an OpenAI proxy work?

An OpenAI proxy exposes endpoints in the OpenAI API format, so applications using the OpenAI SDK change only the base URL. The proxy translates each request for the target provider, whether OpenAI, Anthropic, AWS Bedrock, or a self-hosted model, and returns an OpenAI-shaped response. Bifrost works this way and also accepts Anthropic and Google GenAI SDK requests.

Run Your LLM Proxy Layer on Bifrost

Teams that start by searching for an LLM proxy usually need forwarding, routing, governance, and observability in one layer that runs inside their own infrastructure. Bifrost delivers that as an open-source AI gateway with about 11 microseconds of overhead, and the Bifrost resources hub covers deployment, governance, and migration in more depth. To see how Bifrost can replace a basic LLM proxy across your teams, book a demo with the Bifrost team.