Open Source AI Gateway: Route, Govern, and Secure AI Traffic

TL;DR

• An open source AI gateway is a self-hostable layer that routes, governs, and secures traffic between AI applications, LLM providers, and MCP tools through one API.
• The best AI gateway for enterprises covers all three jobs in one runtime: provider failover, per-consumer budgets and access control, and guardrails on prompts, responses, and tool calls.
• Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second and connects to 25+ providers and 10,000+ models through one OpenAI-compatible API.
• Bifrost applies the same CEL-based guardrail rules to LLM prompts, LLM responses, and MCP tool arguments and results.
• AI Gateway + Bifrost Edge extends gateway policy to desktop chat apps, browser AI, and coding agents that were never configured to use a gateway.

Most teams adopt an open source AI gateway to fix one problem, usually provider outages or runaway spend, and then discover that routing, governance, and security are the same control problem viewed from three angles. Bifrost, the open-source AI gateway written in Go by Maxim AI, is the best choice for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability, because it enforces all three jobs in a single request path. This guide defines what an open source AI gateway must do, compares the main types of gateway on those jobs, and shows how Bifrost routes, governs, and secures LLM, MCP, and endpoint AI traffic.

What Is an Open Source AI Gateway?

An open source AI gateway is a self-hostable proxy layer that sits between AI applications and model providers, exposing one API while it routes requests, enforces access and budget policy, and inspects content. Because the code is open, teams can audit it, run it inside their own network, and extend it without waiting on a vendor.

Figure 1: Routing, governance, and security are enforced once, at the gateway, instead of inside every application.

As Figure 1 shows, the gateway becomes the one place every caller passes through. That position is what makes it useful. A deeper look at the concept is in our guide to the AI gateway as the control plane for enterprise LLM traffic. In practice, an AI gateway (also called an LLM gateway or LLM proxy) performs three jobs:

• Route: send each request to the right provider, model, and API key, and fail over when a provider returns errors.
• Govern: decide who may call which models, at what budget and rate, and record what they did.
• Secure: inspect prompts, responses, and tool calls for sensitive data, prompt injection, and unsafe content.

Bifrost exposes this through a drop-in replacement for existing SDKs: applications change the base URL and keep their code. Behind that one endpoint, Bifrost reaches 25+ providers and 10,000+ models, including self-hosted runtimes such as Ollama and vLLM.

How to Choose the Best AI Gateway: Evaluation Criteria

Choosing the best AI gateway comes down to whether one runtime can route, govern, and secure traffic without adding measurable latency or forcing application changes. Evaluate candidates against the three jobs, then against performance and deployment, because a gateway that fails any one of them pushes that work back into application code.

The table below turns those jobs into concrete questions. It complements the broader comparison of open source LLM gateways, which covers the wider field, and the LLM gateway buyer's guide for procurement teams.

The security criteria map directly to industry risk frameworks. The OWASP Top 10 for LLM Applications ranks prompt injection first and sensitive information disclosure second, and its "unbounded consumption" entry covers the runaway cost that gateway budgets exist to stop.

Open Source AI Gateway Options Compared

Open source AI gateway options fall into four broad types: lightweight LLM proxy libraries, general API gateways with AI plugins, cloud-scoped gateways, and purpose-built AI gateways. They differ less on basic routing, which most handle, and more on governance depth, guardrail coverage, and MCP support.

The pattern is consistent: generic tools reach AI governance by assembling plugins, while a purpose-built gateway ships routing, governance, and guardrails as one runtime. Teams moving off a proxy library can review how Bifrost compares as a LiteLLM alternative before migrating.

Bifrost: The Open Source AI Gateway for Enterprise AI Traffic

The open source Bifrost AI gateway unifies 25+ providers behind one OpenAI-compatible API and enforces routing, governance, and security policy in the same request path. It adds 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks, with a 100% success rate.

Best for: Bifrost is built for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with robust security, policy enforcement, and governance capabilities.

The performance figure matters because an agent can make dozens of model and tool calls per task. Bifrost publishes its benchmark results at 5,000 RPS, and teams can reproduce them on their own hardware before committing. Key capabilities by job:

• Route: automatic retries and fallbacks, weighted key load balancing, CEL routing rules, and semantic caching.
• Govern: virtual keys, hierarchical budgets and rate limits, SSO, RBAC, and data access control.
• Secure: guardrails on prompts, responses, and MCP tool calls, secrets detection, redaction, and signed audit logs.
• Observe: built-in request logging with tokens, cost, and latency, plus Prometheus and OpenTelemetry export.

The sections below cover each job in turn.

Routing: LLM Router, Failover, and Load Balancing

An AI gateway routes traffic by choosing a provider, model, and API key for each request, retrying transient failures, and failing over to another provider when retries are exhausted. Bifrost acts as an LLM router with three layers: runtime routing rules, virtual key restrictions, and weighted key selection.

Figure 2: Retries absorb transient errors inside one provider; fallbacks move the request to the next provider only after retries are exhausted.

Figure 2 shows the order of evaluation. Routing rules use CEL (Common Expression Language) expressions and run before governance provider selection, so a rule can send requests to a different provider based on headers, parameters, capacity, or budget state. Virtual key routing then restricts each consumer to approved providers and models with static weights.

Failure handling is where many gateways stop at a single retry. Bifrost classifies each failure before acting on it:

• Permanent key failures (401, 402, 403) mark the key dead for that request and rotate immediately, with no backoff.
• Rate limits (429) rotate to another key and still apply backoff, because providers often enforce account-level quotas.
• Transient server failures (5xx, DNS, connection errors) retry the same key with exponential backoff and jitter.

When retries are exhausted, automatic fallbacks move the request to the next provider in the chain, which gets its own retry budget. Weighted load balancing spreads traffic across keys, and semantic caching replays responses for identical or semantically similar requests without calling the provider. For patterns beyond these defaults, see five LLM routing strategies for AI gateways.

Governance: Virtual Keys, Budgets, and Access Control

AI gateway governance means controlling who can call which models, how much they can spend, and who can change that policy. In Bifrost, virtual keys are the primary governance entity: each key carries its own model and provider permissions, budget, and rate limits, and can attach to a team or a customer.

Virtual keys authenticate through the headers existing SDKs already send, including OpenAI-style Authorization, Anthropic-style x-api-key, and the native x-bf-vk header. That means governance applies without application changes. Budgets are hierarchical, and every level is checked on each request:

This structure comes from Bifrost budget and rate limit controls, and it maps to how organizations actually allocate AI spend. The Bifrost governance overview walks through the full model.

Governance also covers the people who administer the gateway. Bifrost Enterprise adds role-based access control with Admin, Developer, and Viewer system roles plus custom roles. Data access control scopes each role to own-data, team-data, or all-data, so a developer on one team cannot see another team's virtual keys, prompts, or routing rules. User provisioning through OIDC assigns those roles from identity-provider groups.

Security: LLM Guardrails, Secrets Detection, and Audit Trails

Securing AI traffic at the gateway means validating content before it reaches a model and before a response reaches the caller. Bifrost guardrails evaluate LLM prompts, LLM responses, and MCP tool arguments and results against CEL rules, then block, redact, or modify content according to policy.

Figure 3: The same CEL rules and guardrail profiles cover LLM prompts, completions, and MCP tool arguments and results.

Bifrost guardrails are built on two concepts. Rules define when and what to check; profiles define how, using either Bifrost-managed checks or external providers. A single rule can chain several profiles for layered protection:

• Bifrost-managed profiles: Prompt Guardrails (LLM-as-judge policies), Custom Regex with a built-in PII template, and Gitleaks-backed secrets detection.
• External providers (11): Microsoft Presidio, Azure AI Language PII, AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, CrowdStrike AIDR, Gray Swan, Patronus AI, Check Point's AI Agent Security, Repello Argus, and Singulr AI.

Detection alone is often not enough for regulated data. Guardrail redaction supports three modes: runtime (redact the live request or response), logs only (leave traffic raw but redact stored logs and trace exports), and runtime with reversible placeholders. More implementation detail is in our article on LLM guardrails at the gateway layer.

Accountability depends on two separate records. Built-in request logging captures every request asynchronously with inputs, outputs, tokens, cost, and latency. Audit logs record administrative activity (who changed which key, rule, or role) as events that can be HMAC-signed, exported as JSON, JSON Lines, or Syslog and archive to S3 or GCS.

These controls close a measurable gap. IBM's 2025 Cost of a Data Breach Report found that 97% of organizations with an AI-related security incident lacked proper AI access controls.

Governing MCP Traffic with an MCP Gateway

An MCP gateway governs which Model Context Protocol tools an AI agent can discover and call, and under whose credentials. Bifrost acts as both an MCP client and an MCP server, so it can connect to upstream tool servers and expose a filtered, authenticated tool set to clients such as Claude Desktop and Cursor.

Tool access is the security surface that grows fastest with agents. Bifrost as an MCP gateway for enterprise tool access applies three controls:

• Deny-by-default tool filtering: a virtual key with no MCP configuration gets no tools, and per-key MCP tool filtering builds a strict allow-list that request headers can narrow but never widen.
• Six authentication types: MCP authentication supports None, Headers, OAuth 2.0, Per-User OAuth, Per-User Headers, and Token Exchange (enterprise), so each user can reach tools under their own identity.
• Guardrails on tool calls: the same guardrail rules from the previous section validate tool arguments and results.

Tool catalogs also cost tokens. Code Mode exposes four generic tools and lets the model write code that orchestrates the rest in a sandbox, reducing input tokens by up to 92.8% across multiple MCP servers. The MCP gateway cost and access control breakdown covers the measurements, and securing coding agent access with an AI gateway applies these controls to Claude Code.

Extending Governance to Shadow AI with Bifrost Edge

Shadow AI is AI usage that never passes through sanctioned infrastructure: desktop chat apps, browser AI, and coding agents that employees configure themselves. A gateway only governs traffic that is pointed at it, so AI Gateway + Bifrost Edge extends the same gateway policy to the endpoint without per-app configuration.

Figure 4: The gateway stays the single policy engine; Edge brings traffic that was never configured to use it under the same controls.

The risk is quantified. IBM's 2025 breach research reports that a high level of shadow AI added USD 670,000 to the average breach cost, and 63% of the breached organizations studied had no AI governance policies to prevent it.

The Bifrost AI gateway remains the control plane: virtual keys, budgets, rate limits, guardrails, and audit logs are configured there once. Bifrost Edge runs on each company machine and routes AI traffic from supported apps through that gateway, so every request inherits the same policies.

Administrators can also allow or deny MCP servers across the fleet and roll Edge out through MDM tools such as Jamf, Intune, and Kandji. Bifrost Edge is currently in alpha. The architecture is covered in closing the last mile of AI governance at the endpoint.

Self-Hosted AI Gateway: Open Source vs Enterprise Deployment

A self-hosted AI gateway keeps prompts, responses, and keys inside infrastructure the organization controls. Open source Bifrost runs as a single instance that handles roughly 3,000 to 5,000 RPS, while Bifrost Enterprise adds clustering, identity integration, and the compliance controls regulated teams require.

The dividing line is state. Bifrost keeps provider configuration, budgets, and usage in memory for speed, so running several open source nodes against a shared Postgres database is not supported. Bifrost Enterprise clustering synchronizes that state across nodes with gossip-based discovery and zero-downtime rolling updates.

Open source teams can deploy with Docker or Kubernetes. Enterprise teams can run in-VPC deployments on AWS, Google Cloud, Azure, Cloudflare, or Vercel, and review the Bifrost Enterprise options for on-prem and regulated environments.

For more on deployment trade-offs, see the open source AI gateway guide for self-hosted deployment and our ranked review of open source gateways.

Frequently Asked Questions

What is an AI gateway?

An AI gateway is a proxy layer between AI applications and model providers that exposes one API while routing requests, enforcing access and budget policy, and inspecting content. It replaces per-provider SDK integrations with a single endpoint, so failover, cost control, and security policy are configured once instead of in every application. Open source AI gateways can be self-hosted and audited.

What is the difference between an AI gateway and an API gateway?

An API gateway manages generic HTTP traffic: authentication, rate limiting, and routing by path. An AI gateway understands model traffic, so it can count tokens, track cost per request, fail over between LLM providers, cache semantically similar prompts, and apply guardrails to prompts and responses. API gateways can reach some of this through plugins, but not as one runtime.

Is Bifrost an open source AI gateway?

Yes. Bifrost is an open source AI gateway written in Go, with its source published on GitHub. The open source edition includes routing, fallbacks, load balancing, virtual keys, budgets, semantic caching, the MCP gateway, and built-in observability. Bifrost Enterprise adds guardrails, clustering, SSO, RBAC, audit logs, and in-VPC deployment on top of the same core.

How does an AI gateway secure LLM traffic?

An AI gateway secures LLM traffic by checking content in both directions and controlling who can send it. Input guardrails detect PII, secrets, and prompt injection before a request reaches the model; output guardrails check responses before they reach the caller. Virtual keys, RBAC, and audit logs then control and record access, which addresses the top two risks in the OWASP LLM list.

Can an AI gateway stop shadow AI?

A gateway alone governs only traffic that is configured to use it, so shadow AI in desktop apps and browsers usually bypasses it. AI Gateway + Bifrost Edge addresses this by routing AI traffic on each company machine through the Bifrost gateway, where existing budgets, guardrails, and audit logs apply. Bifrost Edge is currently in alpha.

Does an AI gateway add latency?

A well-built AI gateway adds latency measured in microseconds, which is negligible next to model inference times measured in hundreds of milliseconds or seconds. Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second in sustained gateway benchmarks. Semantic caching can lower end-to-end latency by serving repeated requests without a provider call.

Getting Started with Bifrost

Choosing an open source AI gateway is a decision about where routing, governance, and security policy live. Bifrost keeps all three in one low-overhead runtime, extends them to MCP tools and, with Bifrost Edge, to employee machines, and runs inside the network boundary regulated teams require. Frameworks such as the NIST AI Risk Management Framework ask for exactly this kind of centralized, auditable control.

To see how Bifrost routes, governs, and secures AI traffic across your providers, agents, and teams, book a demo with the Bifrost team, or browse the Bifrost resources library for architecture guides.