Top 9 LLM Routers
An LLM router decides which model, provider, and API key should serve each request. This guide compares 9 LLM router options, including Bifrost, OpenRouter, LiteLLM, Not Diamond, RouteLLM, and Azure model router, by routing method, failover, governance, and deployment.
TL;DR
- An LLM router answers three questions per request: which model should handle it, which provider and key should serve that model, and what happens if the call fails.
- Gateway routers such as Bifrost answer all three, combining rule-based model selection, complexity tiering, health-aware provider selection, retries, and fallbacks with budgets and access control.
- Learned routers such as Not Diamond, Martian, and RouteLLM predict the best model per prompt but leave keys, failover, and governance to other infrastructure.
- Cloud-native routers from Azure and AWS route within their own model catalogs, which limits them to single-cloud estates.
- Many production stacks run a learned model choice behind a gateway router, so model selection and reliability are handled by separate layers.
An LLM router sends each request to the model and provider best suited to it, which lets teams cut cost on simple prompts, reserve frontier models for hard ones, and keep traffic flowing when a provider fails. Bifrost, the open-source LLM router and AI gateway built by Maxim AI, is the best choice for enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability, because it routes across 10,000+ models and 25+ providers with failover and governance in one layer. This guide compares 9 LLM router options by how they route, what they do when a call fails, and where they run.
What Is an LLM Router?
An LLM router is a component that selects the model, provider, and credential for each LLM request according to rules, health signals, or a learned prediction of response quality and cost. It sits between applications and model providers, either as part of an AI gateway or as a standalone service. The explainer on how model routing works covers the concept in depth.

Research has shown why model choice matters. The FrugalGPT paper reported matching the best individual model's performance with up to 98% cost reduction by learning which models to call for which queries. The RouteLLM paper from LMSYS trained routers on human preference data to decide between a strong and a weak model. In production, model choice is only one of the three decisions in Figure 1, and the other two usually decide whether an application stays up.
Types of LLM Routing
LLM routing methods fall into four broad types, and most production routers combine several. Choosing a router starts with knowing which type of decision you need automated.
| Routing type | How it decides | Example use | Typical tools |
|---|---|---|---|
| Rule-based | Expressions on headers, team, model, budget | Send premium tenants to a frontier model | Gateway routers |
| Health-based | Live error rates, latency, rate-limit state | Move traffic off a degraded provider | Gateway routers, marketplaces |
| Complexity or learned | Classifier or trained model scores the prompt | Cheap model for simple prompts | Learned routers, complexity routers |
| Semantic | Embedding similarity to reference prompts or intents | Route by task category | Semantic routers |
The guide to smart LLM routing per request works through each type with examples, and the overview of semantic routing platforms goes deeper on embedding-based routing.
How to Evaluate an LLM Router
Evaluate an LLM router on routing signals, failure handling, provider coverage, governance, latency overhead, and deployment. A router that picks the ideal model but cannot fail over when that model's provider returns errors will cause more incidents than it saves in cost. Provider incidents are frequent enough that Bifrost tracks them on a provider status page.
| Criterion | What to check | Why it matters |
|---|---|---|
| Routing signals | Rules, request metadata, complexity, learned scores | Determines what the router can optimize |
| Failure handling | Retries, key rotation, provider fallbacks, circuit breaking | Provider errors are routine at scale |
| Provider coverage | Hosted APIs, cloud platforms, self-hosted models | Limits which models you can route to |
| Governance | Per-key budgets, rate limits, access by team | Routing decisions have cost consequences |
| Overhead | Added latency per request, classifier cost | Routing runs on every call |
| Deployment | Self-hosted, in-VPC, SaaS | Prompts may contain regulated data |
LLM Routers Compared at a Glance
The nine router options below fall into three groups: gateway routers that handle model choice and reliability together, learned routers focused on per-prompt model selection, and routers built into cloud AI platforms.
| LLM router | Type | Model selection | Failover | Governance | Deployment |
|---|---|---|---|---|---|
| Bifrost | Gateway router | CEL rules, complexity tiers | Retries, key rotation, fallback chains, adaptive routing | Virtual keys, budgets, rate limits | Self-hosted, in-VPC, air-gapped |
| OpenRouter | Hosted marketplace | Model chosen by caller, model fallback lists | Provider fallbacks, health-weighted routing | Account-level | Managed |
| LiteLLM | Python proxy and SDK | Caller or config | Retries, cooldowns, fallbacks | Virtual keys, budgets | Self-hosted |
| Not Diamond | Learned router | Pre-trained or custom-trained router | Not described | Not described | API and SDKs |
| Martian | Learned router | Router selects per prompt | Not described | Not described | Managed API |
| RouteLLM | Open-source learned router | Strong vs weak model by threshold | Not described | No | Self-hosted |
| vLLM Semantic Router | Open-source policy router | Signal-driven decisions | Not described | Policy-based | Self-hosted |
| Azure model router | Cloud-native | Trained router across a model subset | Fallback within the subset | Azure controls | Azure |
| Amazon Bedrock intelligent prompt routing | Cloud-native | Two models in one family | Fallback model | AWS controls | AWS |
To see which models each provider serves before writing routing rules, browse the Bifrost model library.
1. Bifrost: Gateway LLM Router with Failover and Governance
The Bifrost AI gateway has a routing layer that combines rule-based model selection, complexity tiering, health-aware provider and key selection, retries, and fallback chains. Because the LLM router lives in the gateway, every routing decision is also subject to virtual key budgets, rate limits, and access policy.
Best for: Bifrost is built for enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with strong security, policy enforcement, and governance capabilities.

The routing stages in Figure 2 map to documented features:
- Routing rules: CEL-based routing rules evaluate request headers, parameters, team, and capacity metrics such as budget used, scoped from virtual key to team, customer, and global, where the first match wins.
- Complexity tiers: the complexity router assigns Simple, Medium, or Complex tiers using a decision model or embedding similarity to reference phrases, and runs only when a rule references the tier; the Decisions API across eight providers supplies the decision models.
- Provider and key selection: weighted load balancing splits traffic across keys, and Enterprise adaptive routing picks providers and keys from live health, latency, and load.
- Failure handling: retries and fallbacks classify each failure, rotate keys on rate limits or rejected credentials, back off on transient server errors, and move to the next provider with its own retry budget.
- Session affinity: requests that share a session ID stay on the same provider and key, which keeps provider prompt caches warm for agents and coding tools.
Bifrost exposes one OpenAI-compatible API to 10,000+ models across 25+ providers and adds about 11 microseconds of overhead per request at 5,000 RPS.
The benchmark of the upgraded complexity router and the closer look at adaptive routing show these stages in production detail.
The Bifrost LLM gateway page summarizes the full routing feature set.
Gateway-Based LLM Routers
Gateway-based LLM routers put routing in the request path alongside credentials, retries, and logging. They suit teams that want one integration point for many providers rather than a separate router service in front of each application.
2. OpenRouter
OpenRouter is a hosted API that routes requests across many model providers through one key. For each model, its default routing weighs price, throughput, time to first token, rate limits, reliability, and tool-call quality, and it removes providers with poor error rates until they recover.
- Controls:
sortby price, throughput, or latency;order,only, andignoreto restrict providers; amodelsarray for model fallbacks; and data policies that limit routing to providers that do not store data or to zero-data-retention endpoints. - Fit: teams that want broad model access with consolidated billing. OpenRouter is a managed service, so prompts pass through its infrastructure.
3. LiteLLM
LiteLLM is a Python SDK and proxy whose Router supports several strategies: simple-shuffle (the recommended default), rate-limit-aware, latency-based, least-busy, lowest-cost, and custom routing. It retries rate-limit errors with exponential backoff, cools down failing deployments, and supports context-window and content-policy fallbacks, with Redis to share state across instances.
- Fit: Python-centric teams that want a self-hosted proxy with virtual keys.
- Consideration: teams comparing throughput and memory under load can review the Bifrost and LiteLLM router comparison and the LiteLLM alternatives page.
Learned Model Routers
Learned model routers predict which model will give the best response for a given prompt, usually trading off quality against cost. They address the first routing question well and generally rely on other infrastructure for provider failover, credentials, and budgets. The guide on cutting LLM token costs with model routing shows how teams measure the savings.
4. Not Diamond
Not Diamond is an AI model router and prompt optimization tool. Teams start with a pre-trained router or train a custom router on their own evaluation data, and Not Diamond selects which LLM to use for each query. Prompt optimization refines prompts automatically when teams switch models.
- Integration: Python, TypeScript, and REST APIs.
- Fit: teams with labeled evaluation data that want model selection tuned to their own tasks.
5. Martian
Martian offers a model router behind an OpenAI-compatible API. Its documentation shows the router invoked by setting the model field to router, optionally constrained to a list of candidate models, so the router chooses among them per request.
- Fit: teams that want managed per-prompt routing without training their own router.
6. RouteLLM
RouteLLM is an Apache 2.0 framework from LMSYS for serving and evaluating LLM routers. Its routers (matrix factorization, weighted Elo ranking, a BERT classifier, and an LLM-based classifier) send simple queries to a weaker, cheaper model and hard ones to a stronger model, with a threshold that sets the cost-quality trade-off.
- Claims: the project reports routers that reduce costs by up to 85% while keeping about 95% of GPT-4 performance on MT Bench.
- Fit: research and teams willing to calibrate and operate their own router; it routes between two models rather than across a full catalog.
7. vLLM Semantic Router
vLLM Semantic Router is an open-source project, licensed under Apache 2.0, that describes itself as a decision layer for models and compute. It gives agent frameworks one API across many models and selects or combines models per call according to a configured policy, including constraints on approved locations and priorities for quality, latency, and cost.
- Fit: teams running their own inference fleets that want policy-driven model selection close to the serving layer.
Cloud-Native Model Routers
Azure and AWS each offer a model router inside their AI platforms. These routers are convenient for workloads already committed to one cloud and limited to the models that cloud serves. Teams on AWS often keep Bifrost in front of Amazon Bedrock so they can fail over to other providers.
8. Azure Model Router
Azure model router is a trained language model deployed like any other Foundry model. It analyzes each prompt and routes it to an underlying model in one of three modes: balanced (the default), quality, or cost. Teams can restrict routing to a model subset, which also bounds fallback.
- Underlying models: OpenAI GPT models, Anthropic Claude (deployed separately first), xAI Grok, DeepSeek, Meta Llama, and others listed by Microsoft.
- Limits: the context window is set by the smallest underlying model, and routing uses text input only.
9. Amazon Bedrock Intelligent Prompt Routing
Amazon Bedrock intelligent prompt routing provides a serverless endpoint that routes each request between models in one family based on predicted response quality for the cost. Default routers are preconfigured, and configured routers let teams choose the two models, a fallback model, and a response quality difference threshold.
- Limits: AWS states routing is optimized for English prompts, requires exactly two models from the same family, and cannot adjust to application-specific performance data.
Choosing an LLM Router
Choose an LLM router by deciding which of the three routing questions you need automated and who owns the result. If platform teams need budgets, failover, and self-hosting, a gateway router is the foundation; learned routers can then refine model choice for specific workloads.

Three patterns cover most enterprise needs:
- Gateway router only: rule-based and complexity routing in Bifrost, with fallbacks across providers. This covers cost tiering and reliability without a second service. The guide to enabling automatic fallback when a provider fails walks through the setup.
- Learned router behind a gateway: a learned router picks the model, and the gateway handles provider choice, failover, keys, and budgets.
- Cloud-native router inside one cloud: Azure or Bedrock routing for workloads that never leave that cloud, ideally still behind a gateway for cross-cloud fallback.
The comparison of failover routing gateways adds detail on the reliability side, and the post on LLM routing decisions covers how to log and audit them. For a full gateway evaluation, see the LLM gateway buyer's guide.
Frequently Asked Questions
What is an LLM router?
An LLM router selects which model, provider, and API key should serve each request, using rules, live health signals, or a learned prediction of quality and cost. Routers reduce spend by sending simple prompts to cheaper models and improve reliability by moving traffic away from failing providers. Some routers are standalone services; others are part of an AI gateway such as Bifrost.
What is the difference between an LLM router and an AI gateway?
An LLM router decides where a request goes. An AI gateway is the broader control layer that includes routing plus authentication, budgets, rate limits, caching, guardrails, and observability. Bifrost is an AI gateway with a built-in LLM router, so routing decisions are governed by the same keys and policies as the rest of the traffic. The overview of LLM routers and model routing compares the terms in more depth.
How much can an LLM router save?
Savings depend on how many requests a cheaper model can handle without losing quality. Research results are large: RouteLLM reports up to 85% cost reduction at about 95% of GPT-4 quality on MT Bench, and FrugalGPT reports up to 98% cost reduction in its experiments. Production savings are usually smaller and should be measured on your own traffic; the LLM cost calculator helps estimate the price gap between model tiers.
Is OpenRouter an LLM router?
OpenRouter routes requests across providers for the model you choose, weighting price, throughput, latency, and reliability, and it supports fallbacks across models you list. It is a hosted marketplace rather than a self-hosted router, so it suits teams comfortable sending prompts through a third party. Enterprises needing in-network routing often evaluate self-hosted options as OpenRouter alternatives.
Can an LLM router route by prompt complexity?
Yes. Learned routers such as Not Diamond, Martian, and RouteLLM predict which model fits a prompt. Bifrost includes a complexity router that tags each request Simple, Medium, or Complex using a decision model or embedding similarity, and exposes the tier to routing rules so complex prompts go to frontier models and simple ones to cheaper models.
Does an LLM router add latency?
Every router adds some latency, and learned or complexity routing adds a classification step. Bifrost's gateway overhead is about 11 microseconds per request at 5,000 RPS, while a decision-model classification typically takes several hundred milliseconds, which is why Bifrost runs the complexity classifier only when a routing rule needs it.
Getting Started with an LLM Router
The best LLM router for an enterprise answers all three routing questions (model, provider, and failure handling) and applies budgets and access policy to every decision. Bifrost combines rule-based and complexity routing with adaptive provider selection, retries, and fallbacks in one self-hosted gateway. To see Bifrost route your traffic, book a demo with the Bifrost team.