What Is an LLM Router? How Model Routing Works

TL;DR

• An LLM router is a layer between applications and language models that selects which model and provider handles each request, based on rules, request content, cost, or live performance.
• The RouteLLM study found that sending easy queries to a cheaper model cut costs by over 85% on MT Bench while keeping 95% of GPT-4's quality.
• Model routing strategies fall into five families: static rules, weighted distribution, semantic or complexity-based classification, performance-adaptive selection, and failover.
• A router answers "which model?"; an AI gateway also enforces access control, budgets, rate limits, caching, and audit trails around that decision.
• Bifrost combines CEL routing rules, an embedding-based Complexity Router, weighted virtual keys, and automatic fallbacks, while adding 11 microseconds of overhead per request at 5,000 RPS.

An LLM router is the component that selects, for each incoming request, which large language model and which provider should serve it. Teams running more than one model need this layer to balance cost, quality, and availability without hardcoding that logic into every application. Bifrost, the open-source AI gateway with built-in model routing built by Maxim AI, performs this routing at the gateway layer behind a single OpenAI-compatible API. This guide covers how model routing works and how a router differs from an AI gateway.

What Is an LLM Router?

A router is a decision layer that inspects each request and forwards it to the most suitable model or provider. The routing decision can depend on static configuration, request metadata such as team or user tier, the semantic content of the prompt, remaining budget, or the measured health of each provider.

Without an LLM router, the model choice is fixed in application code, and changing it means a deployment for every service that calls a model. A router moves that decision into one place.

The router sits on the request path. Its job has two parts:

• Selection: pick the target provider and model for the request.
• Recovery: decide what happens when the selected target fails, rate-limits, or times out.

In the Bifrost AI gateway, both parts run in a single pre-request phase, detailed in the provider routing reference. For a survey of the tools that implement this pattern, see the comparison of the top LLM router solutions.

How LLM Routing Works

LLM routing works as a short pipeline: the router reads the request, evaluates it against a set of conditions, resolves a provider and model, and attaches a fallback chain. The request then goes to the chosen target, and the fallback chain takes over if that target returns an error.

A typical request moves through these stages:

1. Extract context: the requested model, headers, calling team or key, and optionally the prompt.
2. Evaluate conditions: rules, classifiers, or scores pick a match.
3. Resolve the target: set a provider/model pair and an API key.
4. Attach fallbacks: queue alternates in priority order.
5. Execute and recover: retry or move to the next fallback on failure.

Bifrost runs this pipeline in a defined order. Routing rules evaluate first, then static governance routing on virtual keys, then the Model Catalog fills in any provider still unset. Explicit rules always win over defaults, which keeps routing behavior predictable during incident review.

Types of Model Routing

Model routing strategies differ in the signal used to choose a target. Most production deployments combine two or three, such as a complexity classifier, traffic weights, and a fallback chain.

Rules and weights decide who may use what, and in what proportion. Semantic and learned routing decide which model is good enough for a prompt. Adaptive and failover routing decide which provider is healthy right now. A detailed breakdown of when each applies is in the guide to LLM routing strategies every AI gateway needs.

Semantic Routing and Complexity-Based Model Selection

Semantic routing classifies a request by meaning rather than by keywords. The router embeds the prompt, compares the embedding against labeled reference examples, and assigns the request the label of its nearest match. That label then drives model selection, so simple requests go to fast, inexpensive models and demanding ones go to frontier models.

Bifrost implements this as the Complexity Router. It embeds the latest user message and assigns one of three tiers, SIMPLE, MEDIUM, or COMPLEX, based on the nearest reference phrase. Bifrost ships 150 default reference phrases, 50 per tier, spread across coding, math, writing, extraction, translation, and agentic tasks.

Several design choices in the open-source Bifrost gateway address the usual weaknesses of semantic routing:

• Pay only when used: classification runs only when a rule references complexity_tier.
• Abstain instead of guessing: a similarity floor lets weak matches fall through to normal routing rather than being blocked.
• Optional LLM fallback classifier: a small chat model names the tier when similarity is inconclusive, capped by a timeout.
• Session-aware routing: within an agent conversation the tier only moves upward, avoiding model switches that reduce provider prompt-cache reuse.
• Auditable decisions: the matched phrase and similarity score are recorded in routing decision logs.

The same embedding approach powers semantic caching, which serves a stored response for a similar prompt without a provider call. More on this category is in the overview of semantic routing platforms for LLM applications.

LLM Router vs AI Gateway

A router selects which model serves a request. An AI gateway is the broader control point that performs that routing and also authenticates callers, enforces budgets and rate limits, caches responses, records observability data, and applies security policy. Routing is one function of a gateway, not a substitute for one.

A router embedded in one application optimizes that application; a gateway applies the same routing and governance to every team. Bifrost, the AI gateway, scopes every routing decision by virtual keys, which are deny-by-default: a key with no provider configuration can call nothing. The Bifrost governance overview covers how access control and routing combine.

LLM Cost Optimization Through Routing

Routing is one of the most direct levers for LLM cost optimization because model prices differ by more than an order of magnitude. Sending everything to the strongest model maximizes cost; sending everything to the cheapest model loses quality on hard prompts.

The RouteLLM paper from UC Berkeley and collaborators, published at ICLR 2025, notes that smaller models can be more than 50 times cheaper per output token than frontier models. In the LMSYS write-up, the authors report cost reductions of over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K compared with using only GPT-4, while retaining 95% of GPT-4's performance.

Bifrost supports three cost controls on the routing path:

• Tiered model selection through complexity rules, sending SIMPLE requests to small models.
• Budget-aware routing through the budget_used variable, which lets a rule redirect traffic once a provider or model budget passes a threshold such as 80%.
• Hierarchical budgets and rate limits at the virtual key, team, and customer level, which cap spend regardless of routing.

A worked example of these savings is in the guide to cutting LLM token costs with model routing.

How Bifrost Handles AI Model Routing

The Bifrost platform handles AI model routing through four layers that run in a fixed order: CEL routing rules, weighted governance routing on virtual keys, adaptive load balancing, and the Model Catalog as the final resolver. Automatic fallbacks then protect every request. Bifrost connects to 25+ providers and 10,000+ models through one OpenAI-compatible API.

Routing rules. Rules are written in the Common Expression Language (CEL) and can reference the requested model, headers, query parameters, virtual key, team, customer, capacity metrics, and complexity_tier. Rules are scoped at the virtual key, team, customer, or global level, and the first match wins, evaluated from most specific to least specific. Rules can also chain and support probabilistic A/B splits.

Weighted routing. A virtual key can list several providers with weights, for example 80% Azure and 20% OpenAI for the same model. Weights are normalized automatically across the providers that support the requested model.

Adaptive load balancing. In Bifrost Enterprise, adaptive load balancing recomputes route weights every 5 seconds from error rate and token-aware latency, removes failing routes with circuit breakers, and returns recovered routes to full traffic within seconds.

Fallbacks and retries. Automatic fallbacks retry 5xx errors with backoff, rotate API keys on 429 and auth failures, and switch providers once retries are exhausted. Weighted key management spreads load across multiple API keys for the same provider.

Bifrost benchmarks show 11 microseconds of overhead per request at 5,000 RPS with a 100% success rate. For a closer look at how routing and fallback interact, see LLM gateway routing, fallback, and governance in Bifrost.

Configuring a Complexity-Based Routing Rule in Bifrost

The lowest-risk way to introduce complexity routing is one rule that sends only COMPLEX requests to a frontier model and leaves other traffic on the existing path, limiting the impact of misclassification while reference phrases are tuned.

A global rule for that carve-out looks like this:

{
  "id": "complexity-complex",
  "name": "Complex → Frontier model",
  "enabled": true,
  "cel_expression": "complexity_tier == \"COMPLEX\"",
  "targets": [{ "provider": "anthropic", "model": "claude-opus-4-5", "weight": 1 }],
  "scope": "global",
  "priority": 0
}

Once classifications look correct, add SIMPLE and MEDIUM rules for small and balanced models, or scope the rule to one team as a pilot first.

CEL conditions combine freely. A rule such as complexity_tier == "COMPLEX" && team_name == "research" limits frontier-model access to one team, and budget_used > 80 redirects traffic to a cheaper model before a budget is exhausted. Because Bifrost is a drop-in replacement for OpenAI-compatible SDKs, none of these rules require changes to application code.

The gateway setup guide covers installation, and each routing decision appears in Bifrost's built-in observability.

Choosing the Best LLM Router for Enterprise Workloads

The best LLM router for an enterprise routes on the right signals and also enforces governance and deployment requirements. Across many teams, routing alone is not enough; the router must sit in shared infrastructure with access control, budgets, and audit data.

An open-source router adds two advantages: the routing logic can be inspected, and self-hosting keeps prompts inside the organization's network. Bifrost as an open-source AI gateway covers both, and Bifrost Enterprise adds in-VPC deployments.

The LLM gateway buyer's guide provides a fuller evaluation framework, and the ranked list of routing tools compares specific options side by side.

Frequently Asked Questions

What are LLM routers?

LLM routers direct each request to the most suitable large language model or provider. They choose targets using rules, request metadata, prompt content, cost thresholds, or live provider health. They let teams mix inexpensive and frontier models and fail over during outages without editing application code.

What is the best LLM router?

The answer depends on the workload. For enterprise production traffic, the strongest choice combines content-aware routing, automatic failover, per-team governance, low overhead, and self-hosted deployment. Bifrost provides all five as an open-source AI gateway, with CEL-based routing rules, a semantic Complexity Router, and 11 microseconds of overhead per request at 5,000 RPS.

What is the difference between an LLM router and an AI gateway?

A router selects which model serves a request. An AI gateway performs that routing and also handles authentication, budgets, rate limits, caching, observability, and security policy for all applications. A gateway is shared infrastructure in front of every app, which enables organization-wide control.

What is LLM-based routing and how does it work?

LLM-based routing uses a language model, or a model trained on preference data, to judge which target should answer a prompt. The classifier predicts difficulty or domain, and the router maps that prediction to a model. Bifrost classifies by embedding similarity first and can fall back to a small chat model.

Does an LLM router add latency?

Every router adds some latency, and the amount depends on its design. Rule-based and weighted routing add almost none; semantic routing adds an embedding call; LLM-based classification adds a full completion. Bifrost itself adds 11 microseconds per request at 5,000 RPS.

Can an LLM router handle provider outages?

Yes, if it supports failover. A resilient router retries transient errors, rotates API keys when one is rate-limited, and moves to a backup provider when the primary keeps failing. Bifrost retries with exponential backoff, then walks a configured fallback chain in which each provider gets its own retry budget.

Getting Started with Bifrost

An LLM router is most effective when it runs inside a gateway that also governs access, cost, and reliability. Bifrost provides content-aware model routing, load balancing, and automatic failover in one open-source AI gateway. To see how Bifrost can route your organization's AI traffic, book a demo with the Bifrost team.