LLM Budget Management: Virtual Keys and Hierarchical Spend Controls

TL;DR

• Virtual keys are gateway-issued credentials that carry their own budget, rate limits, and model access, so an AI budget can be attached to each team, project, or user instead of a shared provider key.
• Bifrost enforces budgets hierarchically at the customer, team, virtual key, and provider-config levels, and a request must pass every applicable budget before it reaches a provider.
• Budgets reset on rolling windows from one minute to one year, or align to calendar boundaries in UTC so gateway spend maps onto the billing month or fiscal quarter.
• Request limits and token limits run in parallel, and a rate-limit breach on one provider excludes only that provider from routing.
• Built-in AI cost tracking attributes every request to its virtual key, model, and provider, with Prometheus and OpenTelemetry export for existing dashboards.

LLM budget management is the practice of allocating, enforcing, and tracking AI spend across the teams, projects, and users that consume large language models, so no single consumer can exceed its allotment. Bifrost, the open-source AI gateway built in Go by Maxim AI, implements this at the request layer: it evaluates budgets before a request reaches a provider and blocks anything that would breach a limit. This post explains why LLM spend is difficult to control, how virtual keys become the unit of budget allocation, and how hierarchical budgets, reset durations, and rate limits give platform teams precise financial governance over production AI workloads.

Why AI Spend Is Hard to Control

AI spend on LLM APIs is hard to control because cost is generated per token, distributed across many teams, and billed after the fact by each provider. Enterprise generative AI spend reached $8.4 billion by mid-2025, more than doubling in six months even as per-token prices fell, according to Menlo Ventures' State of Generative AI in the Enterprise. A separate a16z survey of enterprise CIOs found that leaders expect roughly 75% growth in LLM spend over the following year.

Three structural problems make this spend difficult to govern:

• No native per-consumer limits. Provider API keys are shared credentials. A raw OpenAI or Anthropic key has one account-level bill and no built-in way to cap what an individual team, project, or user consumes.
• Attribution happens after billing. Costs are computed from token counts in the response, so a runaway agent or a misconfigured retry loop is visible only after the tokens are spent.
• Spend crosses provider boundaries. Teams route across multiple providers and models, and each provider prices differently, so a single organizational budget cannot be enforced from any one provider dashboard.

An AI gateway addresses this by sitting in front of every provider and enforcing spend policy at the point of the request, before tokens are consumed. This is the layer where LLM budget management becomes enforceable rather than observational, and it is the same layer where the routing and caching levers described in the guide to cutting LLM API and token costs apply. Teams that first need visibility before enforcement can start with the beginner's guide to tracking token usage.

Virtual Keys: The Unit of Budget Allocation

A virtual key is a credential issued by the gateway that carries its own access permissions, budget, and rate limits, decoupled from the underlying provider API keys. In Bifrost, virtual keys are the primary governance entity. Instead of distributing raw provider keys, a platform team issues a virtual key per team, project, application, or user, and attaches a budget to each one.

Each virtual key carries:

• Independent budgets. A dollar limit checked on every request, evaluated alongside any team or customer budget the key is attached to.
• Token- and request-based rate limits. Throttling on both tokens per period and requests per period.
• Access control. Model and provider filtering, so a key can be restricted to specific models or providers.
• Key restrictions. A virtual key can be limited to specific provider API keys, so its traffic only ever uses the credentials you intend.
• Instant activation status. A key can be enabled or disabled immediately to cut off access without rotating provider credentials.

Virtual keys can also carry a temporary budget override that adds spending capacity for a defined window without changing the base limit, current usage, or reset schedule, which covers a launch week or a backfill job without a permanent increase.

Because the virtual key is the object budgets attach to, it becomes the natural unit of allocation. Callers authenticate with a virtual key (via the x-bf-vk, Authorization, x-api-key, or x-goog-api-key header), and every request inherits that key's spend policy. Bifrost's governance model is built around this pattern, which lets a platform team map financial accountability directly onto the credentials each consumer holds. The complete guide to managing virtual keys and budgets in Bifrost walks through creating keys and attaching the first budget, and the explainer on how platform teams govern LLM access with virtual keys covers the access-control side.

Hierarchical and Cumulative Budgets

Hierarchical budgets let an organization set independent dollar limits at the customer, team, and virtual key levels, where a request must pass every applicable budget before it is allowed. Bifrost enforces a hierarchical budget structure that mirrors how organizations actually allocate money: a customer contains teams, a team contains virtual keys, and a virtual key contains per-provider configurations, with each level holding its own budget.

The hierarchy flows top to bottom:

• Customer has an independent budget.
• Team (one-to-many under a customer) has an independent budget.
• Virtual key (one-to-many under a team, or attached directly to a customer, or standalone) has an independent budget plus rate limits.
• Provider config (one-to-many under a virtual key) has an independent budget plus rate limits.

How cumulative budget checking works

When a request arrives, Bifrost checks all applicable budgets independently and the request proceeds only if every one has sufficient remaining balance. The check runs from the most specific level outward: provider config budget, then virtual key budget, then team budget, then customer budget. If any single budget is exhausted, the request is blocked. When the request succeeds, its cost is deducted from every applicable budget at the same time.

This cumulative model means a team can distribute a shared pool across many virtual keys without any one key overspending the team total. A virtual key might have $10 remaining and its parent team $20 remaining; a request that would fit the key's budget is still blocked if it would breach the team's. A virtual key can belong to either one team or one customer, not both, which keeps each key's cost attribution unambiguous. For multi-tenant platforms that need a budget per end customer as well as per internal team, the budget and rate-limit architecture for multi-tenant LLM platforms shows how the two attachments combine.

Reset Durations and Calendar Alignment

Reset durations define the time window over which a budget or rate limit accumulates before it returns to zero. Bifrost supports reset durations of 1m, 5m, 1h, 1d, 1w, 1M, 1Q, and 1Y, so budgets can be scoped to a minute, an hour, a day, a week, a month, a quarter, or a year depending on how the team plans spend. The 1Q quarter duration applies to budgets only; rate limits carry no quarter definition.

By default a budget uses a rolling window: usage resets once the reset duration has elapsed since the last reset. Bifrost also supports calendar-aligned budgets, where the budget resets at the start of each calendar period in UTC instead of on a rolling clock:

• 1d resets at midnight UTC each day.
• 1w resets Monday at 00:00 UTC each week.
• 1M resets on the first day of each month.
• 1Q resets on the first day of each fiscal quarter, with quarter_start_month available to move the fiscal year.
• 1Y resets on January 1 each year.

Calendar alignment applies only to day, week, month, quarter, and year periods; sub-day durations such as 1h or 30m have no calendar boundary and keep their rolling schedule. Turning alignment on for an existing owner keeps its accumulated usage: each open window adopts the calendar grid and resets at the next boundary rather than immediately. This matters for finance and FinOps teams whose reporting runs on calendar boundaries: a calendar-aligned monthly budget resets on the same instant as the billing cycle, so gateway spend maps cleanly onto the month being reconciled rather than to an arbitrary rolling window.

Token- and Request-Based Rate Limits

Rate limits cap how much traffic a virtual key can generate within a time window, using two thresholds that run in parallel: a request limit and a token limit. Budgets bound the dollars a consumer can spend; rate limits bound the shape of the traffic that spends them, which is why both are needed to stop a runaway agent before the budget is gone. This distinction matters for LLM budget management because token consumption, not request count, drives cost. A single request with a large context window can spend more than a hundred small requests.

Bifrost supports both limit types together:

• Request limits cap the number of API calls within a period, for example 100 requests per minute.
• Token limits cap the total prompt-plus-completion tokens within a period, for example 50,000 tokens per hour.

For a request to pass, it must satisfy both the request limit and the token limit at every applicable level. Rate limits can be configured at the virtual key level and the provider config level, each with its own reset duration. Provider-level rate limiting lets a team throttle OpenAI and Anthropic differently within the same virtual key, and a rate-limit breach on one provider excludes only that provider from routing while the others stay available. Teams and customers do not carry rate limits; those thresholds live at the virtual key and provider config levels, while budgets span the full hierarchy. The guide to LLM API rate limiting with virtual keys and budgets covers how to size the two limits against provider quotas.

AI Cost Tracking and Observability of LLM Spend

AI cost tracking is the ability to see what each team, key, and model actually costs in real time, which is the feedback loop that makes budgets meaningful. Bifrost includes built-in observability that captures every request and response with its input and output tokens, computed cost, latency, provider, and model, all without changes to application code. The logging runs asynchronously, so it adds no latency to the request path.

For teams that centralize telemetry, Bifrost exports metrics through native Prometheus integration and OpenTelemetry (OTLP) for distributed tracing, compatible with Grafana, New Relic, and Honeycomb. Because cost is calculated per request from provider pricing and token usage, spend can be attributed by virtual key, and therefore by team or project, without a separate cost-allocation pipeline. This closes the loop: budgets set the ceiling, rate limits shape the flow, and observability shows where the spend is going while there is still time to adjust.

For alerting on spend before a budget is exhausted, the roundup of AI gateways for LLM budget tracking and spend alerts compares the notification options, and the guide to managing LLM spend across providers covers cross-provider attribution.

Teams looking to reduce spend at the source can also pair budget controls with the MCP gateway's Code Mode, which lowers token costs at scale by having models orchestrate tools through generated code instead of expanding every tool schema inline.

Enterprise Governance at Scale

For large organizations, LLM budget management is one part of a broader access and compliance posture. Bifrost is built for enterprises running mission-critical AI workloads, and its governance layer extends past budgets into role-based access control, SSO and OIDC through providers such as Okta and Microsoft Entra, and immutable audit logs for SOC 2, GDPR, HIPAA, and ISO 27001.

Access profiles let a platform team define reusable budget, rate-limit, and access policies that auto-allocate virtual keys as new teams and users onboard, so spend governance holds its shape as headcount grows. User provisioning through OIDC and directory sync keeps the set of key holders current as people join and leave.

Regulated teams can deploy Bifrost inside their own environment through in-VPC and on-prem options, keeping budget enforcement, spend data, and provider credentials within their security boundary. The same governance resource hub covers how virtual keys, budgets, and rate limits combine into a single control plane for AI spend.

Frequently Asked Questions About LLM Budget Management

What is a virtual key in an AI gateway?

A virtual key is a credential the gateway issues in place of a raw provider API key. It carries its own budget, request and token rate limits, and model and provider permissions, and it can be disabled instantly. In Bifrost, virtual keys are the primary governance entity, so every request inherits the spend policy of the key that authenticated it.

How do hierarchical budgets work?

Hierarchical budgets attach independent dollar limits to customers, teams, virtual keys, and provider configs. When a request arrives, Bifrost checks every applicable budget from the most specific level outward, and the request proceeds only if all of them have sufficient balance. On success, the cost is deducted from each level at the same time.

What happens when an AI budget is exceeded?

The request is blocked before it reaches the provider, so no tokens are consumed and no cost is incurred. The block applies at whichever level ran out first: a virtual key with remaining balance is still blocked if its parent team or customer budget is exhausted. A temporary budget override can add capacity without changing the base limit.

Can budgets align to a calendar month or quarter?

Yes. Setting calendar_aligned on a customer, team, virtual key, or provider config resets its budgets at the start of each calendar period in UTC: midnight for 1d, Monday for 1w, the first of the month for 1M, the first day of the fiscal quarter for 1Q, and January 1 for 1Y. Sub-day windows keep their rolling schedule.

What is the difference between a budget and a rate limit?

A budget caps spend in dollars over a reset period. A rate limit caps request count and token count over a period, regardless of cost. Budgets exist at every level of the hierarchy; rate limits exist only at the virtual key and provider config levels. A request has to pass both, and a rate-limited provider is excluded from routing while others stay available.

How do I set an AI budget per team?

Create a team in Bifrost, attach a budget with a reset duration such as 1M, then issue one or more virtual keys under that team. Each key can carry its own smaller budget and rate limits. Cumulative checking then keeps the team's total spend inside the team budget no matter how many keys share it.

Getting Started with LLM Budget Management in Bifrost

LLM budget management in Bifrost starts with issuing a virtual key per team or project, attaching a budget and a reset duration, and layering token- and request-based rate limits on top. Hierarchical budgets then let you nest team and customer limits above each key, so cumulative checks keep every consumer inside both its own allotment and the organization's total. With built-in observability tracking cost per key in real time, platform teams get financial governance over production AI without building it themselves.

To see how virtual keys, hierarchical budgets, and spend controls work on your own workloads, book a demo with the Bifrost team.