LLM Observability with Prometheus: Metrics and Dashboards

TL;DR

• LLM observability captures latency, token usage, cost, and error rate as metrics so teams can measure, alert on, and debug model behavior in production.
• Bifrost exposes native Prometheus metrics at the gateway, including bifrost_upstream_latency_seconds, bifrost_input_tokens_total, bifrost_output_tokens_total, and bifrost_cost_total, with no changes to application code.
• Prometheus scrapes Bifrost's /metrics endpoint for a single instance; multi-node deployments push to a Prometheus Push Gateway for accurate aggregation.
• Grafana reads Bifrost's Prometheus metrics directly, so latency percentiles, token throughput, spend, and per-provider error rates become live dashboards.
• Because every metric is labeled by provider, model, and virtual key, the same data that powers LLM observability also drives cost governance and routing.

Production LLM applications generate request latency, token consumption, error rates, and per-model costs on every call, yet most teams keep no time-series record of any of it. LLM observability is the practice of capturing those signals as metrics, traces, and logs so engineers can measure performance, control cost, and debug failures in production. Bifrost, the open-source AI gateway built in Go by Maxim AI, emits native Prometheus metrics for every request that passes through it, which makes Prometheus and Grafana a direct path to LLM observability without adding instrumentation to application code. This guide covers the metrics that matter, how to get them into Prometheus by scraping or pushing, and how to turn them into Grafana dashboards and alerts.

What Is LLM Observability?

LLM observability is the discipline of instrumenting large language model traffic so that latency, token usage, cost, error rate, and output quality are all measurable in production. It extends traditional observability, metrics, logs, and traces, to the specific signals an LLM produces: prompt and completion tokens, time to first token, per-provider failures, and dollar cost per request. Without it, a model that grows slower or more expensive over a release looks identical to one that has not changed.

The three pillars map cleanly onto LLM systems. Metrics are the numeric time series covered in this guide, such as request rate and P95 latency. Logs are the full request and response records, which Bifrost captures automatically for every call. Traces connect a single user action across multiple model and tool calls, which is where the OpenTelemetry approach to LLM observability is strongest. Metrics answer how the system is behaving in aggregate; traces answer why one request was slow.

The LLM Metrics That Matter

The metrics worth tracking for LLM observability fall into four groups: latency, throughput, cost, and reliability. Latency includes total request duration and, for streaming, time to first token and inter-token latency. Throughput is request rate and tokens per second. Cost is spend per model and per team. Reliability is error rate, retry count, and provider health.

Prometheus represents each of these with one of its core metric types: counters for monotonic totals like tokens and cost, histograms for latency distributions, and gauges for point-in-time values like in-flight requests. The table below maps the metrics that matter to the Bifrost metric that carries each one.

Token and cost counters are the two most teams miss, and they are the ones that decide the monthly bill. Tracking them alongside latency is the core of monitoring latency and cost in LLM operations, and pairing them with audit records rounds out LLM monitoring metrics, logs, and controls.

Why the AI Gateway Is the Right Place to Emit Metrics

An AI gateway is the single point every LLM request already passes through, which makes it the natural place to emit observability metrics. Instrumenting application code means adding a metrics library to every service, keeping token accounting in sync with each provider's response format, and repeating that work in every language the stack uses. Measuring at the gateway collapses all of it into one layer.

The Bifrost AI gateway is a drop-in replacement for provider SDKs: changing the base URL routes existing OpenAI, Anthropic, or other SDK calls through the gateway, as the drop-in replacement setup shows. From that moment, every request across more than a thousand models and 20-plus providers is measured the same way, with the same metric names and labels, regardless of which provider served it.

Bifrost adds only 11 microseconds of overhead per request at 5,000 requests per second, a figure it documents in its published benchmarks, and the telemetry plugin runs asynchronously so metric collection adds no measurable latency of its own. Gateway-level measurement also normalizes token and cost accounting across providers that report usage differently, so a single query returns comparable numbers for every model in production.

Exposing Bifrost Metrics to Prometheus

Bifrost exposes Prometheus metrics through two methods: a pull-based /metrics endpoint that Prometheus scrapes, and a push-based path to a Prometheus Push Gateway for clustered deployments. The telemetry plugin that produces these metrics is enabled by default, so the /metrics endpoint is available with no extra configuration.

For a single instance, point Prometheus at the endpoint with a standard scrape job:

scrape_configs:
  - job_name: 'bifrost'
    static_configs:
      - targets: ['bifrost-host:8080']
    scrape_interval: 15s

When several instances run behind a load balancer, scraping can hit a different node each time and miss metrics from the others. For those deployments, the Prometheus metrics integration pushes to a Prometheus Push Gateway instead, which aggregates across every node.

Multi-node metric collection pairs with Bifrost clustering for high availability, and both run inside private infrastructure for teams that deploy Bifrost in their own VPC. The table below summarizes when to use each method.

If the gateway has authentication enabled, the /metrics endpoint requires the same basic-auth credentials in the scrape config, otherwise Prometheus receives 401 responses and scraping fails silently.

Building Grafana Dashboards for LLM Observability

Grafana turns Bifrost's Prometheus metrics into LLM observability dashboards by querying Prometheus as a data source and rendering the results as panels. Because the metrics are already labeled by provider, model, and virtual key, a single dashboard can break latency, spend, and error rate down by any of those dimensions without extra instrumentation.

A production LLM dashboard usually opens with four panels: P95 request latency, token throughput, spend per model, and error rate. Grafana's Prometheus data source supports the PromQL that produces each. The histogram_quantile function turns the latency histogram into percentile lines; rate over the token and cost counters gives throughput and burn rate; and dividing error requests by total requests yields an error ratio. Adding a panel for bifrost_cache_hits_total shows how much traffic semantic caching is serving, which is often the fastest way to see cost savings. For teams standardizing on a shared stack, the Bifrost observability overview documents how logs and metrics fit together behind the same UI.

PromQL Queries and Alerts for LLM Monitoring

PromQL is the query language that converts raw Bifrost metrics into the latency, cost, and error signals LLM monitoring depends on. The most common queries are a latency percentile, a spend rate, and an error ratio. Each is a single expression that a Grafana panel or a Prometheus alert rule can evaluate on a schedule.

The following query returns P95 provider latency broken down by model over a five-minute window:

histogram_quantile(0.95,
  sum by (le, model) (rate(bifrost_upstream_latency_seconds_bucket[5m]))
)

Reliability alerts build on the same metrics. Dividing bifrost_error_requests_total by bifrost_upstream_requests_total gives a rolling error ratio that can trigger a page when a provider degrades, which is the signal that should fire before users notice. Bifrost also exposes bifrost_key_rotation_events_total, so a spike in key rotations flags a provider that is rate-limiting a specific API key, and pairs directly with automatic fallbacks and load balancing across keys. Alerting on cost is equally direct: a rate query over bifrost_cost_total catches a runaway spend before the invoice does. These patterns are the backbone of a full LLM monitoring practice.

Prometheus vs OpenTelemetry for LLM Telemetry

Prometheus and OpenTelemetry solve different halves of LLM telemetry: Prometheus is a metrics store and query engine, while OpenTelemetry is a vendor-neutral standard for emitting metrics, traces, and logs to any backend. They are complementary rather than competing. Prometheus answers aggregate questions like P99 latency by model; OpenTelemetry traces connect a single request across the gateway, the model, and any downstream tools.

Bifrost supports both. The Prometheus path gives immediate aggregate metrics, and the OpenTelemetry integration exports traces using the GenAI semantic conventions to collectors like Grafana Cloud, New Relic, or Honeycomb, with a native Datadog connector for teams already on that stack.

Most production setups run Prometheus for dashboards and alerts, then add OTel traces for deep debugging, an approach detailed in tracing and metrics with OpenTelemetry and in broader guidance on AI observability platforms.

From Metrics to Governance: Closing the Loop

The same Prometheus metrics that describe LLM behavior also drive governance, because Bifrost labels every metric with the virtual key, team, and customer that made the request. A spend counter is an observability signal and a budget input at the same time, which turns a dashboard reading into an enforceable limit rather than a number someone notices too late.

That connection is what separates monitoring from control. Once spend and usage are visible per team, virtual keys enforce budgets and rate limits against exactly those numbers, so a team that crosses its monthly ceiling is capped automatically.

The governance model treats metrics as the feedback loop for access and cost policy, and the governance resource hub walks through the patterns most teams adopt first. Selecting an approach that ties observability to control is the theme of most current LLM monitoring tool evaluations.

Frequently Asked Questions

What is observability in LLMs?

Observability in LLMs is the ability to understand a model application's behavior in production from the signals it emits: latency, token usage, cost, error rate, and output quality. It combines metrics for aggregate trends, logs for individual request records, and traces for request-level debugging. The goal is to measure performance and cost, detect regressions, and diagnose failures without guessing.

What metrics should you track for LLM observability?

Track four groups: latency (total request time and time to first token), throughput (request rate and tokens per second), cost (spend per model and per team), and reliability (error rate, retries, and provider health). Bifrost exposes each as a labeled Prometheus metric, such as bifrost_upstream_latency_seconds for latency and bifrost_cost_total for spend, so all four are available from one endpoint.

Can Prometheus monitor LLM applications?

Yes. Prometheus monitors LLM applications by scraping metrics that describe request latency, token usage, cost, and errors, then storing them as time series for querying and alerting. The practical requirement is a source that emits those metrics in Prometheus format. Bifrost provides that source at the gateway, exposing a /metrics endpoint and Push Gateway support with no application-code changes.

What is the difference between LLM monitoring and observability?

LLM monitoring tracks known metrics against thresholds and alerts when something crosses them. LLM observability is broader: it instruments the system so engineers can investigate questions that were not defined in advance, correlating metrics, logs, and traces to explain unexpected behavior. Monitoring tells you a provider is slow; observability lets you find out which model, key, and request path caused it.

How do I build a Grafana dashboard for LLM metrics?

Add Prometheus as a Grafana data source, then build panels with PromQL over Bifrost's metrics. Use histogram_quantile on bifrost_upstream_latency_seconds_bucket for latency percentiles, rate over bifrost_cost_total for spend, and a ratio of error to total requests for reliability. Labels for provider, model, and virtual key let one dashboard break every panel down by those dimensions.

Which LLM observability approach is best for self-hosted deployments?

For self-hosted or regulated environments, emitting metrics at an open-source gateway keeps all observability data inside your own infrastructure. Bifrost runs in your VPC or on-prem, exposes native Prometheus metrics and OpenTelemetry export, and requires no third-party agent, so latency, cost, and reliability data never leaves your network. This suits teams with data-residency or compliance requirements that rule out hosted-only tools.

Get Started with LLM Observability on Bifrost

LLM observability starts with a metrics source, and Bifrost gives you one at the gateway: native Prometheus metrics, OpenTelemetry export, and full request logs for every model in production, with no application-code changes and 11 microseconds of overhead. Explore the Bifrost resource library and the documentation overview to see how metrics, dashboards, and governance fit together, or book a demo with the Bifrost team to design LLM monitoring for your stack.