LLM Observability at the Gateway: What to Measure, and Where

TL;DR

• LLM observability is the practice of capturing tokens, cost, latency, errors, fallbacks, cache behavior, and tool-call activity for every model request, then routing that data into the same metrics and tracing systems your platform team already runs.
• The AI gateway is the correct measurement point because it is the single choke point every LLM request crosses, which removes the need to instrument each application, SDK, or agent independently.
• Bifrost, the open-source AI gateway built by Maxim AI, exports the full metric set natively through a Prometheus /metrics endpoint, an OpenTelemetry (OTLP) plugin, and a Datadog connector, with 11 microseconds of overhead per request at 5,000 requests per second.
• The metric set worth instrumenting on day one is input and output tokens, cost, time to first token (TTFT), tail latency, error rate by provider, fallback and retry counts, cache hit ratio, and tool-call duration for MCP-invoked tools.
• Time to first token searches are up 88% year over year, reflecting how streaming latency and per-request cost have become the two dominant reliability signals for teams running LLM workloads in production.

Production LLM traffic fails in ways HTTP monitoring does not catch: a provider returns 200 OK while streaming its first token 4 seconds late, another silently swaps a cheaper distilled model into a route, and cost per conversation triples over a weekend nobody looked at Grafana. Bifrost, the open-source AI gateway built in Go by Maxim AI, gives platform teams one place to measure and export those signals across every provider and application.

This post walks through the signals worth instrumenting for LLM observability, why the gateway is the right layer to capture them, and how Bifrost's built-in observability, plus the Prometheus endpoint and OpenTelemetry plugin, fit into existing tooling.

What Is LLM Observability?

LLM observability is the practice of capturing tokens, cost, latency, errors, fallbacks, cache behavior, and tool-call activity for every model request, and routing that telemetry into the metrics, tracing, and logging systems the rest of the platform already uses. It differs from generic API monitoring in what it records: token counts and USD cost per request, streaming timings such as time to first token, per-provider error attribution across a fallback chain, cache hit ratio by lookup mode, and duration of tools invoked by the model through the Model Context Protocol (MCP).

The goal is the same as any observability program: detect regressions, attribute cost, debug failed requests, and answer capacity questions with data. LLM observability adds the fields that make those questions answerable for AI workloads specifically. A five-nines HTTP dashboard says nothing about whether a model quietly started truncating outputs, whether a provider is silently rate-limiting one virtual key, or whether a new MCP tool is timing out 3% of the time.

Why the AI Gateway Is the Right Measurement Point

The AI gateway is the right measurement point because every LLM request from every application in the organization crosses it exactly once. Instrumenting there produces a complete, provider-agnostic view with no per-application SDK changes, no per-language exporter to install, and no gap when a team ships a new service. Anything above the gateway (application SDKs) or below it (provider APIs) captures only a slice.

Three practical consequences follow from that placement. First, the gateway sees which model actually served the request after routing, fallback, and retries, so its metrics reflect what happened, not what the client asked for. Second, it can attribute every request to a virtual key, team, customer, or business unit, because governance labels are enforced there. Third, it can time the parts of the pipeline the client cannot see: queue wait, cache lookup, provider selection, and MCP tool execution. Bifrost adds 11 microseconds of overhead per request at 5,000 RPS in the published benchmarks, so measurement at the gateway is not a latency tax.

Gateway-Side vs SDK-Side vs Provider-Side Observability

Not every layer answers every question. The table below compares the three common measurement points on the questions platform teams actually ask.

SDK-side instrumentation still has a role for application-level spans (the user turn, the retrieval step). Provider dashboards remain useful for reconciling invoices. Gateway-side is the layer that answers infrastructure questions across every model and app in one place.

The Signals to Instrument at the Gateway

Nine metric families cover the operational surface of an LLM workload. Each is exported by Bifrost through Prometheus, OTLP, or the Datadog connector, and can be scraped, pushed, or streamed into whatever stack the platform team already runs. The table below maps each signal to why it matters, where it is measured, and the exact metric name Bifrost emits.

Every metric above carries a common label set including provider, model, virtual_key_id, team_id, and customer_id, so any of them can be sliced by tenant or team without extra work.

Tokens and Cost

Input and output tokens are counted per response as separate counters, so a runaway completion (large output on a small prompt) and a bloated prompt template (large input on a small output) each stand out. Cost is emitted as bifrost_cost_total in USD, computed against the gateway's pricing table after the response returns. Because the counter carries virtual_key_id and team_id labels, per-team chargeback is a Prometheus query rather than a nightly reconciliation script.

Time to First Token and Tail Latency

TTFT is the streaming metric that correlates most directly with user-perceived latency. Bifrost records it as a histogram (bifrost_stream_first_token_latency_seconds) alongside inter-token latency and full upstream latency, so histogram_quantile(0.95, ...) on any of them is a one-line PromQL query. The bifrost_active_requests gauge covers in-flight concurrency. A companion post on monitoring latency and cost in LLM operations breaks the p95 and p99 patterns down further.

Errors, Fallbacks, and Retries

bifrost_error_requests_total breaks failures down by status_code, provider, and model, which is what matters when triaging a spike. Retry activity is captured in bifrost_request_retries (a histogram bucketed 0/1/2/3/5/10), and key rotations (a per-key failure that pushed Bifrost to a different API key) are counted in bifrost_key_rotation_events_total with a fail_reason label of rate_limit_error, authentication_error, or billing_error. That is enough to distinguish "our provider is unhealthy" from "one of our API keys is exhausted." Fallback chains are configured on the Retries and Fallbacks page.

Cache Hit Ratio

Bifrost's semantic caching supports two lookup paths: direct hash matching (deterministic exact replay) and semantic similarity matching (embedding-based). bifrost_cache_hits_total carries a cache_type label so hit ratio can be tracked per path. A sudden drop typically points to a missing x-bf-cache-key header, not to a cache outage, and the label breakdown surfaces that in seconds.

MCP Tool Calls

Agent workloads spend meaningful time inside tools, not inside the model. bifrost_mcp_client_operation_duration_seconds is a histogram of MCP tool-call duration, labeled by mcp_client, mcp_tool_name, and error_type (empty on success, auth_required or _OTHER on failure). It captures the tool-execution side of an agent turn separately from the model call, so a slow agent can be attributed to a specific tool without adding tracing to the tool code. Related: the seven metrics worth tracking for AI agent observability.

Where to Export the Data

Three transports cover essentially every observability stack in production. All three ship in Bifrost and can run simultaneously; the choice depends on where dashboards already live.

Prometheus Scrape or Push Gateway

The Prometheus route is the shortest path to a working dashboard. Bifrost exposes a /metrics endpoint whenever the telemetry plugin is enabled (on by default), and any Prometheus server can be pointed at it with a standard scrape_configs entry. A companion piece covers Prometheus metrics and dashboards for LLM traffic in detail. For multi-node clusters where scraping through a load balancer would miss nodes, Bifrost also supports pushing metrics to a Prometheus Push Gateway.

OpenTelemetry (OTLP)

The OTel plugin exports traces (and, in newer versions, metrics) to any OTLP collector over HTTP or gRPC. Bifrost emits spans following the OpenTelemetry GenAI semantic conventions, so backends that already understand GenAI attributes get rich model, prompt, and token metadata without custom parsing.

Session grouping (group_traces_by_session) collapses every request that shares an x-bf-session-id header into one trace, the correct view for a multi-turn agent run or a Claude Code session. A deeper walkthrough on OpenTelemetry traces and metrics for LLM observability covers the collector setup end to end.

Datadog Connector

For teams standardized on Datadog, the Datadog connector is a native integration built on dd-trace-go v2 rather than a generic OTLP hop. It sends APM traces, Datadog LLM Observability payloads, and DogStatsD metrics in one plugin, and supports both Agent mode (through a local Datadog Agent) and Agentless mode (direct to Datadog's intake APIs). LLM Observability spans land in Datadog's LLM Obs dashboards with model, prompt, and cost attributes already populated.

Implementation Walkthrough: Metrics in Ten Minutes

The minimum viable observability setup takes three steps once Bifrost is running. Nothing changes in application code; every metric described above is captured automatically by the telemetry plugin.

Step 1: Enable telemetry. The telemetry plugin is on by default. If disabled, re-enable it in config.json or through the UI. The /metrics endpoint becomes live on the same port Bifrost serves inference on (default 8080).

Step 2: Point Prometheus at Bifrost. A single scrape_configs entry is enough:

scrape_configs:
  - job_name: 'bifrost'
    static_configs:
      - targets: ['bifrost-host:8080']
    scrape_interval: 15s

For clusters behind a load balancer, configure push_gateway_url so every node pushes metrics into a shared Push Gateway.

Step 3: Add the OTel or Datadog plugin as needed. Enable the otel plugin with a collector_url, or the datadog plugin with an agent address (or an API key for agentless mode). Both are configured entirely in config.json and do not require rebuilding the gateway.

Once the plugins are running, three starter queries produce a usable dashboard:

# 95th-percentile time to first token, per model
histogram_quantile(0.95,
  sum by (le, model) (rate(bifrost_stream_first_token_latency_seconds_bucket[5m])))

# USD cost per hour, per team
sum by (team_name) (rate(bifrost_cost_total[1h])) * 3600

# Cache hit ratio, per cache type
sum by (cache_type) (rate(bifrost_cache_hits_total[5m]))
  / sum by (cache_type) (rate(bifrost_upstream_requests_total[5m]))

Full Traces, Not Just Metrics

Metrics answer "how many" and "how fast." A trace answers "what happened in this specific request." Bifrost's built-in observability captures a full log for every request, including the input messages, model parameters, chosen provider, output, token counts, cost, and latency, and records the retry attempt_trail: an ordered array of every attempt with its key_id, fail_reason, and whether it triggered a key rotation. That is enough to reconstruct why a request took 8 seconds when the p50 for that model is 400 milliseconds.

For long-term retention and compliance, audit logs record administrative activity (who changed which config, when, and to what) with HMAC-signed events. Log exports stream request and response payloads to S3 or GCS so the logs database stays small. Together they cover the audit-trail requirements for SOC 2, GDPR, HIPAA, and ISO 27001 workloads. Related: LLM monitoring metrics, audit logs, and controls.

Governance Labels: Attribution Without Correlation

A metric without attribution answers only fleet-wide questions. Bifrost stamps every request with the identifiers platform teams actually query on: virtual_key_id, team_id, customer_id, and business_unit_id, alongside human-readable name pairs. Those labels flow through to Prometheus, OTLP, and Datadog automatically. Per-team cost, per-customer error rate, and per-virtual-key rate-limit hits become one PromQL expression rather than a cross-service correlation job. The governance resource page covers how virtual keys become the primary attribution entity and how budgets and rate limits attach to them.

Beyond Metrics: Evaluation and Session-Level Quality

Gateway metrics answer operational questions with high resolution. Model-quality questions (whether an agent completed the task, whether hallucinations increased after a prompt change, whether a rewrite regressed answer quality on a golden set) sit above the gateway. Maxim AI's evaluation platform reads the same gateway-side traces Bifrost produces and adds session-level scoring, offline evaluations, and simulation. Bifrost measures what happened in production; the evaluation platform measures whether what happened was correct.

Frequently Asked Questions

What is observability in LLMs?

Observability in LLMs is the practice of capturing tokens, cost, latency (including time to first token), errors, retries, fallbacks, cache behavior, and tool-call activity for every model request, then routing that telemetry into the metrics, tracing, and logging systems the platform team already runs. It expands generic API observability with the fields specific to LLM workloads, so teams can debug streaming latency, attribute cost to tenants, and confirm that failover actually fired.

Which LLM observability tool is the best?

The right answer depends on where the observability stack lives. Teams on Prometheus and Grafana get the fastest path by scraping the gateway's /metrics endpoint. Teams on Datadog use the native connector for APM traces, LLM Observability, and DogStatsD metrics. Teams on OpenTelemetry point Bifrost at their existing OTLP collector and inherit whatever backend it feeds.

Does Datadog have LLM observability?

Yes. Datadog offers a dedicated LLM Observability product with dashboards for prompts, completions, tokens, and cost. Bifrost integrates natively with it through the Datadog connector, which uses dd-trace-go v2 and Datadog's LLM Obs SDK to send APM traces, LLM Observability payloads, and DogStatsD metrics from every request through the gateway. The connector supports both Agent mode and Agentless mode.

What is time to first token (TTFT)?

Time to first token is the elapsed time between when the LLM gateway forwards a request to a provider and when the first response token arrives on the streaming connection. It is the dominant perceived-latency signal for streaming user interfaces because it determines how long the user stares at an empty cursor before text starts appearing. Bifrost records TTFT as a histogram (bifrost_stream_first_token_latency_seconds) labeled by provider and model, so p95 TTFT per model is a one-line PromQL query.

How does an AI gateway differ from generic API monitoring for LLM traffic?

An AI gateway records fields generic API monitoring does not: token counts (input and output) per request, USD cost per response, TTFT and inter-token latency for streaming, per-provider error attribution across a fallback chain, cache-hit type (direct vs semantic), and MCP tool-call duration. It also stamps every request with virtual key, team, and customer labels for native attribution. Generic APM tools record HTTP status and latency, which is a small fraction of what an LLM workload needs.

Do I need OpenTelemetry, Prometheus, and Datadog, or just one?

Just one is enough to start, and Bifrost lets teams pick whichever their platform stack already uses. Prometheus is the shortest path if there is no observability backend yet. OpenTelemetry is the right choice when a collector already exists. The Datadog connector fits when Datadog is the standard. All three plugins can run at the same time during a migration.

Where should governance labels be applied?

At the gateway. Bifrost enforces governance through virtual keys and stamps virtual_key_id, team_id, customer_id, and business_unit_id on every metric, span, and log automatically. That means per-team cost, per-customer error rate, or per-virtual-key rate-limit rotations are queries against existing labels, not cross-service joins between the gateway, the identity provider, and the application.

Get Started with LLM Observability on Bifrost

Every metric in this post is captured by Bifrost natively, and every export path (Prometheus, OpenTelemetry, and the Datadog connector) is configured entirely in config.json without touching application code. Teams running regulated or high-scale AI workloads can pair this with Bifrost Enterprise for clustering, in-VPC deployment, RBAC, and audit logs.

To see how gateway-side LLM observability fits into your platform stack, book a demo with the Bifrost team, or start from the LLM Gateway Buyer's Guide for benchmarks and integration walkthroughs.