Observability for LLM Traffic: Collect, Filter, Enrich, Route

TL;DR

• An observability pipeline for LLM traffic moves telemetry through four stages: collect at the AI gateway, filter payloads and low-value spans, enrich with cost and routing context, and route each signal to the backend that can use it.
• LLM telemetry breaks conventional pipelines on four axes: multi-kilobyte prompt payloads, unbounded label cardinality, personal data inside spans, and telemetry cost that scales with token volume.
• Only the gateway can enrich a request with cost per call, cache hit type, fallback reason, and virtual-key attribution, because those values are computed at the gateway and never exist in application code.
• Bifrost, the open-source AI gateway, emits those fields as Prometheus labels, OpenTelemetry span attributes, and request-log fields, and adds 11 microseconds of overhead per request at 5,000 RPS.

Observability pipelines for LLM traffic fail for a specific reason: the telemetry an LLM request produces looks nothing like the telemetry an HTTP service produces, and pipelines built for the second are fed the first. A single chat completion can carry a 40 KB prompt, a customer's email address in the user turn, and a dollar cost that no application-side instrumentation can compute. Bifrost, the open-source AI gateway built in Go by Maxim AI, sits at the one point in the request path where all of those can be handled before the data reaches Prometheus, an OpenTelemetry Collector, or a data warehouse. This post walks through the four pipeline stages and what the gateway contributes at each.

What Is an Observability Pipeline for LLM Traffic?

An observability pipeline is a processing layer that sits between the systems producing telemetry and the backends storing it, and that collects, filters, enriches, and routes logs, metrics, and traces in transit. For LLM traffic, the pipeline's collection point is the AI gateway, because the gateway is the only component that sees every model call from every application.

The four stages are the same ones the OpenTelemetry Collector formalizes as receivers, processors, and exporters. What changes for LLM telemetry is the content: a language model request produces the full conversation history, tool definitions, model parameters, streamed output chunks, token counts, and a cost figure that depends on a pricing table the application never sees. The companion guide to LLM observability at the gateway covers what to measure; this post covers how the data should move. Bifrost participates from the collect stage onward: its built-in request logging captures inputs, outputs, tokens, cost, and latency for every request asynchronously, and its Prometheus and OpenTelemetry plugins export the same data in the formats downstream tooling expects.

Why LLM Telemetry Breaks Conventional Observability Tooling

Conventional observability tooling assumes small, structured, low-cardinality events that contain no regulated data. LLM traffic violates all of those assumptions at once, so a pipeline tuned for microservice telemetry will drop LLM data, exceed its storage budget, or store personal data it was never meant to hold.

Each row is a pipeline design problem. Payload size argues for filtering content out of spans and offloading bodies to object storage. Cardinality argues for a fixed label set on metrics and unbounded detail only in logs and traces. Sensitive content argues for redaction before export. Cost-of-call being a gateway-only fact argues for enrichment at the gateway. The LLM monitoring metrics, audit logs, and controls an enterprise needs depend on those four decisions.

Bifrost keeps the pipeline's own overhead out of the request path: the logging plugin writes in background goroutines and adds under 0.1 ms, and the gateway adds 11 microseconds of overhead per request at 5,000 RPS in sustained benchmarks.

Stage 1: Collect LLM Telemetry at the AI Gateway

Collection is the stage where the pipeline decides what it will ever be able to know. For LLM observability, the gateway is the correct collection point because it is the single place that sees every request, provider response, retry, and routing decision, whichever SDK, framework, or coding agent produced the call.

Application-side instrumentation captures only what the application knows: the prompt it sent and the response it received. It cannot see that the request was retried on a rate-limited key, served by a fallback provider, or answered from cache. The Bifrost AI gateway captures those events because it performs them, in three parallel outputs:

• Request logs, stored in SQLite, Postgres, or ClickHouse through the log store, with full content, attempt_trail for every retry, and routing_engines_used for every routing decision
• Prometheus metrics from the telemetry plugin, exposed at /metrics or pushed to a Push Gateway, with a fixed, bounded label set
• OpenTelemetry traces from the OTel plugin, emitted as gen_ai.* spans over OTLP/HTTP or OTLP/gRPC to any collector

Metrics answer aggregate questions cheaply, traces show one request end to end, and logs hold the payload. A complete guide to LLM logging and OTel tracing in Bifrost covers configuration for each; for pipeline design, the point is that all three originate from the same hook, so their fields agree by construction. Caller context is carried too: any x-bf-lh- request header is captured into log metadata, and forwarded headers appear on the llm.call span as gen_ai.request.extra_header.<name> attributes.

Stage 2: Filter Payloads, Plugin Spans, and Content

Filtering is where the pipeline controls its own cost. An LLM span carrying a 40 KB prompt is two orders of magnitude larger than a conventional span of a few hundred bytes, and a trace recording every plugin hook carries sixteen plugin spans per request (eight built-in plugins, two hooks each). The filter stage decides what leaves the gateway.

Bifrost exposes three filtering controls at the source, where the bytes never cross the network:

The client-level disable_content_logging flag governs only the Bifrost log store, so a team that wants content out of both the local database and the OTLP export sets both flags. For payloads that must be retained but not in the hot database, log exports stream request and response bodies to S3 or GCS object storage while the logs database keeps only metadata and pointers; object_storage_exclude_fields can skip raw prompts entirely. Only S3 and GCS are supported today; Azure Blob and warehouse destinations are not implemented. Sensitive data is also a filter-stage decision that belongs at the gateway rather than in a collector processor, as the guide to gateway-level PII redaction before provider transmission explains.

Stage 3: Enrich With Cost, Cache Hits, Fallback Reason, and Virtual Key

Enrichment is the stage that justifies putting the pipeline's entry point at the gateway. Four fields that every cost report, incident review, and chargeback model depends on are computed at the gateway and exist nowhere else: the dollar cost of the call, whether the response came from cache, why a fallback fired, and which virtual key the request ran under.

A downstream processor cannot add these because it lacks the inputs: the pricing catalog and exact token counts, the cache lookup result, the failed attempt's error type, and the governance layer that authenticated the request. Bifrost has all four when the response completes, and writes them to every output.

The cost figure comes from the Model Catalog, which resolves pricing for the provider and model that actually served the request, including a fallback model. The cache signal distinguishes a direct hash hit from a semantic hit; the semantic caching plugin records the embedding model, similarity score, and threshold in cache_debug, so a pipeline can separate exact-match savings from similarity-match savings, as the deep dive on semantic caching for LLMs explains.

Fallback enrichment is deliberately categorical. The routing trail records only the error type (rate_limit_error, authentication_error, billing_error) and HTTP status for each failed attempt, never the upstream provider message, because providers can echo API keys or user input in error bodies, so automatic fallbacks are auditable without pushing secrets into the trace store. Virtual-key attribution turns a telemetry pipeline into a chargeback pipeline: because virtual keys carry team, customer, and project identity, every sample, span, and log row arrives already labeled with who to bill.

Stage 4: Route Signals to Prometheus, the OpenTelemetry Collector, and Data Stores

Routing sends each signal to the backend suited to its shape and retention. Metrics belong in a time-series database, traces in a trace backend reached through an OpenTelemetry Collector, and full payloads in object storage or a warehouse. A single LLM request in Bifrost can fan out to all three without any backend receiving data it cannot use.

The Prometheus path is the most common starting point and is covered in LLM observability with Prometheus metrics and dashboards. Prometheus observability for a gateway cluster has one trap: scraping /metrics through a load balancer samples whichever node answered, so clustered deployments should push metrics through the Push Gateway or the OTel plugin's metrics_enabled mode to a central collector.

The OpenTelemetry path carries the richest data. Bifrost's genai_extension trace format follows the OpenTelemetry GenAI semantic conventions, so gen_ai.provider.name, gen_ai.request.model, and token usage attributes are named the way collector processors and trace backends expect. From the collector, the transform, filter, and attributes processors apply a second round of shaping before export.

The walkthrough of OpenTelemetry for LLM observability shows that second round with a working collector config. Regulated deployments that need these connectors inside a private network run them from the in-VPC enterprise build.

Controlling High Cardinality Metrics in LLM Pipelines

High cardinality metrics are the fastest way for an LLM observability pipeline to exceed its storage budget. Every unique combination of label values creates a new time series, and LLM traffic offers thousands of candidates: model names, prompt versions, users, sessions, tenants, and tool names. The pipeline has to fix a bounded label set for metrics and push everything else into traces and logs.

The Prometheus instrumentation guidelines recommend keeping the cardinality of most metrics below 10 and treating anything that could exceed 100 as a signal to move that dimension out of monitoring into a general-purpose processing system. LLM telemetry crosses that line the moment user or session identifiers become labels.

Bifrost enforces the boundary at the source:

• The HTTP path label carries the matched route template (/v1/messages/batches/{batch_id}), not the raw URL, so cardinality is bounded by registered routes rather than by every model or resource ID in a URL
• The raw complexity score from semantic routing is excluded from labels because it is unbounded; only complexity_tier (SIMPLE, MEDIUM, COMPLEX) is labeled, and the score stays in request logs and trace attributes
• Governance identity (virtual key, team, customer, project) is labeled because it is bounded by the entities an organization configures, which is exactly the set a cost report needs

The rule this encodes: label what an organization provisions in its governance configuration, and log or trace what users generate at runtime. A per-session breakdown is a trace query, not a Prometheus query.

Keeping Personal Data Out of LLM Spans and Logs

Personal data in LLM telemetry is routine, because the user's message is the payload. A pipeline that exports spans with content to a third-party trace backend has copied every email address and support transcript into a system that was never in scope for the original data-processing agreement. Redaction therefore has to happen before the data leaves the gateway and apply consistently to logs, traces, and connector exports. Bifrost's guardrail redaction offers three modes:

• runtime rewrites detected text in the live request and response with a replace, mask, or hash strategy, so the provider, the logs, and every export receive the redacted form
• logs_only leaves runtime content untouched but stores reversible placeholders such as [EMAIL-1] in Bifrost logs and sends placeholderized content to trace-export connectors
• runtime_reversible redacts at runtime with reversible placeholders so a user with the Logs:Reveal permission can recover originals inside Bifrost, while connectors receive placeholders only

In every mode the reveal mapping stays inside Bifrost, and Prometheus receives no request or response content at all. Detection can come from an external guardrail provider or from the in-process custom regex guardrail, whose built-in PII Detection template covers email addresses, US phone numbers, US Social Security numbers, and credit-card-like numbers; it is pattern-based and does not detect personal names. The guardrail options page covers the managed and provider-backed detectors.

One boundary matters for compliance teams: Bifrost audit logs record administrative activity (who changed which configuration, when), with HMAC-signed entries and S3 or GCS archival. They are not a copy of request traffic, which lives in the request logs and the pipeline above; the two should be retained under separate policies. That division is part of deciding what to measure and where at the gateway, and regulated deployments formalize it in the Bifrost Enterprise tier.

Frequently Asked Questions

What is the difference between an observability pipeline and an OpenTelemetry Collector?

An OpenTelemetry Collector is one implementation of an observability pipeline: its receivers, processors, and exporters map to the collect, filter and enrich, and route stages. For LLM traffic, the collector cannot enrich a span with cost, cache hit type, or fallback reason because it never had those inputs; the gateway supplies them, and the collector shapes and forwards what the gateway emits.

How do you keep PII out of LLM traces?

Redact at the gateway before export, and apply the same policy to logs and connectors. Bifrost's runtime redaction mode rewrites detected text before the provider call so every downstream system receives the redacted form; logs_only and runtime_reversible send placeholders to connectors while keeping the reveal mapping inside Bifrost. For content that should never leave, disable_content_logging on the OTel profile drops message bodies while keeping cost, tokens, and attribution.

How does Bifrost export Prometheus metrics from a multi-node cluster?

Bifrost supports pull-based scraping of /metrics on each node and push-based export to a Prometheus Push Gateway or an OTLP metrics endpoint. For clusters behind a load balancer, push is the reliable option, because a scrape that reaches the balancer samples only the node that answered. Push interval, job name, instance ID, and basic auth are configurable.

Getting Started with Bifrost Observability Pipelines

Observability pipelines for LLM traffic succeed when the collect stage sits where the cost, cache, fallback, and attribution context already exists: the AI gateway. Bifrost emits all four fields to metrics, traces, and logs, filters content and plugin spans at the source, redacts personal data before export, and routes each signal to Prometheus, any OpenTelemetry Collector, Datadog, Kafka, BigQuery, or object storage without changing application code. Start with the Bifrost resources hub for configuration references, or book a demo to design an observability pipeline with the Bifrost team for your provider mix and compliance requirements.