Open Source Observability Platform for LLM and Agent Workloads

TL;DR

• An open source observability platform for LLM traffic is assembled from four building blocks: the OpenTelemetry Collector for pipelines, Prometheus for metrics, Grafana for dashboards, and Jaeger (or another OTLP backend) for traces.
• A generic stack records HTTP status codes and latency but has no native concept of tokens, per-request cost, provider fallback events, or MCP tool-call spans, so those signals have to be emitted by something that sits in the request path.
• Bifrost emits all four missing signals at the gateway layer: Prometheus counters for tokens and cost, OpenTelemetry GenAI spans for every request type, a fallback_index label on request metrics, and MCP client spans for tool calls.
• Bifrost exports to Prometheus (scrape or Push Gateway), any OTLP collector over HTTP or gRPC, and Datadog natively, and offloads request payloads to S3 or GCS for long-term retention.

An open source observability platform is a self-hosted stack that collects metrics, traces, and logs from production systems and stores them where engineers can query and alert on them. For teams running LLM and agent workloads, the standard stack (OpenTelemetry Collector, Prometheus, Grafana, Jaeger) covers infrastructure signals well and covers AI-specific signals badly, because nothing in it knows what a token, a fallback, or a tool call is. Bifrost, the open-source AI gateway built by Maxim AI, closes that gap by emitting LLM-native metrics and traces from the one place every request already passes through, and it is the best choice for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability. This post compares the open source building blocks, is specific about what each one cannot see in AI traffic, and shows how Bifrost feeds them.

What Is an Open Source Observability Platform?

An open source observability platform is a set of freely licensed components that collect, store, and visualize telemetry (metrics, traces, and logs) so engineers can understand system behavior without depending on a vendor-hosted backend. The four projects most teams combine are the OpenTelemetry Collector, Prometheus, Grafana, and Jaeger; the first, second, and fourth are Cloud Native Computing Foundation projects, and Grafana is maintained by Grafana Labs under an open source license.

The appeal is control. Data stays inside the network boundary, retention is a storage decision rather than a pricing tier, and every component can be replaced without rewriting instrumentation, because the OpenTelemetry protocol (OTLP) is the shared interface between them. The Grafana Labs Observability Survey 2026 found that 77% of respondents rate open source and open standards as important to their observability strategy, and the most-cited reason was avoiding vendor lock-in.

For LLM applications, the same stack is the right foundation and the wrong instrumentation. A Prometheus histogram of request latency cannot distinguish a slow model from a slow provider retry, and a Jaeger trace of an HTTP call to an inference API shows a single span where the interesting work (prompt size, token counts, tool execution, fallback to a second provider) is invisible. Bifrost exists in the request path to make those signals visible, and the built-in observability layer captures tokens, cost, latency, and tool calls for every request before exporting them to whichever backend the team runs.

Open Source Observability Platforms Compared at a Glance

The four open source observability platforms below are complementary rather than competing: each owns one signal type or one stage of the pipeline. The table summarizes what each component does, which signal it handles, and what it needs from an upstream emitter to be useful for LLM traffic. Bifrost is listed first because it is the emission point that feeds the other four.

Two things follow from the table. First, no component in the open source stack produces LLM-specific data on its own; every one of them consumes what an emitter sends. Second, the emitter determines the quality of the whole system, because a metric that was never recorded cannot be dashboarded later. Placing that emitter at the AI gateway rather than in each application means one integration covers every service, SDK, and coding agent that sends traffic through Bifrost.

OpenTelemetry Collector: The Pipeline Layer

The OpenTelemetry Collector is a vendor-neutral agent that receives telemetry over OTLP, applies processors such as batching, filtering, and attribute redaction, and exports it to one or more backends. It is the component that lets a team change storage vendors without touching instrumentation, and it is the normal target for anything Bifrost emits as traces.

The Collector has no opinion about LLM data. It will forward a span named gen_ai.chat with a gen_ai.usage.cost attribute exactly as readily as a span named http.request, but it does not create either. Whether the span carries token counts, the resolved provider, or the model that finally served the request is decided entirely upstream, which is why the OpenTelemetry GenAI semantic conventions matter: they define a shared attribute vocabulary (gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.prompt_tokens, and so on) so that dashboards built for one emitter work for another. The conventions now live in the OpenTelemetry GenAI semantic conventions repository.

Bifrost emits traces in the genai_extension format, which follows those conventions, through its OpenTelemetry plugin. The plugin supports OTLP over HTTP and gRPC, reads OTEL_RESOURCE_ATTRIBUTES from the environment, and can group every request that shares an x-bf-session-id header into a single trace. Teams that already run a Collector point collector_url at it and get LLM traces alongside their existing application spans, a pattern covered in more detail in the guide to OpenTelemetry for LLM observability.

Prometheus and Grafana for Metrics and Dashboards

Prometheus and Grafana together form the metrics half of most open source observability platforms: Prometheus scrapes /metrics endpoints on an interval and stores time series in a local database, and Grafana queries Prometheus to render dashboards and evaluate alert rules. Both are mature, widely deployed, and the default choice for Kubernetes clusters, which is where most LLM gateways run.

The limitation for AI workloads is label design. A generic HTTP exporter labels requests by path, method, and status code, so a Grafana panel can show that /v1/chat/completions returned 200 with a p95 latency of 2.4 seconds. It cannot show input and output token volume, cost per team, cache hit rate, or how many requests succeeded only after falling back to a second provider, because those values were never scraped. Every one of those is a first-class metric in Bifrost: bifrost_input_tokens_total, bifrost_output_tokens_total, bifrost_cost_total, bifrost_cache_hits_total, and bifrost_success_requests_total all carry provider, model, team_id, customer_id, and fallback_index labels, and are documented on the Prometheus metrics page.

Bifrost exposes those metrics two ways. Single-node deployments let Prometheus scrape /metrics directly (with Basic auth when Bifrost authentication is enabled). Clustered deployments push to a Prometheus Push Gateway so that nodes behind a load balancer are never missed, which pairs with clustering for multi-node setups. The companion post on LLM observability with Prometheus walks through the dashboard queries.

Jaeger Tracing for Distributed Requests

Jaeger is a CNCF-graduated distributed tracing backend that stores spans, reconstructs the parent-child tree for each trace, and lets engineers search by service, operation, tag, and duration. It ingests OTLP natively, so the OpenTelemetry Collector is optional in front of it, and Grafana can query Jaeger as a data source so traces and metrics live in one interface.

Jaeger tracing works well for LLM traffic only when the spans it receives are structured for it. A single span wrapping an outbound HTTPS call to a provider tells the reader the call took 3.1 seconds and returned 200. The same request seen from Bifrost is a root span with child spans for the pipeline phases, attributes for the provider, model, temperature, and token usage, and (when the model calls tools) MCP client spans carrying mcp.method.name and gen_ai.tool.name, so a slow agent turn can be attributed to the model, the provider, or a specific tool.

Bifrost sends that shape to Jaeger through the same OTel plugin used for any other OTLP backend, and its latency breakdown view decomposes Bifrost's own overhead per request into components such as serialization, plugins, and routing, which are also exportable as the bifrost_overhead_component_microseconds histogram. Teams that need every tool call recorded as a governed event, not only as a span, should read how MCP gateway observability treats tool execution.

What a Generic Observability Stack Misses for LLM Traffic

A generic observability stack built from the OpenTelemetry Collector, Prometheus, Grafana, and Jaeger misses four categories of signal that decide whether an LLM application is healthy: token consumption, per-request cost, provider fallback events, and tool-call spans. Each is absent for the same reason: the stack records what the HTTP layer exposes, and the HTTP layer does not expose any of them.

The fallback row is the one most teams discover late. Bifrost retries and fallbacks will move a request from a rate-limited key to a healthy one, or from one provider to the next in a configured chain, and the caller receives a normal response. Without a fallback_index label and an attempt_trail record, a dashboard shows 100% success while the primary provider has been failing for an hour. The bifrost_provider_key_up gauge and bifrost_key_rotation_events_total counter surface that condition as a metric Grafana can alert on.

The tool-call row matters for agents specifically. When Bifrost acts as an MCP gateway, every tool invocation passes through it, so a tool that takes four seconds or returns an error appears as its own span with the governance identity (virtual key, team, customer) attached, rather than disappearing inside the agent's turn.

Bifrost as the Emission Point for LLM Observability

Bifrost is an AI gateway that routes requests to 25+ providers and 10,000+ models through one OpenAI-compatible API, which makes it the natural emission point for LLM observability: every request, from every SDK and coding agent, passes through it once. Instrumenting the gateway once replaces instrumenting each application, and it produces the same telemetry shape regardless of which provider served the call.

Three properties make the gateway position work. The first is coverage: the supported providers matrix means a request to a model on a cloud platform, a hosted API, or a self-hosted inference server is logged and traced identically. The second is that the observability plugins run asynchronously in background goroutines, so request logging and span emission do not add to the critical path; Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second with a 100% success rate in sustained benchmarks.

The third property is that governance identity travels with the telemetry. Because virtual keys identify the calling team, customer, and project, every metric and span carries those labels without the application setting them, and a cost or latency panel can be filtered by team from the first scrape.

The request log itself records what generic tooling cannot: the full input history and output message, model parameters, the provider and model that handled the call, token usage, cost, latency, and, when retries happened, the ordered attempt_trail of keys tried and why each failed. Content logging can be disabled globally or per export profile, and when Enterprise guardrail redaction is active, logs and exported spans store the redacted form of any detected content. The complete guide to LLM logging and OTel tracing in Bifrost covers the log schema field by field.

Exporting to Prometheus, OpenTelemetry, and Datadog from Bifrost

Bifrost exports telemetry to three families of backend: Prometheus (pull-based scraping or push to a Push Gateway), any OpenTelemetry collector over OTLP (traces, and optionally metrics, over HTTP or gRPC), and Datadog through a native connector that uses Datadog's own SDKs for APM traces, LLM Observability, and metrics. All three are configured in config.json or from the Bifrost UI, and OTel export settings are defined per profile.

A minimal OpenTelemetry profile that sends GenAI-convention traces to a local Collector and pushes the Prometheus-style metrics over the same connection looks like this:

{
  "name": "otel",
  "enabled": true,
  "config": {
    "service_name": "bifrost",
    "collector_url": "env.OTEL_COLLECTOR_URL",
    "trace_type": "genai_extension",
    "protocol": "grpc",
    "metrics_enabled": true,
    "metrics_endpoint": "env.OTEL_METRICS_URL",
    "disable_content_logging": true
  }
}

The disable_content_logging flag on the profile drops prompt and completion text from exported spans while keeping model, provider, tokens, cost, latency, and governance attribution, which is the usual setting when the trace backend is shared with teams who should not see raw prompts. The Datadog connector offers the same flag and supports agent and agentless modes, and the OTel plugin's per-signal headers let traces and metrics authenticate to different endpoints.

For retention beyond what a traces backend holds, Enterprise log exports offload request and response payloads from the logs database to S3 or GCS while metadata stays queryable in Postgres or ClickHouse; S3 and GCS are the two supported destinations today. Teams that also run semantic caching see cache hits as a separate counter, so a Grafana panel can show cost saved alongside cost spent, a combination discussed in the post on the best open source platform for semantic caching and smart LLM routing.

Self-Hosted Deployment for Regulated Environments

Self-hosting is the reason most teams choose an open source observability platform in the first place, and it applies to the emitter as much as to the backends. Bifrost deploys as a single Go binary or container, runs on Kubernetes with the same Helm patterns as the Collector and Prometheus, and keeps request content inside the network boundary because no telemetry leaves unless an export is configured.

For regulated industries, three capabilities matter beyond the exports themselves. In-VPC deployment runs Bifrost with no public network egress, so prompts and completions never transit a third-party service. Role-based access control governs who can view request logs and who can reveal redacted content, with a separate Logs:Reveal permission for redaction mappings. Audit logs record administrative activity (who changed which virtual key, budget, or provider configuration, and when) with HMAC-signed entries, dashboard filtering, export to JSON, JSON Lines, or Syslog, and optional archival to object storage; they are distinct from request logs and do not duplicate them.

Bifrost Enterprise also adds clustering for high availability, which is where the Push Gateway and OTLP metrics push paths become necessary rather than optional, since per-node scraping behind a load balancer produces gaps. The post on the best open source AI gateway for self-hosted deployment compares the deployment options in more depth, and the governance resource page covers the access-control model that the observability labels are built on.

Teams evaluating gateways more broadly will find that the LLM gateway buyer's guide applies the same observability criteria (tokens and cost per request, fallback visibility, tool-call spans, OTLP as the interface) alongside routing and governance requirements.

Frequently Asked Questions

What is OpenTelemetry?

OpenTelemetry is an open source, vendor-neutral framework for generating, collecting, and exporting telemetry: traces, metrics, and logs. It defines the OTLP wire protocol, semantic conventions for attribute names (including the GenAI conventions for LLM calls), SDKs for instrumenting applications, and the Collector for routing telemetry to backends. Bifrost emits traces and metrics in OTLP through its OTel plugin so any OpenTelemetry-compatible backend can consume them without translation.

Is Grafana open source?

Yes. Grafana is open source under the AGPL-3.0 license and can be self-hosted to build dashboards and alerts over Prometheus, Jaeger, Tempo, Loki, and many other data sources. Grafana Labs also sells a hosted Grafana Cloud service. In an LLM observability stack, self-hosted Grafana queries the Prometheus metrics and OTLP traces that Bifrost emits, and the same dashboards work against Grafana Cloud if the team later moves.

Is Prometheus open source?

Yes. Prometheus is an Apache-2.0 licensed, CNCF-graduated monitoring system that scrapes metrics from HTTP endpoints, stores them in a local time-series database, and evaluates alert rules with PromQL. Bifrost exposes a /metrics endpoint in Prometheus exposition format by default and can push the same metrics to a Prometheus Push Gateway for multi-node clusters, so no additional exporter is needed.

What is LLM observability?

LLM observability is the practice of recording and analyzing the signals specific to large language model traffic: token usage, cost per request, model and provider selection, retries and fallbacks, streaming latency, tool calls, and the prompts and completions themselves. It extends conventional observability, which sees only HTTP latency and status codes, with the data needed to debug quality, control spend, and attribute failures to a model, a provider, or a tool.

What is the difference between observability and monitoring?

Monitoring checks known conditions against thresholds: is the error rate above 1%, is p95 latency under two seconds. Observability is the ability to ask new questions of the system from the telemetry it already emits, such as which team's requests fell back to a secondary provider between 2 and 3 a.m. Metrics support monitoring; traces and structured request logs with rich labels, which Bifrost produces at the gateway, are what make LLM systems observable.

Which open source observability platform is best for LLM workloads?

There is no single open source observability platform that covers LLM workloads alone; the effective setup is the OpenTelemetry Collector for pipelines, Prometheus and Grafana for metrics and dashboards, and Jaeger or another OTLP backend for traces, fed by an emitter that understands LLM traffic. The Bifrost AI gateway fills the emitter role, producing token, cost, fallback, and tool-call telemetry at the gateway and exporting it to all of those backends and to Datadog.

Getting Started with Bifrost and Your Observability Stack

An open source observability platform gives a team ownership of its telemetry, and Bifrost gives that platform the LLM-specific signals it cannot generate on its own: tokens, cost, fallback events, and tool-call spans, exported to Prometheus, OpenTelemetry, and Datadog from a gateway that adds 11 microseconds per request. Enabling the telemetry plugin and pointing one OTel profile at an existing Collector is typically a same-day change, and the drop-in replacement model means applications keep their current SDKs while the gateway takes over emission.

To see Bifrost emitting LLM observability data into your own Prometheus, Grafana, Jaeger, or Datadog deployment, book a demo with the Bifrost team, or start from the resources hub and the open source repository to run it against your stack today.