Rust vs Go for AI Gateways: What the Benchmarks Show

TL;DR

• In a shared AI gateway benchmark, the Rust vs Go latency result was mixed: Go (Bifrost) added less p99 latency at concurrency 1 and 64, Rust (LiteLLM Rust beta) added less at 4 and 16, and a second run at 64 reversed the order.
• The Go and Rust gateways matched at about 2,720 RPS at concurrency 64, but both were capped near a client ceiling of about 2,900 RPS, and both passed 7,700 RPS at concurrency 256, so the tie does not measure capacity.
• Rust used about 5x less CPU in the ceiling run (21.1% versus 105.4%) and less memory in three of four file sets, though Go used less memory in that run and its arm kept default logging on.
• The gap to Python was far larger: +546 ms p99 at concurrency 64 in an apparently single-worker run, against +28.68 ms for Go and +40.28 ms for Rust.
• Language choice matters less to an AI gateway buyer than route coverage, governance, failover, and tail latency with production controls enabled.

Rust vs Go is one of the most common comparisons in backend engineering, and AI gateways now provide a direct test, because LiteLLM's Rust beta and Bifrost, the open-source AI gateway written in Go by Maxim AI, ran against one mock on one host. Most Rust vs Go articles compare toy services, while this benchmark compares two gateways handling the same LLM request. This guide reads the raw files to show where each language leads, where the data cannot separate them, and what should carry more weight in a gateway decision.

Rust vs Go: How the Two Languages Differ

Rust and Go are both compiled languages suited to network services, and they differ mainly in how they manage memory. Rust enforces ownership rules at compile time and has no garbage collector. Go uses a concurrent garbage collector that reclaims memory at runtime.

Figure 1: Rust frees memory at a known point, while Go trades extra heap headroom for simpler allocation, which is one reason Go services often report higher RSS.

An AI gateway is an unusual workload for this debate. It spends most of each request waiting on a model provider, so raw compute speed counts for less than connection handling, allocation behavior under concurrency, and how the runtime responds at the tail.

Rust vs Go Performance in a Shared Gateway Benchmark

LiteLLM's AI Gateway Bench is one of the few places where a Rust gateway and a Go gateway ran against the same mock, on the same host, with the same load driver. The Rust arm was a LiteLLM beta, the Go arm was Bifrost v1.6.4, and the runs date from July 2026.

The table shows added p99 latency over a direct call, from LiteLLM's published latency_vs_concurrency.csv. Lower is better, all error counts are zero, and each cell is one 2,000-request run.

Three cautions apply before reading this as a language result. The two arms did different work, because the published Bifrost configuration leaves logging on by default while LiteLLM reports logging off. No repeated trials or hardware details are published. Each result also reflects one gateway's engineering, not a language in general. Our benchmark review covers the setup in full, and Bifrost's own benchmark results document separate test conditions.

Why Go and Rust Tie on Throughput

Go and Rust gateways both reached about 2,720 RPS at concurrency 64, but that tie comes mostly from the test design. Sixty-four clients waiting about 22 ms per response cannot much exceed 2,900 RPS, and a direct call reached 2,856 RPS. Both gateways reached about 95% of the direct result, so neither was pushed to its limit.

Figure 2: Rust and Go match on throughput because both reach the client ceiling, so the tie does not measure either gateway's maximum capacity.

Little's law explains the pattern: throughput equals concurrency divided by response time. The direct baseline shows it at each level, with 46 RPS at one client and 2,856 RPS at 64 clients. The load driver caps in-flight requests with a semaphore, so each slot waits for a response before starting the next.

Median added latency at concurrency 64 is under 1 ms for both compiled gateways, while added p99 is far higher. That p99 rests on about 20 of 2,000 requests, and a second run recorded 2,744 RPS for Go and 2,814 RPS for Rust with the p99 order reversed. The differences live in the tail, and these runs cannot rank them. Tail latency compounds across fan-out requests, so a capacity test that raises client count until a gateway saturates would tell buyers more than this sweep can. For a sustained-load view, see our high-throughput comparison.

Concurrency 256 supports this reading. LiteLLM excluded the level because the baseline tail exceeded its control limit, but the raw files show direct at 10,686 RPS, Bifrost at 8,266 RPS, and LiteLLM Rust at 7,732 RPS. Both gateways more than doubled their concurrency 64 figures, so 2,720 RPS was not a limit of either one. This was a burst of about a quarter of a second, so the numbers are exploratory.

Is Rust Faster Than Go?

Rust is not consistently faster than Go in this gateway benchmark. In the sweep, the Go gateway has lower p99 at concurrency 1 and 64, and the Rust gateway at 4 and 16. Rust's best result is LiteLLM's headline run at concurrency 16, at +0.66 ms versus +4.54 ms, though the sweep at that level shows +0.33 ms versus +0.89 ms.

Ranking also moves between files from the same benchmark. At concurrency 64, total p99 for Bifrost is 52.7 ms in the sweep and 32.2 ms in rps_per_dollar.csv, while LiteLLM Rust is 64.3 ms and 26.9 ms. With single runs, neither result should settle the question.

Concurrency 256 is unsettled too. On the sweep's driver, p99 was 71.8 ms for Go and 100.9 ms for Rust, while files from an earlier driver show about 5,421 ms and 2,668 ms. Rust did add less time in agent session replays, at 0.03 versus 0.13 seconds. Our review of the benchmark files covers the wider results.

Real model traffic narrows the picture further. LiteLLM's hosted scenario put direct Anthropic streaming at about 530 ms to first token, and reported both its Rust and Python arms as within noise of direct. At that scale, a sub-millisecond gateway difference is hard to detect. LiteLLM's own post says gateway overhead matters mainly at high request rates against fast responses and in agentic loops with many turns.

Rust vs Go Memory Usage for Gateways

Rust used less memory than Go in three of four published file sets, and far less CPU under load. The headline pair is 21.8 MB versus 199.1 MB peak RSS, and two others are 26.1 MB versus 144.2 MB and 36.4 MB versus 298.5 MB. The order reverses in the concurrency 64 ceiling run, where Go peaked at 242.6 MB and Rust at 297.1 MB.

Part of that gap is expected from the languages. Go's collector needs heap headroom between cycles, and Rust has no such requirement. CPU is the clearest difference between the two arms: 21.1% for Rust against 105.4% for Go in the ceiling run, about 5x, with Go's figure including default logging.

Part of the gap is configuration, in a share the files cannot isolate. The Bifrost arm ran its default logging plugin and SQLite log store, and it pre-allocates object pools at startup to reduce allocation during requests. Both can add to baseline memory: Bifrost started near 282 MB in one file and added about 16 MB under traffic. Bifrost 2.x has since reduced memory usage, and Part 2 of our benchmark series will measure that with heap profiles.

The Bigger Gap Is the Python Gateway Run

The largest performance difference in this benchmark sits between the Python run and either compiled gateway, not between Rust and Go. At concurrency 64, the Python gateway added +546.04 ms at p99 and reached 179.8 RPS, about 6% of the ceiling. Go and Rust each stayed near 95%. The published start command sets no worker count and CPU stayed near one core at 98.9%, so this appears to be a single-worker result that more workers would raise.

This matters for buyers because LiteLLM's documentation says /v1/messages and /chat/completions run on Python on main:latest as of November 3, 2026, with Rust serving /ocr and Bedrock /audio/transcriptions. On that image, a benchmark of a Rust build does not describe those routes, although the same docs list rust: true for /v1/messages since v1.94.0. Our guide to how to choose a LiteLLM alternative covers what to check, and the LiteLLM alternatives overview summarizes the options.

How Go Shapes Bifrost

Bifrost uses Go's concurrency model directly on the request path. Each provider gets an independent pool of worker goroutines, so a slow provider stays isolated from others. Object pools built on sync.Pool are warmed at startup to reduce garbage collection pressure, and concurrency and buffer size can be tuned per provider.

• Provider isolation: independent worker pools per provider, with channels for async operations.
• Memory: pre-allocated pools with defaults suited to deployments of up to about 5,000 RPS.
• Overhead: 11 microseconds of internal overhead at 5,000 RPS on a t3.xlarge, a separate measure from the client-side figures above.
• Extensibility: custom plugins written in Go, using the same language and tooling as the gateway.

The Bifrost LLM gateway exposes one OpenAI-compatible API to 10,000+ models across 25+ providers.

What to Evaluate Instead of the Language

Buyers should treat language as an input to a gateway's behavior, not a selection criterion. A well-engineered Go gateway and a well-engineered Rust gateway land within a few milliseconds of each other at moderate concurrency, so product fit, reliability, and tail latency separate them more than runtime efficiency does.

Figure 3: Language runtime efficiency matters most at the bottom of the list, once product fit, reliability, and tail latency are settled.

Bifrost covers those areas with virtual keys, automatic fallbacks, and built-in observability.

Clustering is available in Bifrost Enterprise, and the Enterprise overview covers deployment and compliance.

The LLM gateway buyer's guide expands these criteria. For a broader market view, see how to weigh a LiteLLM replacement and the feature-by-feature comparison.

Frequently Asked Questions

Is Rust better than Go for an AI gateway?

Neither language is better for every AI gateway. In LiteLLM's benchmark, Rust used less CPU and usually less memory, while latency was mixed and throughput was capped by the test design. Rust suits teams that prioritize footprint. Go suits teams that value fast iteration and a simple concurrency model. Engineering quality and feature coverage affect results more than language.

Is Rust faster than Go?

Rust is often faster than Go in CPU-bound work, but this gateway benchmark did not show a consistent winner. In the sweep, Go had lower p99 at concurrency 1 and 64, and Rust at 4 and 16, while a separate concurrency 16 run favored Rust by about 4 ms. Gateways are mostly I/O-bound, so median differences stayed under 1 ms.

Does Go's garbage collector hurt gateway latency?

Go's garbage collector does most of its work concurrently with the application, with brief pauses, and its effect depends heavily on allocation rate. This benchmark does not isolate it. Go's p99 at concurrency 64 was lower than the Rust gateway's in the sweep, so this single run does not show collection as a deciding factor. Object pooling, as Bifrost uses, reduces allocation pressure.

Why is Rust so fast?

Rust compiles to native code, has no garbage collector, and resolves memory ownership at compile time. Its abstractions add little runtime cost, which gives it predictable memory use. For network services, those advantages show up most clearly in footprint and CPU. Waiting on a remote model provider still dominates end-to-end time for an AI gateway.

Is LiteLLM written in Rust?

LiteLLM is mainly a Python project with a Rust core in beta. As of November 3, 2026, its documentation says the Python proxy on main:latest serves /ocr and Bedrock /audio/transcriptions on Rust, while Messages and Chat Completions run on Python. Auth, routing, logging, and spend tracking stay with Python for now.

Test Rust vs Go Gateways on Your Own Traffic

The Rust vs Go question for an AI gateway is best answered with your own routes, concurrency, and controls. Run a saturation test with logging and budgets on, then compare p99, not averages. To run that test on the Bifrost AI gateway, start with the gateway quickstart or book a demo with the Bifrost team.