Best AI Gateways for Google Cloud and Vertex AI

TL;DR

• A GCP AI gateway sits between your applications and Vertex AI, the Gemini API, or other model providers, enforcing identity, budgets, routing, and logging on every model call.
• Bifrost runs on GKE through Helm, calls Vertex AI with Workload Identity instead of key files, and reaches 10,000+ models across 25+ providers through one OpenAI-compatible API.
• Apigee is Google's managed option: token quotas, semantic caching, and Model Armor policies need its Intermediate or Comprehensive environments, which list at $1,460 and $3,431 per month per region on pay-as-you-go.
• Kong AI Gateway 2.x and Agent Router (formerly Envoy AI Gateway) both run on GKE and authenticate to Vertex AI with GKE workload identity; Cloudflare AI Gateway is hosted outside Google Cloud.
• Google's Agent Gateway is still in Private Preview with LLM token policies "coming soon", and GKE Inference Gateway balances self-hosted model servers rather than the Vertex AI API.

A GCP AI gateway is the control point inside your Google Cloud environment that every model call passes through, whether it goes to Gemini or Claude on Vertex AI, to the Gemini API, or to a provider outside Google. Bifrost, the open source AI gateway built in Go by Maxim AI, is the best choice for enterprises running mission-critical AI workloads on Google Cloud that require top-tier performance, scalability, and reliability. This guide compares it with Apigee, Kong AI Gateway, Agent Router, and Cloudflare AI Gateway, plus two Google gateways that solve adjacent problems, using facts from each vendor's documentation read on November 11, 2026.

A note on naming: Google now markets Vertex AI as part of Gemini Enterprise Agent Platform, and cloud.google.com/vertex-ai redirects there. The API endpoint is still aiplatform.googleapis.com, and most gateway documentation still says Vertex AI, so this guide does too.

What a GCP AI Gateway Does

An AI gateway is a proxy layer built for model traffic. On Google Cloud it typically handles five jobs:

• Identity: callers authenticate to the gateway, and the gateway authenticates to Vertex AI with a Google service account, so applications never hold Google credentials.
• Routing and failover: requests go to the right model, region, or provider, and move to another when one returns a 429 or a 5xx.
• Cost control: budgets and token limits per team, project, or application, enforced before the request reaches the provider.
• Guardrails: prompts and responses are checked for injection, unsafe content, and sensitive data, often with Model Armor.
• Observability: every request is logged with model, tokens, cost, and latency, ideally exported to BigQuery or Cloud Logging.

The complete guide to AI gateways for enterprises covers these functions in general. The rest of this guide is about how they work inside Google Cloud.

Figure 1: Every model call passes one gateway inside your project, so identity, budgets, guardrails, and logs apply before traffic reaches Vertex AI or any other provider.

Key Criteria for Choosing an AI Gateway on Google Cloud

Five questions separate the options:

1. Where does it run? Inside your project on GKE or Cloud Run, on Google's managed infrastructure, or on a third party's network. This decides whether prompts leave your Google Cloud environment.
2. How does it authenticate to Vertex AI? Keyless Workload Identity on GKE avoids storing service account JSON files that need rotation.
3. Which providers does it cover? Vertex AI only, Vertex AI plus the Gemini API, or every major provider, including OpenAI, Anthropic, and Amazon Bedrock.
4. What does it govern? Token budgets and rate limits per team, guardrails, and whether MCP tool calls and agent traffic go through the same policies.
5. What does it cost? Open source, per-environment pricing, per-model pricing, or usage-based.

With Workload Identity, the gateway's Kubernetes service account maps to a Google service account that holds the Vertex AI User role:

Figure 2: With Workload Identity Federation for GKE, the gateway calls Vertex AI with short-lived tokens and no service account key files to store or rotate.

Teams comparing clouds can read the sibling guide to AI gateways for AWS deployments, which applies the same criteria to Amazon Bedrock.

GCP AI Gateways Compared at a Glance

Bifrost Enterprise pricing is on the Bifrost pricing page; the open source core is free under Apache 2.0.

1. Bifrost

Bifrost is a high-performance, open source AI gateway that governs model calls, MCP tool calls, and agent-to-agent traffic as one gateway. It runs inside your own infrastructure, adds about 11 microseconds of overhead per request at 5,000 requests per second in published benchmarks, and exposes 10,000+ models across 25+ providers through one OpenAI-compatible API. Its LLM gateway page summarizes the feature set.

Deployment on Google Cloud. The GKE deployment guide installs Bifrost with Helm as a ClusterIP service on linux/amd64 or linux/arm64 nodes, backed by PostgreSQL 16 or later (or SQLite on a single open source replica). Bifrost Enterprise images are distributed through GCP Artifact Registry and pulled with Workload Identity. Cloud Run is supported at Preview level for single-instance deployments with PostgreSQL; Enterprise mesh clustering is not supported on Cloud Run services, so clustered deployments use GKE.

Vertex AI authentication. The Vertex AI provider supports four methods:

• Service account JSON, including impersonated and external_account credentials.
• Application Default Credentials, which resolve to the GKE, Cloud Run, or Compute Engine metadata server, so an attached service account or Workload Identity works with no key file.
• API key, for Gemini and fine-tuned Gemini models only.
• AWS Workload Identity Federation, for a Bifrost deployment on AWS (EKS, ECS, or EC2) calling Vertex AI.

Vertex AI coverage. One Vertex key serves both Gemini and Claude models, with Bifrost converting requests to each model's format. Chat completions, the Responses API, embeddings, Imagen image generation, and Veo video generation are supported. The global endpoint works, and for Claude models published only on Google's multi-region endpoints, Bifrost promotes a single-region request to the matching pool endpoint automatically. Teams using Google's own SDKs can point the GenAI SDK at Bifrost's /genai endpoint, and Gemini CLI runs through Bifrost with a virtual key.

Governance and reliability. Virtual keys carry budgets, rate limits, and model allowances per team or application. With retries configured, when a Vertex key returns a 429, Bifrost rotates to another key with backoff; a 5xx retries the same key; and a fallback provider serves when every attempt fails.

Figure 3: With retries configured, a 429 rotates to another Vertex key with backoff, a 5xx retries the same key, and a configured fallback provider serves when every attempt on Vertex fails.

Google Cloud integrations (Enterprise). Bifrost Enterprise connects to the Google services a platform team already runs:

• Google Model Armor as a guardrail provider, with Sensitive Data Protection de-identification applied inline.
• A BigQuery plugin that writes one row per request with tokens, cost, latency, and team attribution, plus a Pub/Sub connector for custom pipelines.
• GCP Secret Manager for provider keys, and Google Workspace for SSO and SCIM provisioning.

Best for: Bifrost is built for enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with strong security, policy enforcement, and governance capabilities.

The Vertex AI provider guide walks through setup on Google Cloud.

2. Apigee

Apigee is Google Cloud's API management platform, and its AI gateway capabilities are a set of policies and templates rather than a separate product. It is the most Google-native option and the one Google's own search results lead with.

What it does for LLM traffic.

• Token controls: PromptTokenLimit throttles tokens per prompt, and LLMTokenQuota enforces token quotas per API product with calendar or rolling windows, including on streaming responses. Both count text tokens only.
• Semantic caching: SemanticCacheLookup and SemanticCachePopulate, backed by Vector Search.
• Model Armor policies for prompt injection and harmful content.
• Multi-model routing and fallback through Feature Templates; Google's example routes between Vertex AI and OpenAI.
• MCP: existing APIs can be exposed as MCP tools.

Gotchas. The token, semantic cache, and Model Armor policies are Extensible policies that deploy only in Intermediate or Comprehensive environments, and Google's end-to-end AI gateway tutorial requires Comprehensive. On pay-as-you-go, those environments list at $1,460 and $3,431 per month per region, with Extensible proxy calls at $100 per million at the lowest volume tier. The self-managed Apigee hybrid runtime is available only on Subscription, and on hybrid, semantic caching and Model Armor work only for installations on Google Cloud. Google's own pages disagree on whether LLMTokenQuota applies to hybrid.

Best for: organizations that already run Apigee for API management and want AI policies in the same console. Teams weighing alternatives can compare Apigee AI gateway alternatives.

3. Kong AI Gateway

Kong AI Gateway extends Kong's API gateway for LLM traffic. Version 2.x replaces the AI Proxy plugins with "AI Model Provider" and "AI Model" entities managed from Kong Konnect.

Vertex AI support. Kong's Gemini provider covers both the Gemini API and the Vertex AI endpoint, which Kong labels "Gemini Enterprise". Vertex calls use GCP credentials resolved in order from a service account JSON, the GCP_SERVICE_ACCOUNT environment variable, the workload's attached identity (such as a GKE service account), or the VM's role.

Deployment on Google Cloud. Data planes run in your environment, including Kubernetes, under a Konnect control plane. Kong's Dedicated Cloud Gateways run on GCP, but its reference notes that AI Gateway 2.0 on Dedicated Cloud Gateways supports only AWS and Azure, so on Google Cloud you host the AI Gateway data plane yourself, for example on GKE.

Gotchas. AI Proxy Advanced, AI Semantic Cache, AI Rate Limiting Advanced, and the AI GCP Model Armor plugin are AI Gateway Enterprise features. Kong's pricing page lists the Plus plan from $25 per month plus usage, and an AI Gateway model proxy charge of $100 per month for each unique model proxied.

Best for: teams that already standardize on Kong for API traffic. The Kong AI Gateway alternatives comparison covers where it fits less well.

4. Agent Router (Formerly Envoy AI Gateway)

Envoy AI Gateway became Agent Router, an Agentic AI Foundation project, in September 2026. The code, maintainers, and Apache 2.0 license are unchanged, as are the CRDs, the aigw CLI, and the Helm charts; v1.2.1 shipped on November 6, 2026.

Vertex AI support. Agent Router's Vertex AI guide supports Gemini on Vertex AI and Claude on Vertex AI as separate backend schemas. It recommends Application Default Credentials with GKE Workload Identity, and also supports service account key files and Workload Identity Federation.

Deployment and features. It is Kubernetes-native, built on Envoy Gateway and the Gateway API, so it fits GKE clusters that already use them. It routes to 22 providers, offers token limits per team, app, or model, and includes an MCP gateway. It also supports the InferencePool API for self-hosted models, the same API GKE Inference Gateway uses.

Best for: Kubernetes teams that want an Apache 2.0 gateway configured through CRDs and are comfortable assembling governance, dashboards, and budgets from Kubernetes resources. The guide to AI gateways for Kubernetes and Helm compares it with other cluster-native options.

5. Cloudflare AI Gateway

Cloudflare AI Gateway is a hosted gateway on Cloudflare's network. Requests go to gateway.ai.cloudflare.com, so traffic leaves your Google Cloud project before reaching Vertex AI.

Vertex AI support. Cloudflare supports Vertex AI through a provider-specific endpoint and its unified OpenAI-compatible endpoint. Authentication uses a service account JSON stored in Cloudflare (the recommended option), a base64-encoded service account in each request, or a gcloud access token. The Gemini API works with an API key.

Gotchas. Core features, including analytics, caching, and rate limiting, are free. Caching matches identical requests only; semantic caching is not available. Accounts whose first gateway was created on or after September 24, 2026 store logs in Workers Logs, with 200,000 events per day and 3-day retention on the free plan.

Best for: teams that accept a hosted gateway outside Google Cloud and want quick analytics and caching with little setup. See the Cloudflare AI Gateway alternatives for self-hosted options.

Google's Agent Gateway and GKE Inference Gateway

Two Google products carry "gateway" in their names but solve different problems:

• Agent Gateway (Gemini Enterprise Agent Platform) governs agent traffic: user-to-agent, agent-to-MCP-tool, and agent-to-agent calls over MCP, A2A, REST, and gRPC, with Agent Identity, Agent Registry, and Model Armor. Google announced it in Private Preview in April 2026. Its token management policies for LLM traffic are marked "coming soon", and today it runs only Google-managed for agents on Agent Runtime and Gemini Enterprise; GKE, Cloud Run, and cross-cloud deployments are early-preview sign-ups.
• GKE Inference Gateway has been generally available since December 2025 at no additional cost. It load-balances model servers you host in your own GKE cluster, using KV cache, queue length, and accelerator utilization, with prefix-cache-aware routing and LoRA serving. Its documentation does not describe fronting the Vertex AI API or external providers.

A team serving open models on GKE can use GKE Inference Gateway behind a multi-provider gateway, so self-hosted and Vertex AI models share one set of keys and budgets. Bifrost can route to self-hosted models through its vLLM and SGLang providers alongside Vertex AI. For agent traffic, Bifrost already governs MCP tool calls and agent-to-agent calls in the same gateway.

How to Choose the Right AI Gateway for Google Cloud

Start from the governance you need across providers, then let the platform you already run decide the rest:

• Need budgets and audit logs across Vertex AI and other providers, inside your project: Bifrost on GKE.
• Already run Apigee for API management: Apigee's AI policies, budgeted for Intermediate or Comprehensive environments.
• Standardized on Kong or on Kubernetes Gateway API resources: Kong AI Gateway or Agent Router on GKE.
• Serving only your own models on GKE: GKE Inference Gateway.
• Fine with prompts leaving Google Cloud: Cloudflare AI Gateway.

Figure 4: Start from the governance you need across providers; the platform you already run on Google Cloud decides the rest.

Coding agents are a common reason to add a gateway on Google Cloud. Running Claude Code on Vertex AI through a gateway and routing Gemini CLI and Codex through a gateway both put spend per developer under one budget.

Frequently Asked Questions

Does Google Cloud have an AI gateway?

Yes. Apigee includes AI gateway capabilities: token quotas, semantic caching, Model Armor policies, and multi-model routing, mostly in its Intermediate and Comprehensive environments. Google's Agent Gateway governs agent and MCP traffic but is in Private Preview, and GKE Inference Gateway balances self-hosted model servers. Teams that want a self-hosted, multi-provider GCP AI gateway can run Bifrost on GKE instead.

Can an AI gateway call Vertex AI without service account keys?

Yes. On GKE, Workload Identity Federation maps the gateway's Kubernetes service account to a Google service account with the Vertex AI User role, and the gateway gets short-lived tokens from the metadata server. Bifrost supports this through Application Default Credentials, and the guide to using Vertex AI models through Bifrost shows the configuration.

Can I use Claude models on Vertex AI through a gateway?

Yes. Bifrost, Kong AI Gateway, Agent Router, and Cloudflare AI Gateway all support Vertex AI, and Bifrost and Agent Router document Claude on Vertex AI explicitly. In Bifrost, one Vertex key serves both Gemini and Claude models, and requests are converted to each model's native format.

How do I handle Vertex AI 429 errors?

Configure several Vertex AI keys, for example in different projects or regions, and let the gateway rotate between them. With retries configured, Bifrost rotates to another key with backoff on a 429, retries the same key on a 5xx, and falls back to another provider, such as the Gemini API or Anthropic, when every Vertex attempt fails.

Is Apigee an AI gateway?

Apigee is an API management platform with AI gateway capabilities added as policies. Its token quota, semantic cache, and Model Armor policies are Extensible policies available in Intermediate or Comprehensive environments, and multi-model routing is built from Feature Templates. It suits teams already on Apigee; teams starting fresh may prefer a purpose-built gateway.

How do I track Vertex AI costs per team?

Put every team behind its own gateway credential and attribute each request's tokens and cost to it. In Bifrost, virtual keys carry budgets and rate limits per team, and the Enterprise BigQuery plugin writes one row per request with cost and team attribution, so spend can be queried in SQL next to the rest of your billing data.

Deploy Bifrost as Your GCP AI Gateway

Bifrost gives Google Cloud teams one self-hosted GCP AI gateway for Vertex AI, the Gemini API, and every other provider, running inside your own project: keyless Workload Identity on GKE and budgets per team, plus Model Armor guardrails and BigQuery logs on Bifrost Enterprise. The complete guide to AI gateways explains how a gateway fits an enterprise platform, and the enterprise deployment page covers in-VPC options. To see Bifrost on your own Google Cloud traffic, book a Bifrost demo.