Best HIPAA-Compliant LLM Gateways

TL;DR

• A HIPAA-compliant LLM gateway enforces the Security Rule's technical safeguards on every request that carries PHI and routes that PHI only to models covered by a BAA or hosted in your own environment.
• No LLM gateway is "HIPAA certified"; compliance comes from BAAs across the data path plus how the gateway is deployed and configured.
• Bifrost runs inside the organization's own VPC or data center, so Maxim AI never receives prompts, responses, or telemetry, and one policy covers model, MCP tool, and agent traffic.
• Azure API Management and Google Apigee fall under their cloud's BAA, Kong suits teams already on its API platform, and Cloudflare AI Gateway is a hosted option that requires a Cloudflare Enterprise BAA.
• PHI also persists after the model call: request logs, trace exports, response caches, and fallback providers without a BAA each need their own controls.

Every LLM request that sends protected health information (PHI) to a hosted model is a disclosure to a business associate, so a HIPAA-compliant LLM deployment depends on controlling which model receives that data, who sent it, and where copies of it are kept. Bifrost, the open-source AI gateway written in Go and built by Maxim AI, is the best choice for healthcare enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability, and it runs entirely inside the organization's own infrastructure. This guide compares five LLM gateways on the controls HIPAA actually asks for: BAA-scoped routing, PHI redaction, access control, audit trails, and deployment model.

What Is a HIPAA-Compliant LLM Gateway?

A HIPAA-compliant LLM gateway is a control layer between healthcare applications and language models that enforces HIPAA's technical safeguards on every request carrying PHI. It authenticates each caller, routes PHI only to BAA-covered or self-hosted models, detects and redacts sensitive data, and keeps an audit trail inside the covered entity's environment.

HIPAA compliance is not a product label. The HHS summary of the HIPAA Security Rule requires administrative, physical, and technical safeguards for electronic PHI, which a gateway can support but not satisfy on its own. The broader HIPAA requirements for LLM applications extend past the gateway, though it is the one place most technical controls for AI traffic can be enforced together.

Deployment model decides who else enters the BAA chain. HHS guidance on HIPAA and cloud computing treats a cloud provider that creates, receives, maintains, or transmits ePHI as a business associate, even when it cannot decrypt the data, so a hosted gateway that sees PHI needs its own BAA. A self-hosted gateway such as the Bifrost AI gateway keeps the vendor out of the data path.

The technical safeguards in 45 CFR 164.312 map directly to gateway controls:

Figure 1: The gateway is the single place where provider scope, PHI guardrails, and logging policy apply to every clinical AI workload.

Key Criteria for Evaluating HIPAA-Compliant AI Tools

HIPAA-compliant AI tools for LLM traffic should be judged on where PHI can travel and where it can persist, not on a compliance badge. The criteria below cover the data path end to end, from the caller's identity to the last log line, and each maps to a question a security reviewer will ask before PHI goes live.

The LLM gateway buyer's guide covers general routing, performance, and cost criteria, and the roundup of self-hosted AI gateways compares options for teams that have ruled out hosted services.

HIPAA-Compliant LLM Gateways Compared at a Glance

The five gateways differ most on deployment model, which decides whether the gateway vendor joins the BAA chain, and on how they treat PHI in logs. Bifrost and Kong can run entirely in your infrastructure, Azure API Management and Apigee rely on their cloud's BAA and a vendor-hosted management plane, and Cloudflare AI Gateway is hosted only.

1. Bifrost

The Bifrost AI gateway routes, governs, and secures model calls, MCP tool calls, and agent-to-agent traffic for 10,000+ models across 25+ providers through one OpenAI-compatible API. For PHI workloads, the deciding facts are deployment and custody: Bifrost runs inside the customer's infrastructure, and Maxim AI never receives its prompts, responses, files, or telemetry.

Best for: Bifrost is built for enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with strong security, policy enforcement, and governance capabilities.

Bifrost's enterprise compliance program covers SOC 2 Type II, ISO 27001, HIPAA, and GDPR, and Maxim AI never takes custody of customer data. The features that matter for HIPAA:

• Self-hosted and air-gapped deployment. Bifrost ships as a single binary, with in-VPC deployments on AWS, GCP, and Azure and a documented air-gapped mode that loads pricing data from local files.
• BAA-scoped routing. A virtual key with no provider configs blocks all providers by default, so a PHI key can be limited to BAA-covered models, and CEL-based routing rules can route by header, team, or customer.
• PHI guardrails on requests and responses. Bifrost guardrails combine built-in Custom Regex, Secrets Detection, and Prompt Guardrails with external providers such as Microsoft Presidio, Azure AI Language PII, AWS Bedrock Guardrails, and Singulr AI, which enforces PII and PHI policies.
• Redaction for live traffic and logs. Detectors that support redaction can rewrite values at runtime, in logs only, or reversibly, and the PII Detection template covers email, US phone, US SSN, and card-like numbers as a base for MRN and member ID patterns.
• Content logging controls. Content logging can be turned off globally or per virtual key, so a PHI workload keeps cost, latency, and attribution metadata without storing prompts.
• Identity and least privilege. Enterprise deployments can require virtual key authentication on every inference call, sync users through OIDC and SCIM 2.0, and scope log visibility with RBAC and data access control.
• Audit and secrets. Signed audit logs record administrative changes with retention and S3 or GCS archival, and secret management keeps provider keys in an external vault such as AWS Secrets Manager or HashiCorp Vault.
• Tool and agent traffic under the same policy. Guardrails apply to MCP tool arguments and results, and MCP tool filtering limits which EHR or scheduling tools each key can call.

Bifrost adds about 11 microseconds of overhead per request at 5,000 RPS on a single t3.xlarge in published benchmarks; guardrails that call external providers add their own network time. The AI Guardrails page covers the full detector lineup.

Figure 2: Redaction and blocking run before the provider call, and the log write follows the per-key content setting.

2. Kong AI Gateway

Kong AI Gateway adds AI plugins to Kong's API gateway for multi-provider proxying, load balancing, and prompt controls. Its data plane can run in your environment under Kong Konnect, or the full stack can be self-hosted.

Best for: Teams that already run Kong for API management and want AI traffic on the same platform.

• Routing. The AI Proxy plugin supports 17 providers, including Azure OpenAI, Amazon Bedrock, Anthropic, and Vertex AI, and AI Proxy Advanced (Enterprise) adds priority-based tiered failover.
• PHI controls. The PII Sanitizer plugin, part of the AI Gateway Enterprise offering, redacts sensitive data in requests and responses through a separate PII service container.
• Logging. Payload logging is off by default; enabling it also places prompts in OpenTelemetry traces, and sanitizer audit logs can include the original text of each detected entity.

Kong's public documentation does not state HIPAA eligibility or BAA terms for AI Gateway, so the HIPAA posture rests on keeping the data plane, logs, and traces inside your environment. The comparison of AI gateways for regulated industries covers these trade-offs.

3. Azure API Management

Azure API Management adds AI gateway policies to Microsoft's API management service, including token limits, load balancing, and content safety checks for model APIs.

Best for: Healthcare organizations standardized on Azure that already run Azure OpenAI under Microsoft's BAA.

• BAA. Microsoft offers its HIPAA BAA through the Product Terms for in-scope Azure services, with no separate contract to sign; confirm API Management on Microsoft's audit-scope list.
• Routing. Round-robin, weighted, priority-based, and session-aware load balancing come with a circuit breaker and token limits per subscription.
• PHI controls. The content safety policy sends prompts and completions to Azure AI Content Safety for harm categories and prompt shields; the AI gateway documentation does not describe PII redaction.
• Logging. Prompt and completion logging to Log Analytics is optional, with a 2 MB limit per message.

A self-hosted gateway container on Developer and Premium tiers still requires outbound connectivity to Azure; the review of LLM gateways for healthcare, financial services, and government compares it with other options.

4. Google Apigee

Apigee is Google Cloud's API management platform, with AI policies for token quotas, prompt sanitization through Model Armor, and model routing.

Best for: Google Cloud customers that want LLM traffic governed alongside existing Apigee APIs.

• BAA. Apigee and Model Armor both appear on Google Cloud's HIPAA covered-products list, and Google advises against using pre-GA offerings with PHI.
• PHI controls. The SanitizeUserPrompt policy evaluates prompts with Model Armor, which can redact PII through Sensitive Data Protection, and Apigee then allows, blocks, or redacts the request or response.
• Access. LLMTokenQuota enforces token quotas per API product, developer, or app, though the token-policy tutorial excludes Apigee hybrid.

Apigee runs as a Google-managed service or as a hybrid runtime on your Kubernetes clusters with a Google-hosted management plane. Debug-session data masking is off by default, so set PHI masking rules before tracing production traffic; the AI security resource hub covers related LLM risks.

5. Cloudflare AI Gateway

Cloudflare AI Gateway is a hosted proxy on Cloudflare's network that adds logging, rate limiting, data loss prevention, and fallbacks in front of model providers.

Best for: Teams on Cloudflare Enterprise that want a hosted gateway and accept a third-party data path for PHI.

• BAA. Cloudflare enters into a BAA with Enterprise customers whose use involves PHI, and AI Gateway is listed in the scope of its HIPAA attestation.
• PHI controls. Data loss prevention scans prompts and responses for sensitive information and can flag or block them; the DLP documentation does not describe redaction.
• Logging. Logs are enabled by default and can include the prompt and response, and a per-request header skips payload storage.
• Access. Rate limits count requests, not tokens, and AI Gateway token permissions cannot be scoped to one gateway.

Because Cloudflare hosts the gateway, PHI transits Cloudflare's network, which makes Cloudflare a business associate for that traffic. Teams that need PHI to stay on their own network need a self-hosted LLM gateway instead.

Which LLM Providers Sign a HIPAA BAA?

A HIPAA BAA has to cover every model provider that can receive PHI, including the providers a gateway fails over to. The major model APIs offer BAAs, but each limits coverage to specific services and features, so the gateway's routing configuration should mirror the BAA scope rather than the full model catalog.

OpenAI's guidance on its BAA for the API states that signing one does not by itself make an application HIPAA compliant. Bifrost connects to all of these through one API, and its supported providers include self-hosted runtimes such as vLLM and Ollama for PHI that should not leave the network.

The fallback chain is an easy place for BAA scope to break. Retries and fallbacks keep clinical applications available during provider incidents, and scoping the PHI virtual key to BAA-covered providers keeps every fallback inside the agreement.

Figure 3: Failover is a disclosure path, so the fallback provider needs the same BAA coverage as the primary.

HIPAA and AI: Where PHI Leaks After the Model Call

In HIPAA and AI reviews, the model provider is only one of several places PHI lands. PHI persists in request logs, object storage, observability exports, and response caches. Each one is an ePHI store with its own access and retention requirements.

• Request logs. A log row with full prompt content is ePHI; turning content logging off for the PHI key keeps the metadata dashboards need.
• Payload exports. Bifrost offloads payloads to S3 or GCS through log exports, and the retention sweep does not delete bucket objects, so the bucket needs its own lifecycle rule.
• Trace connectors. OpenTelemetry, Datadog, BigQuery, and Kafka connectors each carry their own content flag, so a disabled log store says nothing about what a trace backend receives.
• Response caches. Semantic caching engages only when a request carries a cache key or a default key is configured, and entries are scoped by default to the identity that wrote them.
• Shadow AI. Clinicians pasting notes into consumer chat apps bypass every control above. The Bifrost Gateway stays the policy engine, and Bifrost Edge extends those same guardrails and audit logs to desktop apps, browser AI, and coding agents on managed devices.

Figure 4: Turning off content logging in one destination does not change the others, so each one needs its own PHI decision.

Redaction is not de-identification: the HHS de-identification guidance requires Safe Harbor removal of 18 identifier types or an expert determination, so treat gateway redaction as minimum necessary enforcement on data that is still PHI. The guide to PII redaction at the gateway layer covers detector selection.

Frequently Asked Questions

Is ChatGPT HIPAA compliant?

ChatGPT can be used with PHI only under a signed BAA. OpenAI offers BAAs for the API Platform, for sales-managed ChatGPT Enterprise and Edu accounts, and for eligible clinicians through ChatGPT for Clinicians, and it states that it does not offer a BAA for ChatGPT Business. Applications should route PHI only to a BAA-covered API organization.

Is Claude HIPAA compliant?

Claude can be used with PHI when Anthropic has signed a BAA covering the specific service. Anthropic's BAA covers HIPAA-ready services such as the first-party Messages API and activated Enterprise plans, and excludes features such as the Batch API, Files API, and Console. A gateway in front of Claude should restrict PHI traffic to the covered endpoints.

Do I need a BAA with my LLM gateway vendor?

A BAA is required when the gateway vendor creates, receives, maintains, or transmits PHI, which is the case for hosted gateways. A self-hosted gateway such as Bifrost runs inside your infrastructure and sends no prompts, responses, or telemetry to its vendor, so the BAA chain runs to your cloud host and model providers instead.

Can an LLM gateway de-identify PHI under HIPAA Safe Harbor?

Not on its own. Safe Harbor requires removing all 18 identifier types, including names and all date elements except the year. Gateway redaction reduces the PHI sent to models and stored in logs, supporting the minimum necessary standard, but redacted traffic should still be treated as PHI under a BAA.

Does HIPAA apply to AI agents and MCP tool calls?

Yes. An agent that reads a chart through an MCP server and passes it to a model moves PHI through two systems, and both paths need access control and audit trails. Bifrost applies guardrails to MCP tool arguments and results and limits which tools each key can call, as covered in the governance resource hub.

What are the HIPAA requirements for LLM audit logs?

The Security Rule's audit controls standard requires recording and examining activity in systems that contain ePHI. For LLM traffic, that means attributable request records (who, which model, when) and a separate record of configuration changes. The HIPAA requirements guide for LLM applications maps each standard to implementation steps.

Getting Started with a HIPAA-Compliant LLM Gateway

A HIPAA-compliant LLM gateway gives healthcare teams one enforcement point for BAA-scoped routing, PHI guardrails, least-privilege access, and audit trails across every model, tool, and agent. Bifrost delivers those controls from inside your own infrastructure, with AI governance built on your identity provider and a healthcare and life sciences deployment that never hands PHI to the gateway vendor. To see Bifrost handle PHI traffic in your environment, book a demo with the Bifrost team.