Try Bifrost Enterprise free for 14 days. Request access

9 Best Open Source LLMs to Self-Host

The best open source LLM to self-host is the one whose license, hardware footprint, and runtime support fit the deployment. This guide ranks nine open-weight models, including Qwen3.8, DeepSeek V4.1, GLM-5.3, and gpt-oss, and shows how to serve them behind Bifrost.

9 Best Open Source LLMs to Self-Host

TL;DR

  • As of November 2026, the best open source LLMs to self-host are Qwen3.8, DeepSeek V4.1, GLM-5.3, Kimi K3.1 Pro, gpt-oss, Gemma 4, Mistral Small 4 and Large 4, NVIDIA Nemotron 3.5, and MiniMax M3.
  • Licenses separate these models more than benchmarks: Qwen3.8-27B, gpt-oss, Gemma 4, and Mistral Small 4 use Apache 2.0, DeepSeek V4.1 uses MIT, and Kimi, MiniMax, GLM-5.3, and the largest Qwen and Mistral releases add hosted-service, revenue, or attribution conditions.
  • Hardware sets the shortlist: gpt-oss-20b runs within 16 GB of memory, gpt-oss-120b fits one 80 GB GPU, and 2.8-trillion-parameter Kimi K3.1 Pro needs a multi-GPU node.
  • Bifrost supports vLLM, SGLang, and Ollama as built-in providers, so every self-hosted model sits behind one OpenAI-compatible API with fallbacks, virtual keys, and request logs.

Every model in this guide publishes its weights under a license that lets teams run it on their own GPUs, inside a VPC, a data center, or a disconnected network. Choosing the best open source LLM to self-host comes down to license terms, hardware footprint, and runtime support more than leaderboard rank. Bifrost, the open-source AI gateway built by Maxim AI, is the best choice for enterprises running mission-critical AI workloads that require top-tier performance, scalability, and reliability, and it puts one OpenAI-compatible API in front of those models. This guide ranks nine open source LLMs as of November 2026, then shows how to serve them through Bifrost alongside cloud providers.

What Is a Self-Hosted LLM?

A self-hosted LLM is a large language model whose weights run on infrastructure the organization controls, served by an inference runtime such as vLLM, SGLang, or Ollama instead of a vendor's API. Prompts, responses, and logs stay inside that network, and the team decides which model version runs.

A self-hosted deployment has three layers, shown in Figure 1: the open-weight model files, the runtime that serves them over an HTTP API, and a gateway that controls which applications reach which model. Most open source LLMs are more precisely open-weight models, since they publish weights but not everything needed to rebuild them; the Open Source AI Definition sets out what a fully open release includes.

Applications call the Bifrost AI gateway, which routes to vLLM, SGLang, or Ollama serving open-weight models, with cloud providers as fallback

Figure 1: The model is one layer of a self-hosted LLM deployment; the runtime serves it and the gateway decides who can reach it.

Our reference architecture for on-premise LLM deployment covers the full stack; this guide focuses on the model layer. The Bifrost AI gateway runs as a single binary inside the same network, and no prompts, responses, or telemetry leave it under its in-VPC and on-prem deployment model.

How to Choose the Best Open Source LLM to Self-Host

Choose an open source LLM to self-host by checking five things in order: whether its license permits the intended use, whether its weights fit the available GPUs, whether your runtime supports it, whether its context window covers the workload, and how it performs on your own evaluation set. Vendor benchmarks come last because each developer uses its own evaluation setup.

Criterion What to check Why it matters
License Apache 2.0, MIT, or custom revenue, hosted-service, or attribution clauses Custom terms can require an agreement before serving customers
Hardware footprint Total and active parameters, quantized checkpoints Total parameters set memory; active parameters set speed
Runtime support vLLM recipes, SGLang cookbooks, Ollama builds A model the runtime cannot load is not deployable
Context window Native length and extension method Agent workloads consume context quickly
Task quality Your own prompts, tool calls, and failure cases Published scores use different settings

Teams that do not need weights on their own hardware can use hosted APIs instead, as our comparison of LLM API providers covers. The guide to running an open-weight model in production without code changes explains open weight versus open source.

Open Source LLMs Compared at a Glance

The nine open source LLMs below range from on-device models with a few billion parameters to 2.8-trillion-parameter mixtures of experts. Sizes, context windows, and licenses come from each official model card as of November 2026, and the last column is the prefix used to call each model through the Bifrost AI gateway.

Each prefix maps to a built-in provider: vLLM, SGLang, or Ollama.

Model family Developer Sizes (total / active) Context License Bifrost prefix
Qwen3.8 Qwen (Alibaba) 27B dense; 125B / 6B (+51B n-gram); 2.4T / 95B 262K, extends to 1M Apache 2.0 (27B); Qwen licenses vllm/, sgl/, ollama/
DeepSeek V4.1 DeepSeek 552B / 8B-16B; V4-Pro 1.6T / 49B 1M MIT vllm/, sgl/
GLM-5.3 Z.ai Flash 320B / 18B; flagship 1M MIT (Flash); GLM-5.3 License vllm/, sgl/
Kimi K3.1 Pro Moonshot AI 2.8T / 104B 1M Kimi K3.1 Pro License vllm/, sgl/
gpt-oss OpenAI 117B / 5.1B; 21B / 3.6B 128K Apache 2.0 vllm/, ollama/
Gemma 4 Google DeepMind E2B, E4B, 12B, 26B / 3.8B, 31B 128K-256K Apache 2.0 ollama/
Mistral Small 4, Large 4 Mistral AI 119B / 6.5B; ~1T / ~52B 256K; 1M Apache 2.0; Mistral Ecosystem License vllm/, sgl/
Nemotron 3.5 NVIDIA 120B / 12B; 74B / 9B; 8B 1M; 256K; 64K OpenMDW-1.1 vllm/, sgl/, ollama/
MiniMax M3 MiniMax ~428B / ~23B 1M MiniMax Community License vllm/, sgl/

The 9 Best Open Source LLMs to Self-Host in 2026

These nine models are ranked for self-hosting by license terms, hardware range, runtime support, and the coding and agent results their developers publish. Facts come from each official Hugging Face card as of November 2026.

1. Qwen3.8

Qwen3.8, released in August 2026, spans the widest range of model sizes on this list. Qwen3.8-27B is a dense multimodal model under Apache 2.0 with a 262,144-token native context that extends to about 1 million tokens. Qwen3.8-Flash-Next is a 125B mixture of experts (plus 51B of n-gram embeddings) with 6B activated parameters, and Qwen3.8-2.4T-A95B has 2.4T total and 95B activated.

  • Runtimes: vLLM, SGLang, and TokenSpeed recipes, plus local Ollama builds of the 27B model.
  • Choose it for: one family from a single GPU to a cluster, with tunable reasoning_effort.
  • Through Bifrost: call vllm/Qwen/Qwen3.8-27B after the vLLM provider guide.

2. DeepSeek V4.1

DeepSeek-V4.1-Flash, published in September 2026, is a multimodal mixture of experts with 552B backbone parameters, a separate 196B Engram memory, 8B to 16B activated parameters, and a 1 million token context window. Its October update, V4.1-Flash-1017, improves reasoning, coding, and agentic results with shorter outputs. DeepSeek-V4-Pro-0813 is larger, with 1.6T backbone and 49B activated parameters.

  • License: MIT for code and weights.
  • Runtimes: the V4-Pro card documents vLLM and SGLang; V4.1 ships a Python prompt encoder instead of a Jinja chat template.
  • Through Bifrost: serve it with vLLM and keep an SGLang server as a second provider.

3. GLM-5.3

Z.ai released GLM-5.3 and GLM-5.3-Flash in August 2026. GLM-5.3-Flash is the first natively multimodal model in the GLM-5 series, with 320B total and 18B active parameters under MIT, and it mixes sparse and linear attention to cut long-context cost. GLM-5.3 is the coding-focused flagship, post-trained from the GLM-5.2 base.

  • License: MIT for Flash; the GLM-5.3 License adds conditions only for very large hosted-model operators.
  • Runtimes: SGLang, vLLM, and KTransformers, plus Ascend NPU support.
  • Through Bifrost: call sgl/zai-org/GLM-5.3-Flash from a virtual key reserved for coding tools.

4. Kimi K3.1 Pro

Moonshot AI published Kimi K3.1 Pro on November 2, 2026, post-trained from Kimi K3 (June 2026). Both are multimodal mixtures of experts with 2.8T total and 104B activated parameters, a 1 million token context window, and MXFP4 weights from quantization-aware training. At 4 bits, 2.8T parameters come to about 1.4 TB of weights, so this is a multi-GPU node deployment.

  • License: the Kimi K3.1 Pro License, a modified MIT with conditions for large hosted-model businesses.
  • Runtimes: vLLM recipes and an SGLang cookbook.
  • Through Bifrost: call vllm/moonshotai/Kimi-K3.1-Pro for long-horizon agent work.

5. gpt-oss

OpenAI released gpt-oss in August 2025, the oldest release on this list, and both sizes are built for single-GPU deployment. gpt-oss-120b has 117B parameters with 5.1B active and fits a single 80 GB GPU such as an H100 or MI300X, because its MoE weights were post-trained with MXFP4. gpt-oss-20b has 21B parameters with 3.6B active and runs within 16 GB of memory.

6. Gemma 4

Google DeepMind's Gemma 4, first released in March 2026, comes in five Apache 2.0 sizes: E2B and E4B for mobile and edge devices, a 12B encoder-free model, a 26B A4B mixture of experts with 3.8B active parameters, and a 31B dense model. The small models have 128K context and the larger three 256K. Every size accepts text and images, and E2B, E4B, and 12B also accept audio.

  • Runtimes: local Ollama builds, including 12B, 26B, and 31B, and llama.cpp.
  • Through Bifrost: call ollama/gemma4:26b after the Ollama provider setup.

7. Mistral Small 4 and Mistral Large 4

Mistral Small 4 is a 119B mixture of experts with 6.5B activated parameters, a 256K context window, and per-request reasoning_effort, under Apache 2.0. Mistral Large 4, published on September 30, 2026, scales the family to about 1T total and 52B active parameters with a 1M-token context, and an NVFP4 checkpoint targets Blackwell GPUs.

  • License: Large 4 uses the Mistral Ecosystem Open-Weight License, a modified MIT.
  • Runtimes: vLLM is recommended on both cards; SGLang is supported.
  • Through Bifrost: call vllm/mistralai/Mistral-Small-4-119B-2603.

8. NVIDIA Nemotron 3.5

NVIDIA released Nemotron 3.5 in October 2026 under OpenMDW-1.1, and its cards state the models are ready for commercial use. Nemotron 3.5 Super VL is a 120B hybrid Mamba-2 and Transformer vision-language mixture of experts with 12B active parameters and up to 1M tokens of context. Bolt 74B-A9B targets local agentic tasks, with a 24GB version using native 2-bit quantization, and Bolt E4B is an 8B on-device model.

  • Runtimes: vLLM and SGLang examples on the cards, plus local Ollama builds of Super and Bolt.
  • Through Bifrost: call vllm/nemotron-3.5-super-vl, the served name in the card's vLLM recipe.

9. MiniMax M3

MiniMax M3, published in June 2026, is a multimodal mixture of experts with about 428B total and 23B activated parameters and a 1M-token context window. Its card credits MiniMax Sparse Attention with 9x faster prefill and 15x faster decode than M2 at 1M context. MiniMax-M3.1-Flash, released in October 2026, keeps the same size and focuses on coding, agents, and computer use.

  • License: the MiniMax Community License, which requires attribution and notice for commercial use.
  • Runtimes: SGLang and vLLM.
  • Through Bifrost: call sgl/MiniMaxAI/MiniMax-M3 from a virtual key scoped to internal teams.

License Terms for Open-Weight Models

Open-weight model licenses fall into three groups: permissive licenses (Apache 2.0 and MIT), modified MIT licenses that add conditions for large hosted services, and community licenses that limit commercial use. Most conditions in the second group do not apply to internal self-hosting, but they matter once a model serves external customers.

Model License Main condition
Qwen3.8-27B, gpt-oss, Gemma 4, Mistral Small 4 Apache 2.0 Keep the license and notices
DeepSeek V4.1 and V4-Pro, GLM-5.3-Flash MIT Keep the copyright notice
Qwen3.8-Flash-Next, Qwen3.8-2.4T-A95B Qwen Community License 1.0; Qwen3.8-Max License Separate license for hosted-model businesses (above US$50M revenue for 2.4T)
GLM-5.3 GLM-5.3 License Security review above US$10B hosted-model revenue
Kimi K3, Kimi K3.1 Pro Kimi licenses (modified MIT) Agreement above US$20M hosted-model revenue; name display for very large products
Mistral Large 4 Mistral Ecosystem Open-Weight License Commercial license for hosted-model businesses
Nemotron 3.5 OpenMDW-1.1 Cards state commercial use is permitted
MiniMax M3 MiniMax Community License Attribution and notice for commercial use; authorization above US$20M

This table summarizes license text read in November 2026 and is not legal advice. A gateway can enforce these lines: virtual keys are deny-by-default for providers and list the models each team may call, so a model whose license excludes hosted services stays on internal keys. The same AI governance controls cover self-hosted and cloud models.

Matching Self-Hosted LLMs to Your Hardware

Total parameter count sets how much GPU memory a self-hosted LLM needs, and active parameter count sets how fast it generates tokens. A mixture of experts with 5B active parameters generates quickly, but all of its weights still have to fit in memory, so hardware narrows the shortlist before benchmarks do.

Four hardware tiers from edge devices to multi-GPU nodes, each mapped to self-hosted LLM options such as gpt-oss-20b and Kimi K3.1 Pro

Figure 2: Start from the hardware you already have; the model shortlist follows from memory, not from leaderboard rank.

As Figure 2 shows, model cards and parameter counts place each family in one of four tiers:

  • Laptops and edge devices: Gemma 4 E2B and E4B, and Nemotron 3.5 Bolt E4B for on-device assistants.
  • One consumer GPU: gpt-oss-20b within 16 GB, the 24GB Nemotron 3.5 Bolt, and Gemma 4 12B, 26B A4B, and 31B.
  • One 80 GB GPU: gpt-oss-120b on a single H100 or MI300X.
  • Multi-GPU nodes: DeepSeek V4-Pro (served on one four-GPU GB300 node in its card), Kimi K3.1 Pro, Mistral Large 4, and Qwen3.8-2.4T-A95B.

Replicas of one model behind weighted load balancing across keys add capacity without application changes, and our section on planning on-premise LLM capacity covers sizing.

vLLM vs SGLang vs Ollama for Serving Open Models

vLLM and SGLang are high-throughput inference servers for GPU servers and clusters, and Ollama is a local-first runtime for workstations and single machines. The large models in this list publish vLLM recipes and SGLang cookbooks, while Ollama ships quantized local builds of the smaller ones.

  • vLLM serves OpenAI-compatible endpoints, and Bifrost supports its chat, Responses, embeddings, rerank, and transcription operations; see the vLLM documentation.
  • SGLang serves OpenAI-compatible endpoints at high throughput, and Bifrost supports its chat, Responses, completion, and embeddings operations; see the SGLang documentation.
  • Ollama runs models locally with tool calling; Ollama's library has local builds of Qwen3.8-27B, Gemma 4, gpt-oss, and Nemotron 3.5, but lists GLM-5.3, Kimi K3, MiniMax M3, and Mistral Large 4 as cloud tags only.

Our comparison of LLM gateways for vLLM, SGLang, and Ollama covers the trade-offs at the gateway layer.

How to Run Open Source LLMs Behind Bifrost

Running open source LLMs behind Bifrost takes three steps: register each inference server as a provider, call models with a provider prefix through one OpenAI-compatible endpoint, and scope access with virtual keys. Applications keep their OpenAI SDK code and change only the base URL.

Bifrost, the open-source gateway, supports vLLM, SGLang, and Ollama as built-in providers next to hosted ones, and reaches 10,000+ models across 25+ providers through the same API.

Step 1: Register the inference servers

Add each server in the web UI or config.json. A vLLM key carries the server URL and model name, and its value can stay empty for a local server without authentication:

{
  "providers": {
    "vllm": {
      "keys": [{
        "name": "qwen-gpu-pool",
        "value": "",
        "models": ["Qwen/Qwen3.8-27B"],
        "weight": 1.0,
        "vllm_key_config": {
          "url": "http://vllm-qwen.internal:8000",
          "model_name": "Qwen/Qwen3.8-27B"
        }
      }]
    }
  }
}

SGLang keys take an sgl_key_config URL the same way, and the provider configuration guide covers every option. Other OpenAI-compatible servers, such as llama.cpp, connect as custom providers with their own base URL and TLS settings for internal certificate authorities.

Step 2: Call any model through one endpoint

Point the OpenAI SDK at Bifrost as a drop-in replacement and select the server with the provider prefix:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vllm/Qwen/Qwen3.8-27B",
    "messages": [{"role": "user", "content": "Summarize this incident report"}],
    "fallbacks": ["sgl/zai-org/GLM-5.3-Flash", "openai/<model>"]
  }'

Step 3: Scope access with virtual keys

Create a virtual key per team or application listing the providers and models it may use, with budgets and rate limits attached. A team scoped to vllm and sgl cannot send prompts to a cloud provider, which keeps regulated data inside the network. The same controls govern hosted models, which is the reason to run one LLM gateway for both.

Fallbacks, Governance, and Logs for Self-Hosted LLMs

Self-hosted GPU servers fail like any other service, so production deployments pair each self-hosted LLM with a fallback chain. The Bifrost gateway retries transient errors, moves to the next provider when the primary fails, and checks each fallback attempt against the same virtual key, budget, and rate limit.

A request passes the virtual key and Bifrost routing to a vLLM primary, then falls back to an SGLang replica and a cloud provider

Figure 3: Fallbacks are evaluated by the gateway, and each fallback attempt is still checked against the same virtual key and budget.

Four capabilities keep a mixed self-hosted and cloud fleet manageable:

  • Retries and fallbacks: transient 5xx and network errors retry with exponential backoff, and each fallback provider gets its own retry budget.
  • Routing rules: CEL expressions pick a provider per request from headers, parameters, or team, so regulated traffic stays on self-hosted providers.
  • Built-in observability: each request is logged with tokens, cost, latency, and the provider and model that served it.
  • Low overhead: about 11 microseconds per request at 5,000 RPS (benchmarks), so the gateway does not slow a GPU server down.

Bifrost Enterprise adds high availability, identity-based governance, and audit logs. Our guide to deploying an LLM stack in an air-gapped environment covers the offline install, and the post on automatic fallback with Bifrost shows failover end to end.

Frequently Asked Questions

What is the best open source LLM?

As of November 2026, no single open source LLM is best for every deployment. DeepSeek V4.1 and GLM-5.3-Flash pair strong agentic results with the MIT license, Kimi K3.1 Pro suits demanding agent work on multi-GPU nodes, Qwen3.8-27B is an Apache 2.0 model for one GPU server, and gpt-oss-20b and Gemma 4 cover laptops and workstations.

What is the difference between open weight and open source models?

An open-weight model publishes trained weights that anyone can run and fine-tune. A fully open source AI model also releases the data information and code needed to study and rebuild it, as the Open Source AI Definition sets out. Most models called open source LLMs are open-weight releases.

Is DeepSeek open source?

DeepSeek releases its models as open weights under the MIT license. DeepSeek-V4.1-Flash, its October 2026 update, and DeepSeek-V4-Pro-0813 ship code and weights under MIT, which permits commercial use, modification, and self-hosting as long as the copyright notice is kept.

Is Qwen open source?

Qwen3.8-27B is released under Apache 2.0, so it can be self-hosted and used commercially. Larger Qwen3.8 releases use custom licenses: Flash-Next requires a separate license for hosted-model businesses, and the 2.4T model requires one once such a business exceeds US$50M in yearly revenue.

Is Llama open source?

Meta's Llama 4 weights are published on Hugging Face under the Llama 4 Community License, and repository access requires Meta's approval. Llama 4 Scout and Maverick were released in April 2025, and Meta has not published a newer Llama there as of November 2026, so this list ranks newer families.

What is the best local LLM for coding?

For local coding on one GPU, gpt-oss-20b and Qwen3.8-27B are practical choices, and the 24GB Nemotron 3.5 Bolt targets local agentic tasks. Larger self-hosted coding agents run GLM-5.3 or DeepSeek V4.1-Flash. Our guide to air-gapped Claude Code in regulated industries shows a coding agent on self-hosted models.

Run the Best Open Source LLMs Behind Bifrost

The best open source LLM for a deployment depends on its license, hardware footprint, and runtime, and most teams run more than one. Bifrost, the open-source AI gateway, gives those models and any cloud providers one OpenAI-compatible API with fallbacks, virtual keys, and request logs inside your own network. Review the enterprise deployment options for private AI infrastructure, or book a demo with the Bifrost team to plan a self-hosted LLM rollout.