The fastest self-hosted LLM gateway is LiteLLM: one Docker container, a config.yaml listing your models, a master key, and port 4000 serving an OpenAI-compatible API. Adding virtual keys, teams, and budgets means adding Postgres, which is where "a container" becomes "a service". Portkey ships an MIT gateway with stronger guardrails, and Kong AI Gateway is right if Kong already fronts your APIs. The licence is free in all three cases; the total cost of ownership is your on-call rotation, and for small teams a hosted gateway is usually cheaper.
- LiteLLM in one command:
docker run, aconfig.yaml, port 4000. - No Postgres means no virtual keys, no teams, no persistent budgets. Just a proxy.
- The gateway is in the hot path of every model call. Plan for two replicas, not one.
- Portkey OSS is MIT; Kong is Apache 2.0; LiteLLM is MIT with a commercially licensed
enterprise/directory. - Self-hosting wins on data residency and at high volume. It loses on everything else below about $10k/month of tokens.
- Your prompts stop leaving your perimeter. That is the reason that survives a CFO conversation.
LiteLLM in ten minutes
Write a config file
The model_list maps the name your applications use onto the provider model behind it. Keys are read from the environment rather than written into the file.
model_list:
- model_name: gpt-5.5
litellm_params:
model: openai/gpt-5.5
api_key: os.environ/OPENAI_API_KEYRun the container
docker run \
-v $(pwd)/litellm_config.yaml:/app/config.yaml \
-e OPENAI_API_KEY=<your-openai-key> \
-e LITELLM_MASTER_KEY=sk-1234 \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:latest \
--config /app/config.yamlPoint any OpenAI client at it
The proxy speaks the OpenAI API, so every OpenAI SDK, and every tool that accepts a base URL, works unchanged.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000",
api_key="sk-1234",
)
response = client.chat.completions.create(
model="gpt-5.5",
messages=[{"role": "user", "content": "Say hello in five words."}],
)Add the reliability settings before you add traffic
Retries, timeouts, cooldowns, and fallbacks belong in the same file. The full set is in the failover guide.
litellm_settings:
num_retries: 3
request_timeout: 10
allowed_fails: 3
cooldown_time: 30
fallbacks: [{"gpt-5.5": ["claude-sonnet-5"]}]From a container to something you can page on
The compose file below is roughly the minimum honest production shape: the proxy, a database for keys and spend, and Redis for cross-replica rate limiting and cooldown state.
services:
litellm:
image: docker.litellm.ai/berriai/litellm:latest
command: ["--config", "/app/config.yaml"]
ports: ["4000:4000"]
volumes:
- ./litellm_config.yaml:/app/config.yaml
environment:
LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
DATABASE_URL: postgresql://litellm:${PG_PASSWORD}@db:5432/litellm
REDIS_URL: redis://cache:6379
OPENAI_API_KEY: ${OPENAI_API_KEY}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
depends_on: [db, cache]
db:
image: postgres:17
environment:
POSTGRES_USER: litellm
POSTGRES_PASSWORD: ${PG_PASSWORD}
POSTGRES_DB: litellm
volumes: ["pgdata:/var/lib/postgresql/data"]
cache:
image: redis:7
volumes:
pgdata:Things that are not in that file and need to be somewhere: TLS termination, two replicas behind a load balancer so a deploy is not an outage, log shipping, Postgres backups, an upgrade process for a project that ships frequently, and a runbook for "the gateway is down and therefore every AI feature in the company is down".
The two alternatives worth knowing
Portkey OSS gateway
MIT-licensed, 12,766 GitHub stars in August 2026, and explicitly positioned with no paywalled core: the self-hosted tier advertises the universal API, retries and timeouts, routing, guardrails, automatic fallbacks, load balancing, and a basic dashboard, with no request limit. Choose it over LiteLLM when guardrails and access control are the point and provider breadth is not.
The catch is the same one every open-core project has: the observability you actually want at scale lives in the hosted product, where Developer is free at 10,000 logs a month and Production is $49/mo at 100,000. Self-hosting the gateway and paying for the platform is a supported combination and often the sensible one.
Kong AI Gateway
Kong Gateway is Apache 2.0 with 44,000 stars, and the AI Gateway is a set of LLM-aware plugins on top of it: prompt governance, token-based rate limiting, and provider routing for OpenAI, Anthropic, and others. If Kong already fronts your APIs, this is the cheapest possible adoption because the deployment, the auth model, and the on-call rotation already exist. If it does not, you are adopting Kong to get an LLM gateway, which is a much larger decision than it looks.
| Project | Licence | Stars | Best when |
|---|---|---|---|
| LiteLLM | MIT core, commercial enterprise/ | 56,762 | You want the widest provider coverage |
| Portkey Gateway | MIT | 12,766 | Guardrails and access control are the requirement |
| Bifrost | Apache 2.0 | 7,432 | Throughput and overhead are the requirement |
| Kong | Apache 2.0 | 44,003 | Kong already fronts your APIs |
What self-hosting actually costs
The licence is $0. The bill is elsewhere, and it is worth writing down before you commit.
| Cost | Rough shape |
|---|---|
| Compute | Two small instances plus Postgres and Redis. Real, and the smallest number here. |
| Initial build | A few days to something working. A few weeks to something you would page on. |
| Upgrades | A fast-moving project. Budget a maintenance window every month or two, forever. |
| On-call | The gateway is now tier-1 infrastructure. This is the actual cost. |
| Incident debt | The first outage caused by your own gateway will cost more than a year of a hosted plan. |
The comparison that matters: Portkey Production is $49/mo, Helicone Pro is $79/mo, and Cloudflare AI Gateway's core analytics, caching, and rate limiting are free. Against those numbers, a quarter of an engineer is not a rounding error. Self-hosting wins clearly in three situations: prompts may not leave your perimeter, your volume is large enough that percentage fees dominate, or you need provider or deployment support the hosted products do not offer. Outside those, a hosted gateway is usually the cheaper honest answer.
Questions people ask
How do I self-host an LLM gateway?
The quickest route is LiteLLM in Docker: mount a config.yaml listing your models, set LITELLM_MASTER_KEY and your provider keys as environment variables, publish port 4000, and point any OpenAI client at http://localhost:4000. Virtual keys, teams, and persistent budgets require adding Postgres, and production adds Redis, TLS, and a second replica.
What port does the LiteLLM proxy use?
Port 4000 by default. The Docker quickstart publishes it with -p 4000:4000, and clients set base_url to http://localhost:4000 with the master key or a virtual key as the API key.
Is LiteLLM free for commercial use?
The core is MIT-licensed and free to run in production with no request limits. The enterprise/ directory inside the same repository is under a separate commercial licence, which is why GitHub cannot resolve a single licence for the repo. Enterprise features such as SSO, SCIM, audit logs, and SLAs are quoted per gateway capacity rather than per token, and the price is not published.
Is self-hosting an AI gateway cheaper than a hosted one?
Below roughly $10,000 a month in tokens, usually not. Portkey Production is $49/mo and Helicone Pro is $79/mo, while self-hosting costs compute plus a real share of an engineer once you count upgrades and on-call. Self-hosting wins on data residency, at high volume where percentage fees dominate, and when you need deployment options hosted vendors do not offer.
Which open source LLM gateway should I self-host?
LiteLLM for provider breadth and community size, Portkey Gateway (MIT) when guardrails and access control matter most, Bifrost (Apache 2.0) when throughput is the constraint, and Kong AI Gateway when Kong already fronts your APIs and adopting it costs you nothing new.
Sources
Every figure above was read from these pages on August 2026. Vendors reprice without notice; if you find a stale number, tell us.