An LLM gateway is one endpoint in front of many model providers. It centralises seven things that otherwise get reimplemented in every service: credential storage, model routing, failover, caching, rate limiting, observability, and spend control. You need one when you have more than one provider or more than one team; you do not need one when a single service calls a single provider with a single key, because a gateway is a network hop, a dependency, and sometimes a percentage fee. The main options in August 2026 are OpenRouter, LiteLLM, Portkey, Helicone, Bifrost, Kong, the Vercel and Cloudflare platform gateways, and gateways built into products such as Continuum.
- A gateway is one endpoint, many providers. Your app holds one key; the gateway holds the rest.
- The seven jobs: auth, routing, failover, caching, rate limits, observability, spend control.
- A router picks the model. A proxy forwards bytes. A gateway usually does both, plus governance.
- An API gateway is not an LLM gateway until it can count tokens and handle streaming.
- One service, one provider, one key: you do not need a gateway. You need a retry loop.
- Self-hosted gateways are free to license and not free to run. That is the whole trade.
The definition, and the request path
An LLM gateway is a service that accepts model requests in one API shape, usually OpenAI-shaped, and forwards them to whichever provider actually serves that model. Your application knows one base URL and one API key. The gateway knows your Anthropic key, your OpenAI key, your Google key, your self-hosted endpoint, and the rules about which one to use.
Concretely, the request path changes from two hops to three:
app -> api.anthropic.com (one key, one provider, one failure mode)
app -> api.openai.com (another key, another SDK, another retry loop)
app -> gateway -> api.anthropic.com
-> api.openai.com
-> your vLLM box
(one key, one SDK, one place that knows the rules)The interesting word in that diagram is rules. A gateway is not valuable because it forwards bytes; a reverse proxy does that in ten lines of nginx config. It is valuable because it is the one place where "use the cheap model for classification", "fall back to Claude if OpenAI 500s", "this team may not spend more than $400 a week", and "log every prompt for the audit trail" can be true for every service at once, instead of being reimplemented, inconsistently, in each of them.
The seven jobs
Every product in this category does some subset of these. The differences between vendors are almost entirely about which subset they do well, and whether the rest is present but shallow.
| Job | What it means in practice | Who it is for |
|---|---|---|
| Credential custody | Provider keys live in the gateway, not in twelve services and four CI secrets stores. Apps get scoped virtual keys you can revoke individually. | Security, and anyone who has rotated a leaked key |
| Model routing | Send cheap work to a cheap model and hard work to a strong one, by rule or by classifier. See LLM routing. | Cost owners |
| Failover | A 429, a 529, or a timeout retries against a second provider instead of surfacing to the user. See LLM failover. | Anyone with an SLA |
| Caching | Identical prompts return a stored answer. Exact-match caching is common; semantic caching is common and much riskier. | High-volume, repetitive workloads |
| Rate limits and budgets | Per key, per team, per model. A runaway loop stops at the gateway rather than at the invoice. | Platform teams |
| Observability | Every request logged with model, tokens, latency, cost, and the prompt if you allow it. This is the feature people actually buy. | Everyone |
| Spend control | Attribution by team, project, or user, so "who spent $9,000 last month" has an answer. | Finance, eventually |
Two of those deserve a caveat. Semantic caching, where a request that is merely similar to a previous one returns the cached answer, sounds like free money and is a correctness hazard: "delete the staging bucket" and "delete the production bucket" are semantically close. Turn it on for FAQ-shaped traffic and leave it off for anything that acts. Prompt logging is the feature that turns your gateway into a system of record for whatever your users typed, which is a compliance decision, not a debugging convenience.
Gateway, router, proxy, API gateway: the words untangled
These four terms are used interchangeably in marketing copy and mean different things in an architecture diagram.
| Term | What it actually denotes | Example |
|---|---|---|
| LLM proxy | Forwards requests to an upstream model API, possibly rewriting auth. Minimum viable version of the category. | A ten-line FastAPI app |
| LLM router | Chooses which model answers a given request. Can be a static rule or a trained classifier. | openrouter/auto, Continuum Auto Model Mode |
| LLM gateway | A proxy plus a router plus governance: keys, budgets, logs, retries. The superset. | LiteLLM, Portkey, OpenRouter |
| API gateway | Generic HTTP ingress: routing, authn, rate limits, for any API. Becomes an AI gateway when it learns tokens and SSE streaming. | Kong, Envoy, Apigee |
The distinction that matters operationally is the last one. A generic API gateway counts requests; an LLM gateway counts tokens. A rate limit of "100 requests per minute" is close to meaningless when one request can be 4,000 tokens and the next can be 400,000. Kong's answer was to ship AI-specific plugins on top of the existing Kong Gateway (Apache 2.0, 44,000 stars on GitHub as of August 2026) rather than to build a second product, which is the right call if you already run Kong and the wrong one if you do not.
When you need one, and when it is overkill
Reasons that are good enough on their own
- More than one provider. The moment you have both an Anthropic key and an OpenAI key in production, you have a routing decision, and it will otherwise live in an
ifstatement someone copy-pasted. - More than one team. Shared keys make spend unattributable. Virtual keys per team are the cheapest possible fix.
- An availability requirement. Provider outages are real and correlated with your busiest hour. Failover has to live somewhere.
- An audit or residency obligation. "Show me every prompt that left the building last quarter" is a gateway question.
Reasons that are not
- "We might switch providers one day." The OpenAI SDK already takes a
base_url. Switching is a config change, not an architecture. - "We want retries." Every official SDK retries transient failures with exponential backoff already. Anthropic's retries twice by default and honours
retry-after. - "We want to see our costs." If you are one developer, your provider console already shows this, and so does a local tool that reads your agent logs.
The options, named
One line each, honestly. Prices and repository figures were read in August 2026; the detail is in the full comparison and the pricing table.
| Option | Shape | The honest one-liner |
|---|---|---|
| OpenRouter | Hosted aggregator | The fastest way to reach 400+ models with one key. You pay a 5.5% fee on credit purchases, and you are trusting a middleman with your traffic. |
| LiteLLM | Self-hosted, MIT core | The default open-source answer, 56,700 stars. Enormous provider coverage, and an operations burden that people underestimate. |
| Portkey | OSS gateway + hosted | Governance and guardrails are the strong suit. The MIT gateway is real; the useful platform around it starts at $49/mo. |
| Helicone | Observability-first | Best if what you actually want is logs and dashboards. Weaker as a routing layer. Free to 10,000 requests a month, then $79/mo. |
| Bifrost | Self-hosted, Apache 2.0 | Newer, Go, sells itself on throughput and overhead. Smaller ecosystem than LiteLLM, so you will read source sooner. |
| Kong AI Gateway | Plugins on Kong | Correct choice if Kong already fronts your APIs. Heavy if it does not. |
| Vercel AI Gateway | Platform gateway | Zero markup on tokens, including BYOK. Natural if you are already on Vercel, irrelevant if you are not. |
| Cloudflare AI Gateway | Platform gateway | Analytics, caching, and rate limiting are free. Unified billing carries a 5% fee on purchased credits. |
| Continuum | Built into a workbench | A gateway fused to a coding-agent IDE with BYOK free forever. Not a standalone proxy for application traffic. |
The split that matters when you choose: hosted aggregators (OpenRouter, Vercel, Cloudflare) sell you convenience and take either a fee or a platform lock-in, while self-hosted gateways (LiteLLM, Portkey OSS, Bifrost, Kong) sell you control and charge you in engineering hours. There is no third option where you get both, and the vendors claiming otherwise are describing their enterprise tier.
Questions people ask
What is an LLM gateway?
A single API endpoint that sits between your application and multiple model providers. Your code sends one request shape to one URL with one key; the gateway holds the provider credentials and handles routing, failover, caching, rate limits, logging, and spend attribution.
What is the difference between an AI gateway and an API gateway?
An API gateway routes and rate-limits generic HTTP traffic and counts requests. An AI gateway understands model requests: it counts tokens, handles server-sent-event streaming, knows what a model costs, and can fail over between providers that speak different API shapes. Kong is an API gateway that grew AI plugins; LiteLLM is an AI gateway that was never anything else.
Do I need an LLM gateway?
Not if one service calls one provider with one key. Official SDKs already retry transient errors with backoff, and your provider console already shows spend. You need one when you have multiple providers, multiple teams sharing keys, a real availability requirement, or an audit obligation.
Does an LLM gateway add latency?
Yes, one network hop plus the gateway processing time. Well-built gateways add single-digit to low-double-digit milliseconds, which is negligible next to a multi-second generation. The real latency risk is not the hop, it is a gateway that is geographically far from either you or the provider.
Is an LLM gateway the same as an LLM router?
No. A router chooses which model answers. A gateway is the superset that includes routing plus credential custody, failover, budgets, and logging. Every gateway contains a router; a router on its own is a much smaller thing.
What is the best open-source LLM gateway?
LiteLLM by adoption, with 56,700 GitHub stars in August 2026 and the widest provider coverage. Portkey and Bifrost are the credible alternatives, and Kong makes sense if it already fronts your APIs.
Sources
Every figure above was read from these pages on August 2026. Vendors reprice without notice; if you find a stale number, tell us.