What is LiteLLM? The SDK, the proxy, and what running it costs you

The two halves

Almost every confused LiteLLM question comes from not knowing which half is being discussed.

The unifying idea in both: everything comes back in OpenAI chat-completions format, and every provider's errors are mapped onto OpenAI exception types. One try/except, one response shape, whether the call went to Anthropic, Bedrock, Vertex, or a local Ollama.

What it is genuinely great at

• Breadth. 100+ provider integrations, including the awkward ones: Bedrock's signing, Vertex's auth, Azure's deployment names. Someone has already fought each of those.
• Virtual keys. Handing a team a key that has a budget, a model allowlist, and a spend report attached is the feature that gets LiteLLM adopted, more than routing ever does.
• No per-token fee, ever. Unlike every hosted gateway, cost does not scale with traffic.
• Nothing leaves your network. For a lot of security reviews this is the entire argument, and no hosted option can match it.
• Error normalization. Genuinely saves weeks across a multi-provider codebase.
• It is where the ecosystem points. Observability vendors, agent frameworks, and internal platforms all have a LiteLLM integration already.

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o
      api_base: https://endpoint.openai.azure.com/
      api_key: "os.environ/AZURE_API_KEY"
      rpm: 200

litellm_settings:
  num_retries: 3
  fallbacks: [{"gpt-4o": ["gpt-3.5-turbo"]}]

router_settings:
  routing_strategy: "simple-shuffle"

general_settings:
  master_key: "sk-1234"

Four top-level keys: model_list, litellm_settings, router_settings, general_settings. Secrets are referenced as os.environ/NAME, never inlined.

The operational reality

This section exists because the README does not have one, and because "just self-host it" is advice that costs teams real money when it is taken at face value.

Config sprawl

That four-key config above is what a demo looks like. A production one carries an entry per model per deployment per region, per-key rate limits, budget definitions, guardrail configs, callback routing, cache settings, and alerting. It is YAML that encodes routing policy, and it grows every time someone adds a model or a team. Nothing is wrong with it, but it becomes infrastructure that needs review, staging, and rollback like any other, and teams that treat it as a config file rather than as code find that out during an incident.

Upgrade churn

Fifteen tags in twelve days, spanning -dev, -rc, and stable, on a project with nearly five thousand open issues. That is velocity, and it is also the reason you cannot pin a version and walk away: fixes you need ship alongside changes you did not ask for. LiteLLM publishes -stable tags precisely because of this, and running those rather than the newest tag is the right default.

Security

Later the same month, on 29 April 2026, came CVE-2026-42208: a SQL injection in the proxy's API key verification path, reachable by an unauthenticated request with a crafted Authorization: Bearer header sent to any LLM API route. It affected v1.81.16 through v1.83.6 and was fixed in v1.83.7, with v1.83.10-stable recommended. External trackers rated it 9.3 and reported exploitation attempts within roughly a day of the advisory going public. The same advisory names CVE-2026-30623 (command injection via Anthropic's MCP SDK) and GHSA-4xpc-pv4p-pm3w (authentication bypass via Host header injection).

To their credit the response was substantive rather than defensive: an external audit engagement with Veria Labs, a bug bounty with $500 to $3,000 payouts, and a five-business-day commitment on GitHub security advisories. But the lesson for anyone choosing this is unambiguous. A self-hosted gateway holds every provider key your company owns. It is the highest-value target in your infrastructure, and running it means running a patch cadence, not just a container.

Pricing

The licence is activated by setting LITELLM_LICENSE to a token; the Swagger page then reads "Enterprise Edition". A 30-day trial key is issued without a sales call.

Who should run it

Run LiteLLM if

• Prompts cannot leave your network, and that is a hard requirement rather than a preference.
• You have a platform team, or at least a named owner with an on-call rotation.
• Inference spend is high enough that a percentage fee is real money (roughly $5k a month and up).
• You need virtual keys with budgets across many internal teams, which is its best feature.
• You are already multi-cloud and need Bedrock, Vertex, and Azure behind one interface.

Do not run LiteLLM if

• You are a solo developer or a small team. A hosted gateway's fee is cheaper than your weekend.
• Nobody will own patching. See the April 2026 list.
• You want a model catalogue you do not have accounts for. That is OpenRouter, not this.
• Your requirement is dashboards rather than routing. That is Helicone or Portkey.

Questions people ask

Keep reading

Continue through LLM gateways and routing with LiteLLM alternatives: seven options, sorted by the job you need done.