A working failover design has four parts. Classify errors: 429, 500, 504, 529 and connection failures are retryable, while 400, 401, 403, 404 and 413 are permanent and must never be retried against a second provider. Retry with exponential backoff and honour retry-after. Fall back to a capability-equivalent model, not merely to a cheaper one. And handle the agent case, where a turn that already ran tool calls cannot simply be replayed. LiteLLM expresses most of this in a dozen lines of YAML.
- Retryable: 429, 500, 504, 529, connection errors. Never retryable: 400, 401, 403, 404, 413.
- 529 is overloaded, 429 is you. They need different responses: wait versus slow down.
- Official SDKs already retry twice with backoff and honour
retry-after. Do not stack a second retry loop on top. - A fallback to a weaker model is silent quality degradation unless you log it.
- A mid-task agent is not idempotent. Fail over the request, never the whole turn.
- Test failover with a deliberately broken key, before the outage, not during it.
Classify the error before you do anything
Retrying the wrong error is how a bad request becomes a rate limit. Anthropic's error codes are a good reference set because most providers follow the same shape.
| Code | Type | Retry? | What to do |
|---|---|---|---|
| 400 | invalid_request_error | Never | Your payload is wrong. A second provider will reject it too. |
| 401 | authentication_error | Never | Key is malformed, revoked, or expired. Page someone. |
| 402 | billing_error | Never | Payment problem. Fall over to a different account, not a retry. |
| 403 | permission_error | Never | Key lacks access to that resource. |
| 404 | not_found_error | Never | Usually a model id that does not exist on this provider. |
| 413 | request_too_large | Never as-is | Over 32 MB on the Messages API. Fall back to a larger-context model or trim. |
| 429 | rate_limit_error | Yes, backoff | Honour retry-after. Repeated 429s mean slow down, not retry harder. |
| 500 | api_error | Yes | Provider-side. Retry with exponential backoff, then fall over. |
| 504 | timeout_error | Yes, carefully | Consider streaming instead. A retried long request often times out again. |
| 529 | overloaded_error | Yes, then fail over | The provider is saturated across all users. Waiting alone will not fix it. |
One more, and it is the one that bites: errors after a 200. On a streaming response the connection succeeds and the failure arrives mid-stream as an error event. None of your HTTP-level retry logic sees it. Handle stream errors explicitly or your failover will look perfect in testing and do nothing in production.
Pick capability-equivalent pairs, not cheaper ones
The point of a fallback is to finish the request at acceptable quality. Falling back from a flagship to a small model means the request technically succeeded and the answer is worse, which for a coding agent means a wrong patch rather than an error message. Pair on tier, and cross vendors so a single provider outage cannot take both.
| Tier | Primary | Cross-vendor fallback | Prices |
|---|---|---|---|
| Frontier | Claude Opus 5 ($5 / $25) | GPT-5.6 Sol ($5 / $30) | Near parity |
| Workhorse | Claude Sonnet 5 ($2 / $10) | GPT-5.6 Terra ($2 / $12) | Near parity |
| Cheap and fast | Claude Haiku 4.5 ($1 / $5) | GPT-5.6 Luna ($0.20 / $1.20) | Luna is far cheaper |
| Long context | Claude 4.6 and later, 1M window at standard rates | Provider-dependent | Check the window before you fail over |
Also worth writing down: a fallback that fires silently is a bug. Log the fall, stamp the response with the model that actually answered, and alert if the fallback rate crosses a threshold. Otherwise the first thing you learn is that quality dropped last Tuesday and nobody knows why.
Config that works
LiteLLM expresses the whole design declaratively, which is the main practical argument for a gateway over hand-rolled retry code: the policy is in one file rather than in every service.
litellm_settings:
num_retries: 3
request_timeout: 10
allowed_fails: 3
cooldown_time: 30
fallbacks: [{"primary-model": ["backup-model"]}]
context_window_fallbacks: [{"small-window": ["large-window"]}]
content_policy_fallbacks: [{"claude-2": ["my-fallback-model"]}]
default_fallbacks: ["claude-opus"]| Setting | What it does |
|---|---|
num_retries | Attempts per model before moving to the next one in the chain. |
request_timeout | Seconds before a request counts as failed. Set it below your own client timeout. |
allowed_fails / cooldown_time | After N failures, take a deployment out of rotation for N seconds. This is the part hand-rolled retries always miss. |
context_window_fallbacks | Route to a bigger-window model when the prompt does not fit. Needs enable_pre_call_checks: true under router_settings. |
content_policy_fallbacks | A refusal is not an outage. Sending it to a different vendor is a policy decision, so decide it deliberately. |
default_fallbacks | The catch-all. Model-specific entries override it. |
The cooldown is the setting to actually tune. Without it, an unhealthy deployment keeps receiving traffic, failing, and consuming your retry budget on every request. Thirty seconds of exile costs nothing and stops a degraded provider from dominating your error rate.
The hard case: an agent mid-task
Everything above assumes a request is safe to repeat. Inside an agent loop it frequently is not. By the time a turn fails, the agent may have already written three files, run a migration, and opened a pull request. Replaying the turn replays those effects.
- Fail over the request, not the turn. The unit you retry is a single model call inside the loop, with the conversation history and tool results so far intact. A fallback model receiving that history continues the task; a fresh turn redoes it.
- Make tool calls idempotent where you can. "Write file X with content Y" is safe to repeat. "Append a row" is not. The ones that are not need an idempotency key that survives the retry, which is a property of your tools, not of your gateway.
- Never auto-retry a side-effecting call across a provider switch without knowing whether the first attempt landed. A 504 means the request timed out, not that it did not happen.
- Preserve provider-specific state deliberately. Thinking blocks, cached prefixes, and tool-result formats do not carry across vendors. Anthropic rejects modified or reordered thinking blocks with a 400, so a naive cross-vendor replay of a partial transcript fails on a rule you did not know existed.
- Prefer surfacing a resumable error to a silent switch. For an interactive coding agent, "the provider is overloaded, continue with the other account?" is usually better than swapping models under the user mid-file.
Health checks and the thing to test
A health check that calls GET /models proves the endpoint is up, not that inference works. The useful probe is a tiny real completion, one token out, run every 30 to 60 seconds per deployment, feeding the same cooldown state your traffic uses.
Break a key on purpose
Point the primary at an invalid credential in staging and send real traffic. You are testing that the fallback fires, that it fires fast enough, and that the log tells you it happened.
Simulate 529 rather than 500
Overload behaviour is the case people get wrong, because it is not a hard failure: the provider still answers, slowly and intermittently. Timeouts, not errors, are what you will actually see.
Watch what happens to cost
Failover changes which vendor you are billed by, mid-incident. If your fallback is on metered API pricing and your primary is a subscription, an outage can be expensive in a way nobody modelled. Put an alert on the fallback rate, not just on the error rate.
Questions people ask
What is an Anthropic 529 error?
A 529 overloaded_error means the Claude API is temporarily overloaded, generally because of high traffic across all users rather than anything specific to your account. Backoff helps a little; the reliable remedy is a fallback to another provider. It is distinct from 429 rate_limit_error, which means your own account hit a limit.
Which LLM API errors should I retry?
Retry 429 (honouring retry-after), 500, 504, 529, and connection-level failures with exponential backoff. Never retry 400, 401, 403, 404, or 413: those are permanent and a second provider will reject them the same way, so retrying only converts a bad request into rate-limit pressure.
How do I set up fallbacks in LiteLLM?
Add fallbacks to litellm_settings as a list of maps from a primary model to an ordered list of backups, alongside num_retries, request_timeout, allowed_fails, and cooldown_time. context_window_fallbacks handles prompts too large for the primary and needs enable_pre_call_checks set true under router_settings.
Do the official SDKs already retry?
Yes. Anthropic's SDKs retry transient failures (connection errors, rate limits, 5xx) with exponential backoff twice by default and honour the retry-after header, and each client takes a max-retries option. Adding your own retry loop on top multiplies the attempts, which is how a brief blip becomes a self-inflicted rate limit.
Is it safe to fail over in the middle of an agent task?
Only at the level of a single model call, with the conversation and tool results so far preserved. Replaying a whole turn replays its side effects, and a 504 tells you the request timed out rather than that it did not happen. Side-effecting tools need idempotency keys, and provider-specific state such as thinking blocks does not survive a cross-vendor switch.
Sources
Every figure above was read from these pages on August 2026. Vendors reprice without notice; if you find a stale number, tell us.