LLM API timeout errors: which layer actually killed your request

Three timeouts, and the one everybody sets

A single HTTP call has several independent clocks, and conflating them is why "we set a timeout" so often fails to fix anything.

Getting this wrong in the usual direction, one large total timeout and nothing else, means a request to a black-holed host waits the full total before failing, when a five-second connect timeout would have failed it immediately and let a retry succeed.

The layer that actually killed it

Here is the crux. Both major SDKs wait ten minutes; almost nothing between you and them does. Line up the real numbers and the winner is obvious.

The failure shape tells you which layer without any instrumentation:

That branded-versus-unbranded distinction for 502 and 504 is one of the more useful facts in Cloudflare's documentation and almost nobody knows it.

Streaming is not an optimisation, it is the fix

Most intermediate timeouts are inactivity timeouts, not duration limits. Cloudflare defines the Proxy Read Timeout precisely as "a timeout value between two successive read operations to your origin server". So a response emitting tokens every few hundred milliseconds resets that clock continuously and can run far longer than 125 seconds. A non-streaming request producing nothing for three minutes cannot.

Anthropic enforces the same conclusion from the other end, and does it in the client so you cannot get it wrong quietly:

The user-facing wording of that guard is Streaming is required for operations that may take longer than 10 minutes. The TypeScript SDK takes a different route to the same place, scaling the default timeout with max_tokens up to sixty minutes rather than refusing. And the API side has its own dedicated status for this: a 504 timeout_error, documented as "the request timed out while processing", with the recommendation to use the streaming Messages API for long-running requests.

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=64000,
    messages=messages,
) as stream:
    message = stream.get_final_message()   # identical to a non-streaming result

Timeouts worth setting, by operation

One global timeout cannot serve an autocomplete and a repository-wide refactor. Set them per operation, and set them from the product requirement rather than from the model.

from openai import OpenAI
import httpx2

client = OpenAI(
    timeout=httpx2.Timeout(120.0, connect=5.0),
    max_retries=2,
)

# a specific call that legitimately needs longer
resp = client.with_options(timeout=600.0).responses.create(
    model="gpt-5.6-sol",
    input=big_prompt,
)

• Raise every layer together or none. Raising the SDK timeout to 900 seconds behind a 60 second ALB changes nothing except which error you see.
• Keep the connect timeout short always. It is the one number that should never scale with the work.
• Budget the whole chain. If your API gateway allows 30 seconds, an LLM call inside it cannot have a 120 second timeout in any meaningful sense.
• Do not retry a timeout blind on an agent turn. The turn may have already run tools. This is the same duplicate-execution problem retries cause on 5xx errors.

Retry, or resume?

For a stateless completion, retrying a timeout is correct and cheap. For an agent turn it is neither, because you do not know how far the turn got, and a replay can re-execute side effects that already happened. Past roughly a minute of work, the right pattern stops being retry and becomes resume.

OpenAI documents background mode for exactly this: send background: true, poll while the response is queued or in_progress, and resume a stream from a sequence_number instead of replaying the turn. Two caveats from the same page: time to first token is higher than a synchronous call, and cancellation is idempotent even though the inference endpoints have no idempotency key. There is also a hard ceiling on the WebSocket surface, sixty minutes, with the error Responses websocket connection limit reached (60 minutes). Create a new websocket connection to continue.

Anthropic's equivalent is the Message Batches API, recommended in the errors documentation as the alternative to holding an uninterrupted connection, and priced at half rate in both directions.

Questions people ask

Keep reading

Continue through Errors and fixes with zsh: command not found: claude - the actual fix.