Skip to main content

Timeouts

The timeout set in router is for the entire length of the call, and is passed down to the completion() call level as well.

Global Timeouts​

from litellm import Router 

model_list = [{...}]

router = Router(model_list=model_list, 
                timeout=30) # raise timeout error if call takes > 30s 

print(response)

litellm_settings.request_timeout (or the REQUEST_TIMEOUT env var) is the proxy-wide default when no router or deployment timeout is set. It applies to the native /v1/responses and /v1/messages streams as well, where it bounds each wait for the next chunk, so a stalled upstream ends the stream with an error instead of hanging. See request timeouts on pass-through routes for the full precedence

Custom Timeouts & Stream Timeouts (Per Model)​

For each model, you can set timeout and stream_timeout under litellm_params:

  • timeout → maximum time for the complete response.
    Use this to cap long-running completions.

  • stream_timeout → maximum time to wait for the first chunk (i.e., first token) in a streaming response.
    Use this to abort “hanging” providers (e.g., Bedrock slow start) and retry another model.

Some requests can't be streamed by the provider, for example a json_schema response format on a Bedrock model without native structured output. LiteLLM then makes one non-streaming call and sends the answer back as chunks once it is complete, so no chunk arrives until the whole answer is written. These requests use the longer of timeout and stream_timeout, so a short stream_timeout meant for stalled streams does not cut off a slow structured answer.

The same rules apply when you call litellm.completion(), litellm.acompletion(), or litellm.aresponses() directly without a Router. Pass stream_timeout next to timeout, and a live stream is bounded by stream_timeout, a call that is not streamed by timeout, and a faked stream by the longer of the two. Earlier versions ignored stream_timeout on direct SDK calls, so a short value you set there now cuts off a live stream that takes longer than it to send a chunk.

from litellm import Router 
import asyncio

model_list = [{
    "model_name": "gpt-6-luna",
    "litellm_params": {
        "model": "azure/chatgpt-v-2",
        "api_key": os.getenv("AZURE_API_KEY"),
        "api_version": os.getenv("AZURE_API_VERSION"),
        "api_base": os.getenv("AZURE_API_BASE"),
        "timeout": 300, # sets a 5 minute timeout
        "stream_timeout": 30, # sets a 30s timeout for streaming calls
    }
}]

# init router
router = Router(model_list=model_list, routing_strategy="least-busy")
async def router_acompletion():
    response = await router.acompletion(
        model="gpt-6-luna", 
        messages=[{"role": "user", "content": "Hey, how's it going?"}]
    )
    print(response)
    return response

asyncio.run(router_acompletion())

Per Service Tier Timeouts​

A deployment can give requests that ask for a specific service_tier their own timeout. OpenAI's flex tier, for example, is slower than the default tier, so a timeout sized for standard traffic cuts flex requests off too early. Set timeout_by_service_tier and stream_timeout_by_service_tier under litellm_params, keyed by tier name:

model_list:
  - model_name: gpt-6-luna
    litellm_params:
      model: openai/gpt-5.1
      api_key: os.environ/OPENAI_API_KEY
      timeout: 610
      stream_timeout: 610
      timeout_by_service_tier:
        flex: 900
      stream_timeout_by_service_tier:
        flex: 900

A request with "service_tier": "flex" in its body now waits up to 900 seconds on /v1/chat/completions, /v1/responses and every other route that accepts service_tier. A request without service_tier, or with a tier that has no entry in the map, keeps the deployment's timeout and stream_timeout, so adding the maps changes nothing for existing traffic.

Per attempt, LiteLLM picks the first value that is set: a timeout sent with the request (the timeout parameter or the x-litellm-timeout header), then timeout_by_service_tier for the request's tier, then the deployment's timeout, then the router and global defaults described above. Streaming requests go through the same order with stream_timeout_by_service_tier and stream_timeout; a tier that is only listed in timeout_by_service_tier keeps the deployment's stream_timeout for its streams. Values must be numbers of seconds; the proxy refuses to start on anything else. Tier names are not checked against the provider, so a misspelled key is simply never matched.

Keepalive Pings for Idle Streaming Connections​

timeout and stream_timeout cap how long a request is allowed to run. A separate problem is that load balancers and reverse proxies in front of the proxy often close connections that look idle, even when the client is legitimately waiting on a response. Streaming requests to models with long silent gaps before the first token, such as extended or adaptive thinking models, or otherwise slow providers, can trip these idle-connection timeouts before any content arrives.

Set keepalive_seconds under a deployment's litellm_params to keep the connection alive during these gaps. Once a stream goes silent for longer than keepalive_seconds, the proxy sends an SSE comment frame (: ping) down the connection, repeating every keepalive_seconds until real content resumes. Comment frames are part of the SSE spec, and clients and intermediate proxies are expected to ignore them, so they don't affect the response your application sees.

model_list:
  - model_name: claude-opus
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY
      keepalive_seconds: 15
curl http://0.0.0.0:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{
    "model": "claude-opus",
    "messages": [{"role": "user", "content": "Think step by step about..."}],
    "stream": true
  }'

keepalive_seconds is operator-only by default. A client's request-level keepalive_seconds has no effect unless the deployment also sets allow_client_keepalive_override: true, since letting any client enable heartbeats at will would let it keep an idle-looking stream alive past a load balancer's timeout indefinitely, tying up a max_parallel_requests slot for longer than intended.

model_list:
  - model_name: claude-opus
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY
      keepalive_seconds: 15
      allow_client_keepalive_override: true

With override allowed, a request can change the deployment's default, including disabling it with an explicit 0:

curl http://0.0.0.0:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{
    "model": "claude-opus",
    "messages": [{"role": "user", "content": "Think step by step about..."}],
    "stream": true,
    "keepalive_seconds": 1
  }'

If allow_client_keepalive_override isn't set, that same request body is silently ignored and the deployment's own configured value applies instead. A deployment-level keepalive_seconds: 0 is a hard disable that takes priority over everything, including a grant of override permission: it can't be re-enabled by a request no matter what. The effective value is clamped to the range 1-300 seconds.

keepalive_seconds can also be set with an x-litellm-keepalive-seconds header instead of a request body field, for clients that can set custom headers more easily than extra body fields:

curl http://0.0.0.0:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H 'x-litellm-keepalive-seconds: 1' \
  -d '{
    "model": "claude-opus",
    "messages": [{"role": "user", "content": "Think step by step about..."}],
    "stream": true
  }'

The header goes through the same allow_client_keepalive_override gate as the body field, so it has no effect on a deployment that hasn't opted in either.

A proxy-wide default​

keepalive_seconds is per deployment. To apply one interval across every deployment, and to every pass-through route, set sse_keepalive_ping_interval_seconds under litellm_settings:

litellm_settings:
  sse_keepalive_ping_interval_seconds: 15

A deployment's own keepalive_seconds still wins where it is set, and a deployment-level 0 still hard-disables. The global value only applies where nothing more specific does.

One nuance applies before the upstream has answered, since no deployment has served the request yet: a per-deployment value is only used when every deployment behind the requested model name carries the same one. Where they disagree, the global value applies until the serving deployment is known, after which its own setting takes over for the rest of the stream.

Silence before the upstream answers at all​

Some providers withhold their response headers until the first token, so on those the model's whole thinking time passes before the proxy has anything to relay. sse_keepalive_ping_interval_seconds covers that window too: when the upstream call has not come back within one interval, the proxy opens the SSE response and starts sending : ping comments, then replays the real response onto the same connection once it arrives.

Two things follow from opening the response that early, and both are why this stays off until you set an interval:

  • The status line is committed before the outcome is known, so a request that fails after the first ping arrives as an SSE error frame under a 200 rather than as an HTTP error status. Any error transformation your callbacks apply still applies to that frame
  • LiteLLM's x-litellm-* response headers are not known yet, so they are absent on a stream that pinged

Setting Dynamic Timeouts - Per Request​

LiteLLM supports setting a timeout per request

Example Usage

from litellm import Router 

model_list = [{...}]
router = Router(model_list=model_list)

response = router.completion(
    model="gpt-6-luna", 
    messages=[{"role": "user", "content": "what color is red"}],
    timeout=1
)

Testing timeout handling​

To test if your retry/fallback logic can handle timeouts, you can set mock_timeout=True for testing.

This is currently only supported on /chat/completions and /completions endpoints. Please let us know if you need this for other endpoints.

curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $LITELLM_API_KEY" \
    --data-raw '{
        "model": "gemini/gemini-3.8-flash",
        "messages": [
        {"role": "user", "content": "hi my email is ishaan@berri.ai"}
        ],
        "mock_timeout": true # KEY CHANGE
    }'
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails, built for production.
Learn more →