Skip to main content

Timeouts

The timeout set in router is for the entire length of the call, and is passed down to the completion() call level as well.

Global Timeouts​

from litellm import Router 

model_list = [{...}]

router = Router(model_list=model_list, 
                timeout=30) # raise timeout error if call takes > 30s 

print(response)

Custom Timeouts & Stream Timeouts (Per Model)​

For each model, you can set timeout and stream_timeout under litellm_params:

  • timeout → maximum time for the complete response.
    Use this to cap long-running completions.

  • stream_timeout → maximum time to wait for the first chunk (i.e., first token) in a streaming response.
    Use this to abort “hanging” providers (e.g., Bedrock slow start) and retry another model.

from litellm import Router 
import asyncio

model_list = [{
    "model_name": "gpt-3.5-turbo",
    "litellm_params": {
        "model": "azure/chatgpt-v-2",
        "api_key": os.getenv("AZURE_API_KEY"),
        "api_version": os.getenv("AZURE_API_VERSION"),
        "api_base": os.getenv("AZURE_API_BASE"),
        "timeout": 300 # sets a 5 minute timeout
        "stream_timeout": 30 # sets a 30s timeout for streaming calls
    }
}]

# init router
router = Router(model_list=model_list, routing_strategy="least-busy")
async def router_acompletion():
    response = await router.acompletion(
        model="gpt-3.5-turbo", 
        messages=[{"role": "user", "content": "Hey, how's it going?"}]
    )
    print(response)
    return response

asyncio.run(router_acompletion())

Keepalive Pings for Idle Streaming Connections​

timeout and stream_timeout cap how long a request is allowed to run. A separate problem is that load balancers and reverse proxies in front of the proxy often close connections that look idle, even when the client is legitimately waiting on a response. Streaming requests to models with long silent gaps before the first token, such as extended or adaptive thinking models, or otherwise slow providers, can trip these idle-connection timeouts before any content arrives.

Set keepalive_seconds under a deployment's litellm_params to keep the connection alive during these gaps. Once a stream goes silent for longer than keepalive_seconds, the proxy sends an SSE comment frame (: ping) down the connection, repeating every keepalive_seconds until real content resumes. Comment frames are part of the SSE spec, and clients and intermediate proxies are expected to ignore them, so they don't affect the response your application sees.

model_list:
  - model_name: claude-opus
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY
      keepalive_seconds: 15
curl http://0.0.0.0:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "claude-opus",
    "messages": [{"role": "user", "content": "Think step by step about..."}],
    "stream": true
  }'

keepalive_seconds is operator-only by default. A client's request-level keepalive_seconds has no effect unless the deployment also sets allow_client_keepalive_override: true, since letting any client enable heartbeats at will would let it keep an idle-looking stream alive past a load balancer's timeout indefinitely, tying up a max_parallel_requests slot for longer than intended.

model_list:
  - model_name: claude-opus
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY
      keepalive_seconds: 15
      allow_client_keepalive_override: true

With override allowed, a request can change the deployment's default, including disabling it with an explicit 0:

curl http://0.0.0.0:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "claude-opus",
    "messages": [{"role": "user", "content": "Think step by step about..."}],
    "stream": true,
    "keepalive_seconds": 1
  }'

If allow_client_keepalive_override isn't set, that same request body is silently ignored and the deployment's own configured value applies instead. A deployment-level keepalive_seconds: 0 is a hard disable that takes priority over everything, including a grant of override permission: it can't be re-enabled by a request no matter what. The effective value is clamped to the range 1-300 seconds.

keepalive_seconds can also be set with an x-litellm-keepalive-seconds header instead of a request body field, for clients that can set custom headers more easily than extra body fields:

curl http://0.0.0.0:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'x-litellm-keepalive-seconds: 1' \
  -d '{
    "model": "claude-opus",
    "messages": [{"role": "user", "content": "Think step by step about..."}],
    "stream": true
  }'

The header goes through the same allow_client_keepalive_override gate as the body field, so it has no effect on a deployment that hasn't opted in either.

Setting Dynamic Timeouts - Per Request​

LiteLLM supports setting a timeout per request

Example Usage

from litellm import Router 

model_list = [{...}]
router = Router(model_list=model_list)

response = router.completion(
    model="gpt-3.5-turbo", 
    messages=[{"role": "user", "content": "what color is red"}],
    timeout=1
)

Testing timeout handling​

To test if your retry/fallback logic can handle timeouts, you can set mock_timeout=True for testing.

This is currently only supported on /chat/completions and /completions endpoints. Please let us know if you need this for other endpoints.

curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer sk-1234' \
    --data-raw '{
        "model": "gemini/gemini-1.5-flash",
        "messages": [
        {"role": "user", "content": "hi my email is ishaan@berri.ai"}
        ],
        "mock_timeout": true # 👈 KEY CHANGE
    }'
🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →