--- title: "Router - Load Balancing" url: "/docs/routing" canonical_url: "https://docs.litellm.ai/docs/routing" type: "docs" last_updated: "2026-10-03" summary: "LiteLLM Router for load balancing, routing, retries, cooldowns, and fallbacks (failover) across multiple LLM deployments and providers." related: - "/docs/routing-load-balancing" - "/docs/fusion" --- # Router - Load Balancing > Index of all LiteLLM docs: https://docs.litellm.ai/llms.txt LiteLLM manages: - Load-balance across multiple deployments (e.g. Azure/OpenAI) - Prioritizing important requests to ensure they don't fail (i.e. Queueing) - Basic reliability logic - cooldowns, fallbacks, timeouts and retries (fixed + exponential backoff) across multiple deployments/providers. In production, litellm supports using Redis as a way to track cooldown server and usage (managing tpm/rpm limits). :::info If you want a server to load balance across different LLM APIs, use our [LiteLLM Proxy Server](./proxy/load_balancing.md) ::: ## Load Balancing (s/o [@paulpierre](https://www.linkedin.com/in/paulpierre/) and [sweep proxy](https://docs.sweep.dev/blogs/openai-proxy) for their contributions to this implementation) [**See Code**](https://github.com/BerriAI/litellm/blob/main/litellm/router.py) ### Quick Start Loadbalance across multiple [azure](./providers/azure)/[bedrock](./providers/bedrock.md)/[provider](./providers/) deployments. LiteLLM will handle retrying in different regions if a call fails. **SDK** ```python from litellm import Router model_list = [{ # list of model deployments "model_name": "gpt-5.6-luna", # model alias -> loadbalance between models with same `model_name` "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") } }, { "model_name": "gpt-5.6-luna", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") } }, { "model_name": "gpt-5.6-luna", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-5.6-luna", "api_key": os.getenv("OPENAI_API_KEY"), } }, { "model_name": "gpt-5.6-terra", "litellm_params": { # params for litellm completion/embedding call "model": "azure/gpt-5.6-terra", "api_key": os.getenv("AZURE_API_KEY"), "api_base": os.getenv("AZURE_API_BASE"), "api_version": os.getenv("AZURE_API_VERSION"), } }, { "model_name": "gpt-5.6-terra", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-5.6-terra", "api_key": os.getenv("OPENAI_API_KEY"), } }, ] router = Router(model_list=model_list) # openai.ChatCompletion.create replacement # requests with model="gpt-5.6-luna" will pick a deployment where model_name="gpt-5.6-luna" response = await router.acompletion(model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}]) print(response) # openai.ChatCompletion.create replacement # requests with model="gpt-5.6-terra" will pick a deployment where model_name="gpt-5.6-terra" response = await router.acompletion(model="gpt-5.6-terra", messages=[{"role": "user", "content": "Hey, how's it going?"}]) print(response) ``` **PROXY** :::info See detailed proxy loadbalancing/fallback docs [here](./proxy/reliability.md) ::: 1. Setup model_list with multiple deployments ```yaml model_list: - model_name: gpt-5.6-luna litellm_params: model: azure/ api_base: api_key: - model_name: gpt-5.6-luna litellm_params: model: azure/gpt-turbo-small-ca api_base: https://my-endpoint-canada-berri992.openai.azure.com/ api_key: - model_name: gpt-5.6-luna litellm_params: model: azure/gpt-turbo-large api_base: https://openai-france-1234.openai.azure.com/ api_key: ``` 2. Start proxy ```bash litellm --config /path/to/config.yaml ``` 3. Test it! ```bash curl -X POST 'http://0.0.0.0:4000/chat/completions' \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $LITELLM_API_KEY" \ -d '{ "model": "gpt-5.6-luna", "messages": [ {"role": "user", "content": "Hi there!"} ], "mock_testing_rate_limit_error": true }' ``` ### Available Endpoints - `router.completion()` - chat completions endpoint to call 100+ LLMs - `router.acompletion()` - async chat completion calls - `router.embedding()` - embedding endpoint for Azure, OpenAI, Huggingface endpoints - `router.aembedding()` - async embeddings calls - `router.text_completion()` - completion calls in the old OpenAI `/v1/completions` endpoint format - `router.atext_completion()` - async text completion calls - `router.image_generation()` - completion calls in OpenAI `/v1/images/generations` endpoint format - `router.aimage_generation()` - async image generation calls ## Advanced - Routing Strategies ⭐️ #### Routing Strategies - Weighted Pick, Rate Limit Aware, Least Busy, Latency Based, Cost Based Router provides multiple strategies for routing your calls across multiple deployments. **We recommend using `simple-shuffle` (default) for best performance in production.** **(Default) Weighted Pick - RECOMMENDED** **Default and Recommended for Production** - Best performance with minimal latency overhead. Picks a deployment based on the provided **Requests per minute (rpm) or Tokens per minute (tpm)** If `rpm` or `tpm` is not provided, it randomly picks a deployment You can also set a `weight` param, to specify which model should get picked when. **RPM-based shuffling** ##### **LiteLLM Proxy Config.yaml** ```yaml model_list: - model_name: gpt-5.6-luna litellm_params: model: azure/chatgpt-v-2 api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE rpm: 900 - model_name: gpt-5.6-luna litellm_params: model: azure/chatgpt-functioncalling api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE rpm: 10 ``` ##### **Python SDK** ```python from litellm import Router import asyncio model_list = [{ # list of model deployments "model_name": "gpt-5.6-luna", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "rpm": 900, # requests per minute for this API } }, { "model_name": "gpt-5.6-luna", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "rpm": 10, } },] # init router router = Router(model_list=model_list, routing_strategy="simple-shuffle") async def router_acompletion(): response = await router.acompletion( model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}] ) print(response) return response asyncio.run(router_acompletion()) ``` **Weight-based shuffling** ##### **LiteLLM Proxy Config.yaml** ```yaml model_list: - model_name: gpt-5.6-luna litellm_params: model: azure/chatgpt-v-2 api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE weight: 9 - model_name: gpt-5.6-luna litellm_params: model: azure/chatgpt-functioncalling api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE weight: 1 ``` ##### **Python SDK** ```python from litellm import Router import asyncio model_list = [{ "model_name": "gpt-5.6-luna", # model alias "litellm_params": { "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "weight": 9, # pick this 90% of the time } }, { "model_name": "gpt-5.6-luna", "litellm_params": { "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "weight": 1, } }] # init router router = Router(model_list=model_list, routing_strategy="simple-shuffle") async def router_acompletion(): response = await router.acompletion( model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}] ) print(response) return response asyncio.run(router_acompletion()) ``` **Rate-Limit Aware v2 (ASYNC)** :::warning **Usage-based routing is not recommended for production due to performance impacts.** Use `simple-shuffle` (default) for optimal performance in high-traffic scenarios. Usage-based routing adds significant latency due to Redis operations for tracking usage across deployments. ::: **🎉 NEW** This is an async implementation of usage-based-routing. **Filters out deployment if tpm/rpm limit exceeded** - If you pass in the deployment's tpm/rpm limits. Routes to **deployment with lowest TPM usage** for that minute. In production, we use Redis to track usage (TPM/RPM) across multiple deployments. This implementation uses **async redis calls** (redis.incr and redis.mget). For Azure, [you get 6 RPM per 1000 TPM](https://stackoverflow.com/questions/77368844/what-is-the-request-per-minute-rate-limit-for-azure-openai-models-for-gpt-3-5-tu) **sdk** ```python from litellm import Router model_list = [{ # list of model deployments "model_name": "gpt-5.6-luna", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "tpm": 100000, "rpm": 10000, }, }, { "model_name": "gpt-5.6-luna", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "tpm": 100000, "rpm": 1000, }, }, { "model_name": "gpt-5.6-luna", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-5.6-luna", "api_key": os.getenv("OPENAI_API_KEY"), "tpm": 100000, "rpm": 1000, }, }] router = Router(model_list=model_list, redis_host=os.environ["REDIS_HOST"], redis_password=os.environ["REDIS_PASSWORD"], redis_port=os.environ["REDIS_PORT"], routing_strategy="simple-shuffle", # 👈 RECOMMENDED - best performance enable_pre_call_checks=True, # enables router rate limits for concurrent calls ) response = await router.acompletion(model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}]) print(response) ``` **proxy** **1. Set strategy in config** ```yaml model_list: - model_name: gpt-5.6-luna # model alias litellm_params: # params for litellm completion/embedding call model: azure/chatgpt-v-2 # actual model name api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE tpm: 100000 rpm: 10000 - model_name: gpt-5.6-luna litellm_params: # params for litellm completion/embedding call model: gpt-5.6-luna api_key: os.environ/OPENAI_API_KEY tpm: 100000 rpm: 1000 router_settings: routing_strategy: simple-shuffle # 👈 RECOMMENDED - best performance redis_host: redis_password: redis_port: enable_pre_call_checks: true general_settings: master_key: os.environ/LITELLM_MASTER_KEY ``` **2. Start proxy** ```bash litellm --config /path/to/config.yaml ``` **3. Test it!** ```bash curl --location 'http://localhost:4000/v1/chat/completions' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer $LITELLM_API_KEY" \ --data '{ "model": "gpt-5.6-luna", "messages": [{"role": "user", "content": "Hey, how's it going?"}] }' ``` **Latency-Based** Picks the deployment with the lowest response time. It caches, and updates the response times for deployments based on when a request was sent and received from a deployment. [**How to test**](https://github.com/BerriAI/litellm/blob/main/tests/local_testing/test_lowest_latency_routing.py) ```python from litellm import Router import asyncio model_list = [{ ... }] # init router router = Router(model_list=model_list, routing_strategy="latency-based-routing",# 👈 set routing strategy enable_pre_call_checks=True, # enables router rate limits for concurrent calls ) ## CALL 1+2 tasks = [] response = None final_response = None for _ in range(2): tasks.append(router.acompletion(model=model, messages=messages)) response = await asyncio.gather(*tasks) if response is not None: ## CALL 3 await asyncio.sleep(1) # let the cache update happen picked_deployment = router.lowestlatency_logger.get_available_deployments( model_group=model, healthy_deployments=router.healthy_deployments ) final_response = await router.acompletion(model=model, messages=messages) print(f"min deployment id: {picked_deployment}") print(f"model id: {final_response._hidden_params['model_id']}") assert ( final_response._hidden_params["model_id"] == picked_deployment["model_info"]["id"] ) ``` #### Set Time Window Set time window for how far back to consider when averaging latency for a deployment. **In Router** ```python router = Router(..., routing_strategy_args={"ttl": 10}) ``` **In Proxy** ```yaml router_settings: routing_strategy_args: {"ttl": 10} ``` #### Set Lowest Latency Buffer Set a buffer within which deployments are candidates for making calls to. E.g. if you have 5 deployments ``` https://litellm-prod-1.openai.azure.com/: 0.07s https://litellm-prod-2.openai.azure.com/: 0.1s https://litellm-prod-3.openai.azure.com/: 0.1s https://litellm-prod-4.openai.azure.com/: 0.1s https://litellm-prod-5.openai.azure.com/: 4.66s ``` to prevent initially overloading `prod-1`, with all requests - we can set a buffer of 50%, to consider deployments `prod-2, prod-3, prod-4`. **In Router** ```python router = Router(..., routing_strategy_args={"lowest_latency_buffer": 0.5}) ``` **In Proxy** ```yaml router_settings: routing_strategy_args: {"lowest_latency_buffer": 0.5} ``` **Rate-Limit Aware** This will route to the deployment with the lowest TPM usage for that minute. In production, we use Redis to track usage (TPM/RPM) across multiple deployments. If you pass in the deployment's tpm/rpm limits, this will also check against that, and filter out any who's limits would be exceeded. For Azure, your RPM = TPM/6. ```python from litellm import Router model_list = [{ # list of model deployments "model_name": "gpt-5.6-luna", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") }, "tpm": 100000, "rpm": 10000, }, { "model_name": "gpt-5.6-luna", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") }, "tpm": 100000, "rpm": 1000, }, { "model_name": "gpt-5.6-luna", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-5.6-luna", "api_key": os.getenv("OPENAI_API_KEY"), }, "tpm": 100000, "rpm": 1000, }] router = Router(model_list=model_list, redis_host=os.environ["REDIS_HOST"], redis_password=os.environ["REDIS_PASSWORD"], redis_port=os.environ["REDIS_PORT"], routing_strategy="usage-based-routing", enable_pre_call_checks=True, # enables router rate limits for concurrent calls ) response = await router.acompletion(model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}]) print(response) ``` **Least-Busy** Picks a deployment with the least number of ongoing calls, it's handling. [**How to test**](https://github.com/BerriAI/litellm/blob/main/tests/local_testing/test_least_busy_routing.py) ```python from litellm import Router import asyncio model_list = [{ # list of model deployments "model_name": "gpt-5.6-luna", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), } }, { "model_name": "gpt-5.6-luna", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), } }, { "model_name": "gpt-5.6-luna", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-5.6-luna", "api_key": os.getenv("OPENAI_API_KEY"), } }] # init router router = Router(model_list=model_list, routing_strategy="least-busy") async def router_acompletion(): response = await router.acompletion( model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}] ) print(response) return response asyncio.run(router_acompletion()) ``` **Custom Routing Strategy** **Plugin a custom routing strategy to select deployments** Step 1. Define your custom routing strategy ```python from litellm.router import CustomRoutingStrategyBase class CustomRoutingStrategy(CustomRoutingStrategyBase): async def async_get_available_deployment( self, model: str, messages: Optional[List[Dict[str, str]]] = None, input: Optional[Union[str, List]] = None, specific_deployment: Optional[bool] = False, request_kwargs: Optional[Dict] = None, ): """ Asynchronously retrieves the available deployment based on the given parameters. Args: model (str): The name of the model. messages (Optional[List[Dict[str, str]]], optional): The list of messages for a given request. Defaults to None. input (Optional[Union[str, List]], optional): The input for a given embedding request. Defaults to None. specific_deployment (Optional[bool], optional): Whether to retrieve a specific deployment. Defaults to False. request_kwargs (Optional[Dict], optional): Additional request keyword arguments. Defaults to None. Returns: Returns an element from litellm.router.model_list """ print("In CUSTOM async get available deployment") model_list = router.model_list print("router model list=", model_list) for model in model_list: if isinstance(model, dict): if model["litellm_params"]["model"] == "openai/very-special-endpoint": return model pass def get_available_deployment( self, model: str, messages: Optional[List[Dict[str, str]]] = None, input: Optional[Union[str, List]] = None, specific_deployment: Optional[bool] = False, request_kwargs: Optional[Dict] = None, ): """ Synchronously retrieves the available deployment based on the given parameters. Args: model (str): The name of the model. messages (Optional[List[Dict[str, str]]], optional): The list of messages for a given request. Defaults to None. input (Optional[Union[str, List]], optional): The input for a given embedding request. Defaults to None. specific_deployment (Optional[bool], optional): Whether to retrieve a specific deployment. Defaults to False. request_kwargs (Optional[Dict], optional): Additional request keyword arguments. Defaults to None. Returns: Returns an element from litellm.router.model_list """ pass ``` Step 2. Initialize Router with custom routing strategy ```python from litellm import Router router = Router( model_list=[ { "model_name": "azure-model", "litellm_params": { "model": "openai/very-special-endpoint", "api_base": "https://exampleopenaiendpoint-production.up.railway.app/", # If you are Krrish, this is OpenAI Endpoint3 on our Railway endpoint :) "api_key": "fake-key", }, "model_info": {"id": "very-special-endpoint"}, }, { "model_name": "azure-model", "litellm_params": { "model": "openai/fast-endpoint", "api_base": "https://exampleopenaiendpoint-production.up.railway.app/", "api_key": "fake-key", }, "model_info": {"id": "fast-endpoint"}, }, ], set_verbose=True, debug_level="DEBUG", timeout=1, ) # type: ignore router.set_custom_routing_strategy(CustomRoutingStrategy()) # 👈 Set your routing strategy here ``` Step 3. Test your routing strategy. Expect your custom routing strategy to be called when running `router.acompletion` requests ```python for _ in range(10): response = await router.acompletion( model="azure-model", messages=[{"role": "user", "content": "hello"}] ) print(response) _picked_model_id = response._hidden_params["model_id"] print("picked model=", _picked_model_id) ``` **Lowest Cost Routing (Async)** Picks a deployment based on the lowest cost How this works: - Get all healthy deployments - Select all deployments that are under their provided `rpm/tpm` limits - For each deployment check if `litellm_param["model"]` exists in [`litellm_model_cost_map`](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) - if deployment does not exist in `litellm_model_cost_map` -> use deployment_cost= `$1` - Select deployment with lowest cost ```python from litellm import Router import asyncio model_list = [ { "model_name": "gpt-5.6-luna", "litellm_params": {"model": "gpt-5.6-terra"}, "model_info": {"id": "openai-gpt-4o"}, }, { "model_name": "gpt-5.6-luna", "litellm_params": {"model": "groq/llama3-8b-8192"}, "model_info": {"id": "groq-llama"}, }, ] # init router router = Router(model_list=model_list, routing_strategy="cost-based-routing") async def router_acompletion(): response = await router.acompletion( model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}] ) print(response) print(response._hidden_params["model_id"]) # expect groq-llama, since groq/llama has lowest cost return response asyncio.run(router_acompletion()) ``` #### Using Custom Input/Output pricing Set `litellm_params["input_cost_per_token"]` and `litellm_params["output_cost_per_token"]` for using custom pricing when routing ```python model_list = [ { "model_name": "gpt-5.6-luna", "litellm_params": { "model": "azure/chatgpt-v-2", "input_cost_per_token": 0.00003, "output_cost_per_token": 0.00003, }, "model_info": {"id": "chatgpt-v-experimental"}, }, { "model_name": "gpt-5.6-luna", "litellm_params": { "model": "azure/chatgpt-v-1", "input_cost_per_token": 0.000000001, "output_cost_per_token": 0.00000001, }, "model_info": {"id": "chatgpt-v-1"}, }, { "model_name": "gpt-5.6-luna", "litellm_params": { "model": "azure/chatgpt-v-5", "input_cost_per_token": 10, "output_cost_per_token": 12, }, "model_info": {"id": "chatgpt-v-5"}, }, ] # init router router = Router(model_list=model_list, routing_strategy="cost-based-routing") async def router_acompletion(): response = await router.acompletion( model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}] ) print(response) print(response._hidden_params["model_id"]) # expect chatgpt-v-1, since chatgpt-v-1 has lowest cost return response asyncio.run(router_acompletion()) ``` ## Routing Groups - Per-Model Strategies and Callable Virtual Models Apply different routing strategies to different models in the same router. A **routing group** binds a list of `model_name`s to a strategy and (optionally) strategy args. Models not claimed by any group fall back to the router's top-level `routing_strategy`. A group is also **callable as a model**: request `model: ` and LiteLLM picks among the union of every member's deployments using the group's strategy. Group names appear in `/v1/models`, so clients that discover models from the gateway (Claude Code with `CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1`, Codex) surface them in their pickers. ```bash curl http://localhost:4000/v1/chat/completions \ -H "Authorization: Bearer $LITELLM_KEY" \ -d '{"model": "anthropic-latency", "messages": [{"role": "user", "content": "ping"}]}' ``` Access control treats a group as its own model name: grant `` on a key or team to let it list and call the group. Membership is not expanded in either direction, so a key granted only the group cannot call members directly and a key granted a member cannot call the group. A group name must not collide with an existing `model_name` or `model_group_alias`; config load rejects it. Requests keep the group name as `model_group` in spend logs, with each row recording the member deployment that actually served it. Fallbacks and `model_group_retry_policy` are keyed by name, so give the group its own entries if you need them. Claude Code and Claude Desktop only auto-discover gateway models whose name contains `claude` or `anthropic`, so name groups like `claude-quality` if you want them to appear in the picker without setting `ANTHROPIC_CUSTOM_MODEL_OPTION` by hand. :::tip You can also create, edit, and delete routing groups from the dashboard. See [Manage Routing Groups via UI](./proxy/ui/routing_groups.md). ::: **When to use this:** you want latency-based routing for `gpt-5.6-terra`, but plain weighted-pick for cheaper models, without spinning up a second router. #### Rules - Each `model_name` may belong to **at most one** group. Overlap raises `ValueError` at init. - Models not in any group use the top-level `routing_strategy` / `routing_strategy_args` (an implicit `"default"` group). The name `"default"` is reserved. - Each group can override `routing_strategy_args` (e.g. latency window TTL, TPM ceilings). - The group is resolved per-request based on the post-pre-routing-hook `model` name. **LiteLLM Proxy Config.yaml** ```yaml model_list: - model_name: gpt-5.6-terra litellm_params: model: openai/gpt-5.6-terra api_key: os.environ/OPENAI_API_KEY - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-5.6-terra api_base: os.environ/AZURE_API_BASE api_key: os.environ/AZURE_API_KEY api_version: "2024-08-01-preview" - model_name: cheap-model litellm_params: model: openai/gpt-5.6-luna api_key: os.environ/OPENAI_API_KEY router_settings: # fallback strategy for models not in any explicit group routing_strategy: simple-shuffle routing_groups: - group_name: latency-sensitive models: [gpt-5.6-terra] routing_strategy: latency-based-routing routing_strategy_args: ttl: 3600 ``` Behavior: - `gpt-5.6-terra` → latency-based routing across the OpenAI + Azure deployments. - `cheap-model` → simple-shuffle (the default group). **Python SDK** ```python from litellm import Router router = Router( model_list=[ {"model_name": "gpt-5.6-terra", "litellm_params": {"model": "openai/gpt-5.6-terra"}}, {"model_name": "gpt-5.6-terra", "litellm_params": {"model": "azure/gpt-5.6-terra", "api_base": "...", "api_key": "..."}}, {"model_name": "cheap-model", "litellm_params": {"model": "openai/gpt-5.6-luna"}}, ], routing_strategy="simple-shuffle", # fallback for ungrouped models routing_groups=[ { "group_name": "latency-sensitive", "models": ["gpt-5.6-terra"], "routing_strategy": "latency-based-routing", "routing_strategy_args": {"ttl": 3600}, }, ], ) ``` #### Multiple groups Two groups can use the same strategy with different args; each gets an independent state instance. ```yaml router_settings: routing_strategy: simple-shuffle routing_groups: - group_name: hot-path models: [gpt-5.6-terra, claude-sonnet] routing_strategy: latency-based-routing routing_strategy_args: ttl: 60 # short window — react quickly to latency changes - group_name: batch models: [gpt-5.6-luna, llama-70b] routing_strategy: usage-based-routing-v2 routing_strategy_args: rpm: 10000 ``` #### Updating at runtime Routing groups can be updated via `Router.update_settings(routing_groups=[...])` or the proxy's `/config/update` endpoint. Per-group state is rebuilt on update. ## Session Affinity (Sticky Sessions) Pin every request of a conversation to the deployment that served its first request. Session affinity is a router pre-call check: it runs before the routing strategy picks a deployment and narrows the candidates to the pinned one, so it works with every strategy on this page (`simple-shuffle`, `least-busy`, `usage-based-routing-v2`, `latency-based-routing`, `cost-based-routing`) and with routing groups. Use it when the deployments behind a model group do not share state, for example provider-side prompt caching, or when a conversation has to stay in one region. **Proxy** ```yaml showLineNumbers title="config.yaml" model_list: - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-5.6-terra api_key: os.environ/AZURE_API_KEY_EASTUS api_base: https://eastus.openai.azure.com - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-5.6-terra api_key: os.environ/AZURE_API_KEY_WESTUS api_base: https://westus.openai.azure.com router_settings: routing_strategy: simple-shuffle # any strategy optional_pre_call_checks: ["session_affinity"] deployment_affinity_ttl_seconds: 3600 # optional, default 3600 ``` Send a session id with every request of the conversation. The proxy reads it from the `x-litellm-session-id` header (`x-litellm-trace-id` is interchangeable), from any `x--session-id` header such as `x-claude-code-session-id`, or from `metadata.session_id` in the request body. ```bash curl http://0.0.0.0:4000/v1/chat/completions \ -H "Authorization: Bearer $LITELLM_API_KEY" \ -H "Content-Type: application/json" \ -H "x-litellm-session-id: 7f1c2d1e-2b5a-4a5e-9c1f-0d5a9a3f8b21" \ -d '{"model": "gpt-5.6-terra", "messages": [{"role": "user", "content": "hi"}]}' ``` The `x-litellm-model-id` response header shows which deployment served the request. It stays the same for every request that carries the same session id. **SDK** ```python showLineNumbers from litellm import Router router = Router( model_list=[ { "model_name": "gpt-5.6-terra", "litellm_params": {"model": "azure/gpt-5.6-terra", "api_key": "...", "api_base": "https://eastus.openai.azure.com"}, }, { "model_name": "gpt-5.6-terra", "litellm_params": {"model": "azure/gpt-5.6-terra", "api_key": "...", "api_base": "https://westus.openai.azure.com"}, }, ], routing_strategy="simple-shuffle", # any strategy optional_pre_call_checks=["session_affinity"], deployment_affinity_ttl_seconds=3600, # optional, default 3600 ) response = await router.acompletion( model="gpt-5.6-terra", messages=[{"role": "user", "content": "hi"}], metadata={"session_id": "7f1c2d1e-2b5a-4a5e-9c1f-0d5a9a3f8b21"}, ) print(response._hidden_params["model_id"]) # same deployment for every call with this session_id ``` #### How it works - The first request with a new session id is routed by the strategy as usual. The deployment it lands on becomes the pin for that model group and session id (on the proxy the pin is also scoped to the caller: the virtual key, or the authenticated user id under JWT auth). - Every later request with the same session id is narrowed to the pinned deployment before the strategy runs. - Every request refreshes the pin, so `deployment_affinity_ttl_seconds` bounds the idle time between turns, not the length of a conversation. - Pins live in the router cache. With Redis configured they are shared across proxy instances; without Redis each instance keeps its own. - If the pinned deployment is in cooldown or no longer in the model group, the request falls through to the routing strategy across the remaining healthy deployments. The pin is kept, so the session returns to its deployment once it is healthy again. On versions before v1.97.0 the session is re-pinned to the deployment the strategy picked instead. - Load balancing happens across sessions rather than across requests: each session uses one deployment for as long as it is active. #### Settings | Setting | Description | |---|---| | `optional_pre_call_checks` | Add `session_affinity` to pin by session id. Add `deployment_affinity` to pin by caller instead of, or as well as, session id (a session pin takes priority). On the proxy the caller is the virtual key, or the authenticated user id when the request carries no key (JWT auth). `responses_api_deployment_check` and `encrypted_content_affinity` are covered in [Responses API session continuity](./response_api.md#load-balancing-with-session-continuity). | | `deployment_affinity_ttl_seconds` | Idle TTL of a pin, in seconds. Default `3600`. | | `model_group_affinity_config` | Enable affinity on some model groups only, for example `{"gpt-4.1": ["session_affinity"]}`. Groups not listed use the global `optional_pre_call_checks`. | `deployment_affinity_ttl_seconds` and `model_group_affinity_config` are read at startup: set them in `config.yaml` (or on `Router()`) and restart the proxy. `optional_pre_call_checks` can also be changed on a running proxy with `POST /config/update` and a body of `{"router_settings": {"optional_pre_call_checks": [...]}}`, as long as the key is not set in `config.yaml` (a value there wins, and sending a different one returns a 400). The call needs an admin key, a database, and `STORE_MODEL_IN_DB=True`. The list replaces the previous one and is stored in the database, so every instance polling the database applies it. Names in the new list are turned on right away. A name left out is turned off right away for `prompt_caching`, `enforce_model_rate_limits`, and `encrypted_content_affinity` (the last from v1.104.0), while `session_affinity`, `deployment_affinity`, `responses_api_deployment_check`, and `router_budget_limiting` stay on until a restart. `GET /router/settings` shows the stored list, the config file merged with the database :::info The `session_affinity` option inside `complexity_router_config` on the [Auto Router](./proxy/auto_routing.md) page is a different setting. It pins the auto router's model choice for a session; the pre-call check on this page pins a deployment inside a model group. ::: ## Traffic Mirroring / Silent Experiments Traffic mirroring allows you to "mimic" production traffic to a secondary (silent) model for evaluation purposes. The silent model's response is gathered in the background and does not affect the latency or result of the primary request. [**See detailed guide on A/B Testing - Traffic Mirroring here**](./traffic_mirroring.md) ## Basic Reliability ### Deployment Ordering (Priority) Set `order` in `litellm_params` to prioritize deployments. Lower values = higher priority. When multiple deployments share the same `order`, the routing strategy picks among them. When a request to an `order=1` deployment fails (connection error, 404, 429, etc.), the router automatically tries `order=2` deployments, then `order=3`, and so on. Each order level gets its own set of retries before escalating to the next. If all order levels are exhausted, the router falls through to any configured fallbacks. **SDK** ```python from litellm import Router model_list = [ { "model_name": "gpt-5.6-terra", "litellm_params": { "model": "azure/gpt-4-primary", "api_key": os.getenv("AZURE_API_KEY"), "order": 1, # 👈 Highest priority }, }, { "model_name": "gpt-5.6-terra", "litellm_params": { "model": "azure/gpt-4-fallback", "api_key": os.getenv("AZURE_API_KEY_2"), "order": 2, # 👈 Tried when order=1 fails }, }, ] router = Router(model_list=model_list) ``` **PROXY** ```yaml model_list: - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-4-primary api_key: os.environ/AZURE_API_KEY order: 1 # 👈 Highest priority - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-4-fallback api_key: os.environ/AZURE_API_KEY_2 order: 2 # 👈 Tried when order=1 fails ``` ### Weighted Deployments Set `weight` on a deployment to pick one deployment more often than others. This works across **simple-shuffle** routing strategy (this is the default, if no routing strategy is selected). **SDK** ```python from litellm import Router model_list = [ { "model_name": "gpt-5.6-luna", "litellm_params": { "model": "gpt-5.6-luna", "api_key": os.getenv("OPENAI_API_KEY"), "weight": 1 }, }, { "model_name": "gpt-5.6-luna", "litellm_params": { "model": "gpt-5.6-luna", "api_key": os.getenv("OPENAI_API_KEY"), "weight": 2 # 👈 PICK THIS DEPLOYMENT 2x MORE OFTEN THAN THE ONE ABOVE }, }, ] router = Router(model_list=model_list, routing_strategy="cost-based-routing") response = await router.acompletion( model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}] ) print(response) ``` **PROXY** ```yaml model_list: - model_name: gpt-5.6-luna litellm_params: model: gpt-5.6-luna api_key: os.environ/OPENAI_API_KEY weight: 1 - model_name: gpt-5.6-luna litellm_params: model: gpt-5.6-luna api_key: os.environ/OPENAI_API_KEY weight: 2 # 👈 PICK THIS DEPLOYMENT 2x MORE OFTEN THAN THE ONE ABOVE ``` ### Weighted Failover By default, when a deployment in a model group fails, the router moves on to the next entry in `fallbacks` (a different model group). With `enable_weighted_failover`, the router first retries **inside the same model group** by re-picking a different deployment using the existing weights, and only escalates to cross-group fallbacks once every deployment in the group has been tried. This is useful when you have multiple regional copies of the same model (e.g. Azure `eastus2` + `swedencentral`) and want a failed region to fail over to a healthy peer with the same `model_name`, instead of immediately switching to a different model. **Behavior** - Only active when `routing_strategy="simple-shuffle"` (the default). - On a retryable failure, the failing deployment ID is excluded and a new deployment is picked from the remaining peers in the same model group, respecting `weight` / `rpm` / `tpm`. - Exclusions accumulate across hops: each retry adds the previous failure to the exclusion set, so a deployment that just failed is never picked again in the same request chain. - Capped by `max_fallbacks` (default `5`). - Not triggered for `ContextWindowExceededError` or `ContentPolicyViolationError`, which keep their dedicated fallback paths. - Async-only: honored by `router.acompletion()` and other async entrypoints. The sync `router.completion()` path falls through to regular fallbacks. - Cooldowns still apply: a deployment that crosses `allowed_fails` is cooled down independently of weighted failover. **Order vs. weight** If the same group also uses `order`, the order filter runs **before** the weighted pick. So weighted failover re-picks only among the deployments in the current minimum-order tier. Promotion to the next order tier happens through the existing order-based fallback path. **Config** **SDK** ```python from litellm import Router model_list = [ { "model_name": "gpt-5.6-luna", "litellm_params": { "model": "azure/gpt-5.6-luna", "api_base": "https://eastus2.example.azure.com", "api_key": os.getenv("AZURE_EASTUS2_KEY"), "weight": 1, }, }, { "model_name": "gpt-5.6-luna", "litellm_params": { "model": "azure/gpt-5.6-luna", "api_base": "https://swedencentral.example.azure.com", "api_key": os.getenv("AZURE_SWEDEN_KEY"), "weight": 1, }, }, ] router = Router( model_list=model_list, routing_strategy="simple-shuffle", enable_weighted_failover=True, # 👈 retry within the same model group on failure ) response = await router.acompletion( model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey"}], ) ``` **PROXY** ```yaml model_list: - model_name: gpt-5.6-luna litellm_params: model: azure/gpt-5.6-luna api_base: https://eastus2.example.azure.com api_key: os.environ/AZURE_EASTUS2_KEY weight: 1 - model_name: gpt-5.6-luna litellm_params: model: azure/gpt-5.6-luna api_base: https://swedencentral.example.azure.com api_key: os.environ/AZURE_SWEDEN_KEY weight: 1 router_settings: routing_strategy: simple-shuffle enable_weighted_failover: true # 👈 retry within the same model group on failure ``` **Walkthrough** With the config above and a request to `gpt-5.6-luna`: 1. `simple-shuffle` picks one of the two deployments using `weight`. 2. If the picked deployment raises a provider error (e.g. `RateLimitError`, `InternalServerError`), its deployment ID is added to `metadata._failover_excluded_ids`. 3. The router re-enters `simple-shuffle` with the failed deployment excluded and weights renormalized over what's left. 4. Steps 2–3 repeat until a deployment succeeds, every peer has been excluded, or `max_fallbacks` is reached. 5. Only after all peers are exhausted does the router fall through to any `fallbacks` configured for the group. See [`enable_weighted_failover`](./proxy/config_settings#router_settings---reference) in the router settings reference for the flag. ### Max Parallel Requests (ASYNC) Limit the max concurrent calls made to a deployment. Useful in high-traffic scenarios. If tpm/rpm is set, and no max parallel request limit given, we use the RPM or calculated RPM as the max parallel request limit. The precedence is `max_parallel_requests`, then `rpm`, then `int(tpm / 1000 * 6)` (six concurrent requests per 1000 TPM, minimum 1), then the router's `default_max_parallel_requests`. This means a deployment with only `rpm: 2` set gets a per-process concurrency cap of 2, whatever its provider is. The cap is counted in process, so it is per proxy worker and is not shared across workers or pods A request that arrives while every slot of the deployment is in use fails right away with a 429 whose body names the deployment and its `max_parallel_requests`. There is no wait queue and nothing to configure for one: requests never sit in the proxy waiting for a slot, they either run or get the 429. The 429 is raised before any provider call, so it does not count towards the deployment's cooldown. The router's own retries and fallbacks treat it like any other 429, so a model group with a second deployment fails over to it, and a single deployment is retried `num_retries` times before the caller sees the error. Set `num_retries: 0` if callers should see the rejection immediately Earlier versions queued instead of rejecting: a request over the cap waited, for as long as it took, until a slot freed up, so a deployment with `rpm` or `tpm` set could silently serialize traffic with 200s and long latencies instead of 429s. If you relied on that, raise `max_parallel_requests` (or the `rpm`/`tpm` it is derived from), or handle the 429 in the caller ```python from litellm import Router model_list = [{ "model_name": "gpt-5.6-terra", "litellm_params": { "model": "azure/gpt-5.6-terra", # ... "max_parallel_requests": 10 # 👈 SET PER DEPLOYMENT } }] ### OR ### router = Router(model_list=model_list, default_max_parallel_requests=20) # 👈 SET DEFAULT MAX PARALLEL REQUESTS # deployment max parallel requests > default max parallel requests ``` On the proxy, set `max_parallel_requests` per deployment under `litellm_params` and the default under `router_settings`: ```yaml model_list: - model_name: gpt-5.6-terra litellm_params: model: openai/gpt-5.6-terra api_key: os.environ/OPENAI_API_KEY rpm: 2 # derives max_parallel_requests=2 router_settings: num_retries: 0 # optional, surfaces the 429 to the caller instead of retrying it default_max_parallel_requests: 20 # applies to deployments with no max_parallel_requests, rpm or tpm of their own ``` With the config above, four requests sent at the same time to `gpt-5.6-terra` get two 200s and two immediate 429s: ```json {"error":{"message":"litellm.RateLimitError: Deployment has all max_parallel_requests slots in use. Deployment model_group=gpt-5.6-terra, id=... already has max_parallel_requests=2 requests in flight. Raise max_parallel_requests (or the rpm/tpm it is derived from) for this deployment. Received Model Group=gpt-5.6-terra\nAvailable Model Group Fallbacks=None","type":"throttling_error","param":null,"code":"429"}} ``` [**See Code**](https://github.com/BerriAI/litellm/blob/a978f2d8813c04dad34802cb95e0a0e35a3324bc/litellm/utils.py#L5605) ### Cooldowns Set the limit for how many calls a model is allowed to fail in a minute, before being cooled down for a minute. **SDK** ```python from litellm import Router model_list = [{...}] router = Router(model_list=model_list, allowed_fails=1, # cooldown model if it fails > 1 call in a minute. cooldown_time=100 # cooldown the deployment for 100 seconds if it num_fails > allowed_fails ) user_message = "Hello, whats the weather in San Francisco??" messages = [{"content": user_message, "role": "user"}] # normal call response = router.completion(model="gpt-5.6-luna", messages=messages) print(f"response: {response}") ``` **PROXY** **Set Global Value** ```yaml router_settings: allowed_fails: 3 # cooldown model if it fails > 1 call in a minute. cooldown_time: 30 # (in seconds) how long to cooldown model if fails/min > allowed_fails ``` Defaults: - allowed_fails: 3 - cooldown_time: 5s (`DEFAULT_COOLDOWN_TIME_SECONDS` in constants.py) **Set Per Model** `allowed_fails` and `cooldown_time` can also be set on a single deployment instead of the whole router. A deployment-level value overrides the router-level one for that deployment only, so a flaky third-party endpoint can get a shorter fuse than the rest of your fleet without affecting them. ```yaml model_list: - model_name: fake-openai-endpoint litellm_params: model: predibase/llama-3-8b-instruct api_key: os.environ/PREDIBASE_API_KEY tenant_id: os.environ/PREDIBASE_TENANT_ID max_new_tokens: 256 model_info: allowed_fails: 1 # cool this deployment down after 1 fail, instead of the router default cooldown_time: 0 # disable cooldowns for this deployment ``` `allowed_fails` must be set under `model_info`, not `litellm_params`: unlike `model_info`, `litellm_params` is copied into the actual request sent to the LLM provider, so a router-only setting placed there would leak into that request. `cooldown_time` can be set under either location (`model_info` takes priority if both are set), matching its pre-existing behavior on the router's primary failure path. **Expected Response** ``` No deployments available for selected model, Try again in 60 seconds. Passed model=claude-sonnet-5. pre-call-checks=False, allowed_model_region=n/a. ``` #### **Disable cooldowns** **SDK** ```python from litellm import Router router = Router(..., disable_cooldowns=True) ``` **PROXY** ```yaml router_settings: disable_cooldowns: True ``` ### How Cooldowns Work Cooldowns apply to individual deployments, not entire model groups. The router isolates failures to specific deployments while keeping healthy alternatives available. #### What is a deployment? A deployment is a single entry in your `config.yaml` model list. Each deployment represents a unique configuration with its own `litellm_params`. LiteLLM generates a unique `model_id` for each deployment by creating a deterministic hash of all the `litellm_params`. This allows the router to track and manage each deployment independently. **Example: Multiple deployments for the same model** ```yaml showLineNumbers title="Load Balancing config.yaml" model_list: - model_name: sonnet-4 # Deployment 1 litellm_params: model: anthropic/claude-sonnet-5 api_key: - model_name: byok-sonnet-4 # Deployment 2 litellm_params: model: anthropic/claude-sonnet-5 api_key: api_base: https://proxy.litellm.ai/api.anthropic.com - model_name: sonnet-4 # Deployment 3 litellm_params: model: vertex_ai/claude-sonnet-5 vertex_project: my-project ``` Each deployment gets a unique `model_id` (e.g., `1234567890`, `9129922`, `4982929292`) that the router uses for tracking health and cooldown status. #### When are deployments cooled down? The router automatically cools down deployments based on the following conditions: | Condition | Trigger | Cooldown Duration | |-----------|---------|-------------------| | **Rate Limiting (429)** | Immediate on 429 response | 5 seconds (default) | | **High Failure Rate** | >50% failures in current minute | 5 seconds (default) | | **Non-Retryable Errors** | 401 (Auth), 404 (Not Found), 408 (Timeout) | 5 seconds (default) | During cooldown, the specific deployment is temporarily removed from the available pool, while other healthy deployments continue serving requests. #### Cooldown Recovery Deployments automatically recover from cooldown after the cooldown period expires. The router will: 1. **Monitor cooldown timers** for each deployment 2. **Automatically re-enable** deployments when cooldown expires 3. **Gradually reintroduce** cooled-down deployments to the rotation 4. **Reset failure counters** once the deployment is healthy again #### Real-World Example Consider this high-availability setup with multiple providers: ```yaml showLineNumbers title="Load Balancing config.yaml" model_list: - model_name: sonnet-4 # Primary: Anthropic Direct litellm_params: model: anthropic/claude-sonnet-5 api_key: - model_name: byok-sonnet-4 # BYOK: Customer-managed keys litellm_params: model: anthropic/claude-sonnet-5 api_key: api_base: https://proxy.litellm.ai/api.anthropic.com - model_name: sonnet-4 # Fallback: Vertex AI litellm_params: model: vertex_ai/claude-sonnet-5 vertex_project: my-project ``` **Failure Scenario:** ```mermaid flowchart TD A["Request for 'sonnet-4'"] --> B["Router finds available deployments"] B --> C["Available:
• Anthropic Direct
• Vertex AI"] C --> D["Selects Anthropic Direct"] D --> E{"Request fails with 429?"} E -->|No| F["Success ✅"] E -->|Yes| G["Cooldown Anthropic Direct
for 5 seconds"] G --> H["Next request for 'sonnet-4'"] H --> I["Route to Vertex AI
(only available deployment for model_name='sonnet-4')"] I --> J["Success ✅"] style G fill:#ffcccc style I fill:#ccffcc ``` ### Retries For both async + sync functions, we support retrying failed requests. For RateLimitError we implement exponential backoffs For generic errors, we retry immediately Here's a quick look at how we can set `num_retries = 3`: ```python from litellm import Router model_list = [{...}] router = Router(model_list=model_list, num_retries=3) user_message = "Hello, whats the weather in San Francisco??" messages = [{"content": user_message, "role": "user"}] # normal call response = router.completion(model="gpt-5.6-luna", messages=messages) print(f"response: {response}") ``` We also support setting minimum time to wait before retrying a failed request. This is via the `retry_after` param. ```python from litellm import Router model_list = [{...}] router = Router(model_list=model_list, num_retries=3, retry_after=5) # waits min 5s before retrying request user_message = "Hello, whats the weather in San Francisco??" messages = [{"content": user_message, "role": "user"}] # normal call response = router.completion(model="gpt-5.6-luna", messages=messages) print(f"response: {response}") ``` #### Where `num_retries` can be set, and which one wins `num_retries` can come from four places. They are ranked, highest first: 1. the `x-litellm-num-retries` request header (proxy only) 2. `num_retries` in the request body 3. `num_retries` in a deployment's `litellm_params` in `model_list` 4. `num_retries` in `litellm_settings` (the router-wide default) So a caller can always raise or lower the retry count for one request, including setting it to `0` to disable retries, no matter what the deployment or the global setting says. A deployment value applies whenever the request carries none, and it overrides the global default. `num_retries` is not the same knob as `max_retries`. `num_retries` is LiteLLM's own retry loop, while `max_retries` is the provider SDK's internal retry count. For a call that goes through the router, LiteLLM owns retries and pins the provider client to `max_retries: 0`, so a `max_retries` in the request body or in `litellm_params` has no effect on a proxy request. That is deliberate: it is what stops a deployment `num_retries: N` from being applied twice and turning one request into `(1 + N) ** 2` upstream calls. Use `num_retries` to control how many attempts a request gets. ### [Advanced]: Custom Retries, Cooldowns based on Error Type - Use `RetryPolicy` if you want to set a `num_retries` based on the Exception received - Use `AllowedFailsPolicy` to set a custom number of `allowed_fails`/minute before cooling down a deployment `RetryPolicy` takes one field per error type (`AuthenticationErrorRetries`, `TimeoutErrorRetries`, `RateLimitErrorRetries`, `ContentPolicyViolationErrorRetries`, `BadRequestErrorRetries`, `NotFoundErrorRetries`, `InternalServerErrorRetries`, `ServiceUnavailableErrorRetries`) plus `DefaultRetries` for every error none of those cover. The most specific field wins: `NotFoundErrorRetries` governs any 404 answer, whatever exception class the provider's error body mapped to, `BadRequestErrorRetries` then covers a 4xx the provider reported as an invalid request, and `DefaultRetries` applies last. A field left unset defers to the next one, so a policy that only sets `DefaultRetries` retries 404s too; set `NotFoundErrorRetries: 0` to leave them alone. [**See All Exception Types**](https://github.com/BerriAI/litellm/blob/ccda616f2f881375d4e8586c76fe4662909a7d22/litellm/types/router.py#L436) **SDK** Example: ```python retry_policy = RetryPolicy( ContentPolicyViolationErrorRetries=3, # run 3 retries for ContentPolicyViolationErrors AuthenticationErrorRetries=0, # run 0 retries for AuthenticationErrorRetries NotFoundErrorRetries=0, # never retry a 404 (a deleted response id, an unknown deployment name) DefaultRetries=2, # run 2 retries for every error with no field of its own ) allowed_fails_policy = AllowedFailsPolicy( ContentPolicyViolationErrorAllowedFails=1000, # Allow 1000 ContentPolicyViolationError before cooling down a deployment RateLimitErrorAllowedFails=100, # Allow 100 RateLimitErrors before cooling down a deployment ) ``` Example Usage ```python from litellm.router import RetryPolicy, AllowedFailsPolicy retry_policy = RetryPolicy( ContentPolicyViolationErrorRetries=3, # run 3 retries for ContentPolicyViolationErrors AuthenticationErrorRetries=0, # run 0 retries for AuthenticationErrorRetries BadRequestErrorRetries=1, TimeoutErrorRetries=2, RateLimitErrorRetries=3, NotFoundErrorRetries=0, ServiceUnavailableErrorRetries=2, DefaultRetries=1, ) allowed_fails_policy = AllowedFailsPolicy( ContentPolicyViolationErrorAllowedFails=1000, # Allow 1000 ContentPolicyViolationError before cooling down a deployment RateLimitErrorAllowedFails=100, # Allow 100 RateLimitErrors before cooling down a deployment ) router = litellm.Router( model_list=[ { "model_name": "gpt-5.6-luna", # openai model name "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), }, }, { "model_name": "bad-model", # openai model name "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", "api_key": "bad-key", "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), }, }, ], retry_policy=retry_policy, allowed_fails_policy=allowed_fails_policy, ) response = await router.acompletion( model=model, messages=messages, ) ``` **PROXY** ```yaml router_settings: retry_policy: { "BadRequestErrorRetries": 3, "ContentPolicyViolationErrorRetries": 4, "NotFoundErrorRetries": 0, # never retry a 404 "DefaultRetries": 2 # retries for every error with no field of its own } allowed_fails_policy: { "ContentPolicyViolationErrorAllowedFails": 1000, # Allow 1000 ContentPolicyViolationError before cooling down a deployment "RateLimitErrorAllowedFails": 100 # Allow 100 RateLimitErrors before cooling down a deployment } ``` `AllowedFailsPolicy` also supports `ServiceUnavailableErrorAllowedFails`, `BadGatewayErrorAllowedFails`, and `NotFoundErrorAllowedFails`. #### Per-deployment allowed_fails_policy `allowed_fails_policy` can be scoped to a single deployment by setting it under that deployment's `model_info` instead of `router_settings`. A deployment-level policy takes full precedence over the router-level one for that deployment, so a rate-limited third-party endpoint can cool down after its first `RateLimitError` while the rest of your fleet keeps the router-wide tolerance. ```yaml model_list: - model_name: gpt-5.6-terra litellm_params: model: openai/gpt-5.6-terra api_key: os.environ/OPENAI_API_KEY model_info: allowed_fails_policy: RateLimitErrorAllowedFails: 0 # cool down after the first RateLimitError InternalServerErrorAllowedFails: 5 ``` ### Caching In production, we recommend using a Redis cache. For quickly testing things locally, we also support simple in-memory caching. **In-memory Cache** ```python router = Router(model_list=model_list, cache_responses=True) print(response) ``` **Redis Cache** ```python router = Router(model_list=model_list, redis_host=os.getenv("REDIS_HOST"), redis_password=os.getenv("REDIS_PASSWORD"), redis_port=os.getenv("REDIS_PORT"), cache_responses=True) print(response) ``` **Pass in Redis URL, additional kwargs** ```python router = Router(model_list=model_list, ## CACHING ## redis_url=os.getenv("REDIS_URL"), cache_kwargs= {}, # additional kwargs to pass to RedisCache (see caching.py) cache_responses=True) ``` :::info When configuring Redis caching in router settings, use `cache_kwargs` to pass additional Redis parameters, especially for non-string values that may fail when set via `REDIS_*` environment variables. ::: ## Pre-Call Checks (Context Window, EU-Regions) Enable pre-call checks to filter out: 1. deployments with context window limit < messages for a call. 2. deployments outside of eu-region **SDK** **1. Enable pre-call checks** ```python from litellm import Router # ... router = Router(model_list=model_list, enable_pre_call_checks=True) # 👈 Set to True ``` **2. Set Model List** For context window checks on azure deployments, set the base model. Pick the base model from [this list](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json), all the azure models start with `azure/`. For 'eu-region' filtering, Set 'region_name' of deployment. **Note:** We automatically infer region_name for Vertex AI, Bedrock, and IBM WatsonxAI based on your litellm params. For Azure, set `litellm.enable_preview = True`. [**See Code**](https://github.com/BerriAI/litellm/blob/d33e49411d6503cb634f9652873160cd534dec96/litellm/router.py#L2958) ```python model_list = [ { "model_name": "gpt-5.6-luna", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "region_name": "eu", # 👈 SET 'EU' REGION NAME "base_model": "azure/gpt-5.6-luna", # 👈 (Azure-only) SET BASE MODEL }, }, { "model_name": "gpt-5.6-luna", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "gpt-5.6-luna", "api_key": os.getenv("OPENAI_API_KEY"), }, }, { "model_name": "gemini-3.1-pro-preview", "litellm_params": { "model": "vertex_ai/gemini-3.1-pro-preview", "vertex_project": "adroit-crow-1234", "vertex_location": "us-east1" # 👈 AUTOMATICALLY INFERS 'region_name' } } ] router = Router(model_list=model_list, enable_pre_call_checks=True) ``` **3. Test it!** **Context Window Check** The model ids in this example are illustrative and kept for their context window sizes. ```python keep-model-ids """ - Give a gpt-3.5-turbo model group with different context windows (4k vs. 16k) - Send a 5k prompt - Assert it works """ from litellm import Router import os model_list = [ { "model_name": "gpt-3.5-turbo", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "base_model": "azure/gpt-35-turbo", }, "model_info": { "base_model": "azure/gpt-35-turbo", } }, { "model_name": "gpt-3.5-turbo", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "gpt-3.5-turbo-1106", "api_key": os.getenv("OPENAI_API_KEY"), }, }, ] router = Router(model_list=model_list, enable_pre_call_checks=True) text = "What is the meaning of 42?" * 5000 response = router.completion( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": text}, {"role": "user", "content": "Who was Alexander?"}, ], ) print(f"response: {response}") ``` **EU Region Check** ```python """ - Give 2 gpt-5.6-luna deployments, in eu + non-eu regions - Make a call - Assert it picks the eu-region model """ from litellm import Router import os model_list = [ { "model_name": "gpt-5.6-luna", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "region_name": "eu" }, "model_info": { "id": "1" } }, { "model_name": "gpt-5.6-luna", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "gpt-5.6-luna", "api_key": os.getenv("OPENAI_API_KEY"), }, "model_info": { "id": "2" } }, ] router = Router(model_list=model_list, enable_pre_call_checks=True) response = router.completion( model="gpt-5.6-luna", messages=[{"role": "user", "content": "Who was Alexander?"}], ) print(f"response: {response}") print(f"response id: {response._hidden_params['model_id']}") ``` **Proxy** :::info Go [here](./proxy/reliability.md#context-window-fallbacks) for how to do this on the proxy ::: ## Caching across model groups If you want to cache across 2 different model groups (e.g. azure deployments, and openai), use caching groups. ```python import litellm, asyncio, time from litellm import Router # set os env os.environ["OPENAI_API_KEY"] = "" os.environ["AZURE_API_KEY"] = "" os.environ["AZURE_API_BASE"] = "" os.environ["AZURE_API_VERSION"] = "" async def test_acompletion_caching_on_router_caching_groups(): # tests acompletion + caching on router try: litellm.set_verbose = True model_list = [ { "model_name": "openai-gpt-4o-mini", "litellm_params": { "model": "gpt-5.6-luna", "api_key": os.getenv("OPENAI_API_KEY"), }, }, { "model_name": "azure-gpt-4o-mini", "litellm_params": { "model": "azure/chatgpt-v-2", "api_key": os.getenv("AZURE_API_KEY"), "api_base": os.getenv("AZURE_API_BASE"), "api_version": os.getenv("AZURE_API_VERSION") }, } ] messages = [ {"role": "user", "content": f"write a one sentence poem {time.time()}?"} ] start_time = time.time() router = Router(model_list=model_list, cache_responses=True, caching_groups=[("openai-gpt-4o-mini", "azure-gpt-4o-mini")]) response1 = await router.acompletion(model="openai-gpt-4o-mini", messages=messages, temperature=1) print(f"response1: {response1}") await asyncio.sleep(1) # add cache is async, async sleep for cache to get set response2 = await router.acompletion(model="azure-gpt-4o-mini", messages=messages, temperature=1) assert response1.id == response2.id assert len(response1.choices[0].message.content) > 0 assert response1.choices[0].message.content == response2.choices[0].message.content except Exception as e: traceback.print_exc() asyncio.run(test_acompletion_caching_on_router_caching_groups()) ``` ## Alerting 🚨 Send alerts to slack / your webhook url for the following events - LLM API Exceptions - Slow LLM Responses Get a slack webhook url from https://api.slack.com/messaging/webhooks #### Usage Initialize an `AlertingConfig` and pass it to `litellm.Router`. The following code will trigger an alert because `api_key=bad-key` which is invalid ```python import litellm from litellm.router import Router from litellm.types.router import AlertingConfig import os import asyncio router = Router( model_list=[ { "model_name": "gpt-5.6-luna", "litellm_params": { "model": "gpt-5.6-luna", "api_key": "bad_key", }, } ], alerting_config= AlertingConfig( alerting_threshold=10, webhook_url= "https:/..." ), ) async def main(): print(f"\n=== Configuration ===") print(f"Slack logger exists: {router.slack_alerting_logger is not None}") try: await router.acompletion( model="gpt-5.6-luna", messages=[{"role": "user", "content": "Hey, how's it going?"}], ) except Exception as e: print(f"\n=== Exception caught ===") print(f"Waiting 10 seconds for alerts to be sent via periodic flush...") await asyncio.sleep(10) print(f"\n=== After waiting ===") print(f"Alert should have been sent to Slack!") asyncio.run(main()) ``` ## Track cost for Azure Deployments **Problem**: Azure returns `gpt-4` in the response when `azure/gpt-4-1106-preview` is used. This leads to inaccurate cost tracking **Solution** ✅ : Set `model_info["base_model"]` on your router init so litellm uses the correct model for calculating azure cost Step 1. Router Setup ```python from litellm import Router model_list = [ { # list of model deployments "model_name": "gpt-5.6-luna", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") }, "model_info": { "base_model": "azure/gpt-5.6-luna" # azure/gpt-5.6-luna will be used for cost tracking, ensure this exists in litellm model_prices_and_context_window.json } }, { "model_name": "gpt-5.6-terra", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") }, "model_info": { "base_model": "azure/gpt-5.6-terra" # azure/gpt-5.6-terra will be used for cost tracking, ensure this exists in litellm model_prices_and_context_window.json } } ] router = Router(model_list=model_list) ``` Step 2. Access `response_cost` in the custom callback, **litellm calculates the response cost for you** ```python import litellm from litellm.integrations.custom_logger import CustomLogger class MyCustomHandler(CustomLogger): def log_success_event(self, kwargs, response_obj, start_time, end_time): print(f"On Success") response_cost = kwargs.get("response_cost") print("response_cost=", response_cost) customHandler = MyCustomHandler() litellm.callbacks = [customHandler] # router completion call response = router.completion( model="gpt-5.6-terra", messages=[{ "role": "user", "content": "Hi who are you"}] ) ``` #### Default litellm.completion/embedding params You can also set default params for litellm completion/embedding calls. Here's how to do that: The model ids in this example are illustrative and kept for their context window sizes. ```python keep-model-ids from litellm import Router fallback_dict = {"gpt-4o-mini": "gpt-4.1"} router = Router(model_list=model_list, default_litellm_params={"context_window_fallback_dict": fallback_dict}) user_message = "Hello, whats the weather in San Francisco??" messages = [{"content": user_message, "role": "user"}] # normal call response = router.completion(model="gpt-4o-mini", messages=messages) print(f"response: {response}") ``` ## Custom Callbacks - Track API Key, API Endpoint, Model Used If you need to track the api_key, api endpoint, model, custom_llm_provider used for each completion call, you can setup a [custom callback](https://docs.litellm.ai/docs/observability/custom_callback) ### Usage ```python import litellm from litellm.integrations.custom_logger import CustomLogger class MyCustomHandler(CustomLogger): def log_success_event(self, kwargs, response_obj, start_time, end_time): print(f"On Success") print("kwargs=", kwargs) litellm_params= kwargs.get("litellm_params") api_key = litellm_params.get("api_key") api_base = litellm_params.get("api_base") custom_llm_provider= litellm_params.get("custom_llm_provider") response_cost = kwargs.get("response_cost") # print the values print("api_key=", api_key) print("api_base=", api_base) print("custom_llm_provider=", custom_llm_provider) print("response_cost=", response_cost) def log_failure_event(self, kwargs, response_obj, start_time, end_time): print(f"On Failure") print("kwargs=") customHandler = MyCustomHandler() litellm.callbacks = [customHandler] # Init Router router = Router(model_list=model_list, routing_strategy="simple-shuffle") # router completion call response = router.completion( model="gpt-5.6-luna", messages=[{ "role": "user", "content": "Hi who are you"}] ) ``` ## Deploy Router If you want a server to load balance across different LLM APIs, use our [LiteLLM Proxy Server](/docs/simple_proxy) ## Debugging Router ### Basic Debugging Set `Router(set_verbose=True)` ```python from litellm import Router router = Router( model_list=model_list, set_verbose=True ) ``` ### Detailed Debugging Set `Router(set_verbose=True,debug_level="DEBUG")` ```python from litellm import Router router = Router( model_list=model_list, set_verbose=True, debug_level="DEBUG" # defaults to INFO ) ``` ### Very Detailed Debugging Set `litellm.set_verbose=True` and `Router(set_verbose=True,debug_level="DEBUG")` ```python from litellm import Router import litellm litellm.set_verbose = True router = Router( model_list=model_list, set_verbose=True, debug_level="DEBUG" # defaults to INFO ) ``` ## Router General Settings ### Usage ```python router = Router(model_list=..., router_general_settings=RouterGeneralSettings(async_only_mode=True)) ``` ### Spec ```python class RouterGeneralSettings(BaseModel): async_only_mode: bool = Field( default=False ) # this will only initialize async clients. Good for memory utils pass_through_all_models: bool = Field( default=False ) # if passed a model not llm_router model list, pass through the request to litellm.acompletion/embedding ``` ## Related pages - [Routing & Load Balancing](https://docs.litellm.ai/docs/routing-load-balancing.md) - [Fusion model](https://docs.litellm.ai/docs/fusion.md)