--- title: "Proxy - Load Balancing" url: "/docs/proxy/load_balancing" canonical_url: "https://docs.litellm.ai/docs/proxy/load_balancing" type: "docs" last_updated: "2026-10-14" summary: "Load balance multiple instances of the same model" related: - "/docs/guides/" - "/docs/routing" --- # Proxy - Load Balancing > Index of all LiteLLM docs: https://docs.litellm.ai/llms.txt Load balance multiple instances of the same model The proxy will handle routing requests (using LiteLLM's Router). **Set `rpm` in the config if you want maximize throughput** :::info For more details on routing strategies / params, see [Routing](../routing.md) ::: ## How Load Balancing Works LiteLLM automatically distributes requests across multiple deployments of the same model using its built-in router. the proxy routes traffic to optimize performance and reliability. "simple-shuffle" routing strategy is used by default ### Routing Strategies | Strategy | Description | When to Use | |----------|-------------|-------------| | **simple-shuffle** (recommended) | Randomly distributes requests | General purpose, good for even load distribution | | **least-busy** | Routes to deployment with fewest active requests | High concurrency scenarios | | **usage-based-routing** (bad for perf) | Routes to deployment with lowest current usage (RPM/TPM) | When you want to respect rate limits evenly | | **latency-based-routing** | Routes to fastest responding deployment | Latency-critical applications | | **cost-based-routing** | Routes to deployment with lowest cost | Cost-sensitive applications | :::tip[Deployment Priority] Use the `order` parameter to prioritize specific deployments. [See Deployment Ordering](#deployment-ordering-priority) for details. ::: ## Quick Start - Load Balancing #### Step 1 - Set deployments on config **Example config below**. Here requests with `model=gpt-5.6-luna` will be routed across multiple instances of `azure/gpt-5.6-luna` ```yaml model_list: - model_name: gpt-5.6-luna litellm_params: model: azure/ api_base: api_key: rpm: 6 # Rate limit for this deployment: in requests per minute (rpm) - model_name: gpt-5.6-luna litellm_params: model: azure/gpt-turbo-small-ca api_base: https://my-endpoint-canada-berri992.openai.azure.com/ api_key: rpm: 6 - model_name: gpt-5.6-luna litellm_params: model: azure/gpt-turbo-large api_base: https://openai-france-1234.openai.azure.com/ api_key: rpm: 1440 router_settings: routing_strategy: simple-shuffle # Literal["simple-shuffle", "least-busy", "usage-based-routing","latency-based-routing"], default="simple-shuffle" model_group_alias: {"gpt-5.6-terra": "gpt-5.6-luna"} # all requests with `gpt-5.6-terra` will be routed to models with `gpt-5.6-luna` num_retries: 2 timeout: 30 # 30 seconds redis_host: # set this when using multiple litellm proxy deployments, load balancing state stored in redis redis_password: redis_port: 1992 ``` ## Enforce Model Rate Limits Strictly enforce RPM/TPM limits set on deployments. When limits are exceeded, requests are blocked **before** reaching the LLM provider with a `429 Too Many Requests` error. :::tip[Separate input/output limits] Set `itpm` and `otpm` instead of `tpm`/`rpm` when a provider publishes distinct input and output throughput limits. See [Separate ITPM / OTPM Rate Limits](./io_token_rate_limits). ::: :::info By default, `rpm` and `tpm` values are only used for **routing decisions** (picking deployments with capacity). With `enforce_model_rate_limits`, they become **hard limits**. ::: ### Quick Start ```yaml model_list: - model_name: gpt-5.6-terra litellm_params: model: openai/gpt-5.6-terra api_key: os.environ/OPENAI_API_KEY rpm: 60 # 60 requests per minute tpm: 90000 # 90k tokens per minute router_settings: optional_pre_call_checks: - enforce_model_rate_limits # 👈 Enables strict enforcement ``` ### How It Works | Limit Type | Enforcement | Accuracy | |------------|-------------|----------| | **RPM** | Hard limit - blocked at exact threshold | 100% accurate | | **TPM** | Best-effort - may slightly exceed | Blocked when already over limit | **Why TPM is best-effort:** Token count is unknown until the LLM responds. TPM is checked before each request (blocks if already over), and tracked after (adds actual tokens used). ### Error Response ```json { "error": { "message": "Model rate limit exceeded. RPM limit=60, current usage=60", "type": "rate_limit_error", "code": 429 } } ``` Response includes `retry-after: 60` header. ### Multi-Instance Deployment For multiple LiteLLM proxy instances, add Redis to share rate limit state: ```yaml router_settings: optional_pre_call_checks: - enforce_model_rate_limits redis_host: redis.example.com redis_port: 6379 redis_password: your-password ``` :::info Detailed information about [routing strategies can be found here](../routing) ::: #### Step 2: Start Proxy with config ```shell $ litellm --config /path/to/config.yaml ``` ### Test - Simple Call Here requests with model=gpt-5.6-luna will be routed across multiple instances of azure/gpt-5.6-luna 👉 Key Change: `model="gpt-5.6-luna"` **Check the `model_id` in Response Headers to make sure the requests are being load balanced** **OpenAI Python v1.0.0+** ```python import openai client = openai.OpenAI( api_key="anything", base_url="http://0.0.0.0:4000" ) response = client.chat.completions.create( model="gpt-5.6-luna", messages = [ { "role": "user", "content": "this is a test request, write a short poem" } ] ) print(response) ``` **Curl Request** ```shell curl --location 'http://0.0.0.0:4000/chat/completions' \ --header 'Content-Type: application/json' \ --data '{ "model": "gpt-5.6-luna", "messages": [ { "role": "user", "content": "what llm are you" } ] }' ``` ### Test - Loadbalancing In this request, the following will occur: 1. A rate limit exception will be raised 2. LiteLLM proxy will retry the request on the model group (default retries are 3). ```bash curl -X POST 'http://0.0.0.0:4000/chat/completions' \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $LITELLM_API_KEY" \ -d '{ "model": "gpt-5.6-luna", "messages": [ {"role": "user", "content": "Hi there!"} ], "mock_testing_rate_limit_error": true }' ``` [**See Code**](https://github.com/BerriAI/litellm/blob/6b8806b45f970cb2446654d2c379f8dcaa93ce3c/litellm/router.py#L2535) ## Load Balancing using multiple litellm instances (Kubernetes, Auto Scaling) LiteLLM Proxy supports sharing rpm/tpm shared across multiple litellm instances, pass `redis_host`, `redis_password` and `redis_port` to enable this. (LiteLLM will use Redis to track rpm/tpm usage ) Example config ```yaml model_list: - model_name: gpt-5.6-luna litellm_params: model: azure/ api_base: api_key: rpm: 6 # Rate limit for this deployment: in requests per minute (rpm) - model_name: gpt-5.6-luna litellm_params: model: azure/gpt-turbo-small-ca api_base: https://my-endpoint-canada-berri992.openai.azure.com/ api_key: rpm: 6 router_settings: redis_host: redis_password: redis_port: 1992 cache_params: type: redis max_connections: 100 # maximum Redis connections in the pool; tune based on expected concurrency/load ``` ## Router settings on config - routing_strategy, model_group_alias Expose an 'alias' for a 'model_name' on the proxy server. ``` model_group_alias: { "gpt-5.6-terra": "gpt-5.6-luna" } ``` These aliases are shown on `/v1/models`, `/v1/model/info`, and `/v1/model_group/info` by default. litellm.Router() settings can be set under `router_settings`. You can set `model_group_alias`, `routing_strategy`, `num_retries`,`timeout` . See all Router supported params [here](https://github.com/BerriAI/litellm/blob/1b942568897a48f014fa44618ec3ce54d7570a46/litellm/router.py#L64) ### Usage Example config with `router_settings` ```yaml model_list: - model_name: gpt-5.6-luna litellm_params: model: azure/ api_base: api_key: router_settings: model_group_alias: {"gpt-5.6-terra": "gpt-5.6-luna"} # all requests with `gpt-5.6-terra` will be routed to models ``` ### Hide Alias Models Use this if you want to set-up aliases for: 1. typos 2. minor model version changes 3. case sensitive changes between updates ```yaml model_list: - model_name: gpt-5.6-luna litellm_params: model: azure/ api_base: api_key: router_settings: model_group_alias: "GPT-5.6-luna": # alias model: "gpt-5.6-luna" # Actual model name in 'model_list' hidden: true # Exclude from `/v1/models`, `/v1/model/info`, `/v1/model_group/info` ``` To hide a `model_list` entry itself rather than an alias, set `model_info.discoverable: false` on it ([Hide a model from `/v1/models`](./model_discovery#hide-a-model-from-v1models)) ### Complete Spec ```python model_group_alias: Optional[Dict[str, Union[str, RouterModelGroupAliasItem]]] = {} class RouterModelGroupAliasItem(TypedDict): model: str hidden: bool # if 'True', don't return on `/v1/models`, `/v1/model/info`, `/v1/model_group/info` ``` ## Deployment Ordering (Priority) Set `order` in `litellm_params` to prioritize deployments. Lower values = higher priority. When multiple deployments share the same `order`, the routing strategy picks among them. ```yaml model_list: - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-4-primary api_key: os.environ/AZURE_API_KEY order: 1 # 👈 Highest priority - always tried first - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-4-fallback api_key: os.environ/AZURE_API_KEY_2 order: 2 # 👈 Used when order=1 fails ``` ### How order-based fallback works When a request to an `order=1` deployment fails (connection error, 404, 429, etc.), the router automatically tries `order=2` deployments, then `order=3`, and so on. Each order level gets its own set of retries before escalating to the next. If all order levels are exhausted, the router falls through to any configured [model-level fallbacks](./reliability.md). A hop to a deployment that cannot decrypt the failed deployment's encrypted reasoning (another provider, or another API key) drops those reasoning items and keeps their summaries, so the hop answers instead of failing with `invalid_encrypted_content`; see [fallbacks](./reliability.md#explanation) ```yaml model_list: - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-4-primary api_key: os.environ/AZURE_API_KEY order: 1 - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-4-secondary api_key: os.environ/AZURE_API_KEY_2 order: 2 - model_name: gpt-4-fallback litellm_params: model: openai/gpt-5.6-terra api_key: os.environ/OPENAI_API_KEY router_settings: fallbacks: - gpt-5.6-terra: - gpt-4-fallback # tried after all order levels fail ``` The fallback chain for the above config: `order=1` → `order=2` → `gpt-4-fallback`. For 429 (rate limit) errors specifically, the failed deployment is immediately placed on cooldown. If all `order=1` deployments are on cooldown, the router picks `order=2` deployments directly during retries without waiting for the fallback path. ### Team-scoped models and legacy `model_aliases` {#team-scoped-models-and-legacy-model_aliases} Team-scoped deployments are identified by `model_info.team_id` and `model_info.team_public_model_name`. Requests should use the **public** model name; the router resolves all sibling deployments (same public name, different `api_base` / `order`, etc.) for routing, failover, and deployment `order`. For router internals: when a `team_id` is in scope, optimized lookups key off `(team_id, team_public_model_name)`. If code passes an internal deployment id (e.g. `model_name__`) instead of the public name, routing still works via the usual deployment-name paths, but the team-specific fast path applies only to the public name. **Legacy teams:** Older proxy versions could persist `model_aliases` on the team row mapping a public name to a single internal deployment id (`model_name__`). On each request, pre-call logic may still rewrite `model` to that internal name **before** routing, which collapses to one deployment and can make newer sibling deployments unreachable. **Migration options:** 1. **Recommended for upgrades:** Set environment variable `LITELLM_ENABLE_TEAM_STALE_ALIAS_BYPASS=true` so that when sibling team deployments exist for the public name, the stale alias rewrite is skipped and team-scoped routing (including `order` and failover) applies. See the [Environment variables](./config_settings) table in the proxy settings doc. 2. **Data cleanup:** Remove obsolete `model_aliases` entries for team public names from the team record in the database so only `team_public_model_name` + team model list drive access. If a stale alias is detected and the bypass is **not** enabled, the proxy may emit a **one-time** warning in logs explaining that sibling deployments may be unreachable until the flag is set or aliases are cleaned up. ### When You'll See Load Balancing in Action **Immediate Effects:** - Different deployments serve subsequent requests (visible in logs) - Better response times during high traffic **Observable Benefits:** - **Higher throughput**: More requests handled simultaneously across deployments - **Improved reliability**: If one deployment fails, traffic automatically routes to healthy ones - **Better resource utilization**: Load spread evenly across all available deployments ## Special Considerations for Responses API When load balancing OpenAI's Responses API across deployments with **different API keys** (e.g., different Azure regions or organizations), encrypted content items (like `rs_...` reasoning items) can only be decrypted by the originating API key. **Solution:** Use the `encrypted_content_affinity` pre-call check (requires LiteLLM >= 1.82.3) to automatically route follow-up requests containing encrypted items to the correct deployment: ```yaml model_list: - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-5.6-terra api_base: https://eastus.openai.azure.com/ api_key: os.environ/AZURE_API_KEY_EASTUS model_info: id: "deployment-eastus" - model_name: gpt-5.6-terra litellm_params: model: azure/gpt-5.6-terra api_base: https://westeurope.openai.azure.com/ api_key: os.environ/AZURE_API_KEY_WESTEUROPE model_info: id: "deployment-westeurope" router_settings: optional_pre_call_checks: - encrypted_content_affinity # 👈 Prevents invalid_encrypted_content errors ``` This ensures requests containing encrypted content are routed to the deployment that created them, while other requests continue to load balance normally. When that deployment is not in the healthy pool for a follow-up (it is cooled down, it was removed, or the turn was routed to another model group) and no deployment has the same `api_base` and `api_key`, LiteLLM drops the encrypted reasoning from the request, keeps the rest of the conversation, forwards it to a healthy deployment, and logs a router warning, so the turn succeeds with fresh reasoning instead of failing A fallback hop does the same whether or not the check is on. An order-based hop to the next `order`, a configured `fallbacks` hop to another model group, and the retry of a Responses stream that broke mid-stream each drop the encrypted reasoning the hop target cannot decrypt and keep the summaries, with no per-deployment switch. Keep the check on for an ordered group of two providers all the same: it marks each reasoning item with the deployment that produced it, so a turn dispatched straight to `order: 2` while `order: 1` is cooled down (no hop, so no hop strip) drops the reasoning too, and the turn after a hop keeps the reasoning of the deployment that answers it instead of paying one failed call first **[Learn more about Encrypted Content Affinity →](../response_api.md#encrypted-content-affinity-multi-region-load-balancing)** ## Related pages - [Overview](https://docs.litellm.ai/docs/guides/index.md) - [Routing Strategies](https://docs.litellm.ai/docs/routing.md)