[BETA] Request Prioritization

info

Beta feature. Use for testing only.

Help us improve this

Prioritize LLM API requests in high-traffic.

• Add request to priority queue
• Poll queue, to check if request can be made. Returns 'True':
  • if there's healthy deployments
  • OR if request is at top of queue
• Priority - The lower the number, the higher the priority:
  • e.g. priority=0 > priority=2000

Supported Router endpoints:

• acompletion (/v1/chat/completions on Proxy)
• atext_completion (/v1/completions on Proxy)

Quick Start

from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "gpt-6-luna",
            "litellm_params": {
                "model": "gpt-6-luna",
                "mock_response": "Hello world this is Macintosh!", # fakes the LLM API call
                "rpm": 1,
            },
        },
    ],
    timeout=2, # timeout request if takes > 2s
    routing_strategy="simple-shuffle", # recommended for best performance
    polling_interval=0.03 # poll queue every 3ms if no healthy deployments
)

try:
    _response = await router.acompletion( # ADDS TO QUEUE + POLLS + MAKES CALL
        model="gpt-6-luna",
        messages=[{"role": "user", "content": "Hey!"}],
        priority=0, # LOWER IS BETTER
    )
except Exception as e:
    print("didn't make request")

LiteLLM Proxy

To prioritize requests on LiteLLM Proxy add priority to the request.

curl

curl -X POST 'http://localhost:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-D '{
    "model": "gpt-6-luna-fake-model",
    "messages": [
        {
        "role": "user",
        "content": "what is the meaning of the universe? 1234"
        }],
    "priority": 0 SET VALUE HERE
}'

OpenAI SDK

import openai
client = openai.OpenAI(
    api_key="anything",
    base_url="http://0.0.0.0:4000"
)

# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(
    model="gpt-6-luna",
    messages = [
        {
            "role": "user",
            "content": "this is a test request, write a short poem"
        }
    ],
    extra_body={ 
        "priority": 0 # SET VALUE HERE
    }
)

print(response)

Priority header

Set the x-litellm-priority header to an integer to set the scheduler's request queue priority, for clients that can add headers more easily than body fields. Lower values have higher priority. The header overrides a body priority value, and a value that is not an integer gets the same 400 error as a bad body priority. An empty header counts as unset. With drop_params on, a header value that is not an integer is ignored instead, so the body priority or default_priority applies. Since v1.109.0

The proxy reads the header on the chat completions and text completions routes, such as /v1/chat/completions, /v1/completions and the Azure-style /openai/deployments/{model}/chat/completions, when the router serves the request. It also reads it on /queue/chat/completions. Other routes, such as /v1/responses, /v1/messages, /v1/embeddings and /cursor/chat/completions, ignore the header, and so does a proxy started with litellm --model and no config.

curl

curl -X POST 'http://localhost:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H 'x-litellm-priority: 0' \
-d '{
    "model": "gpt-6-luna",
    "messages": [
        {
        "role": "user",
        "content": "what is the meaning of the universe?"
        }],
    "priority": 5
}'

OpenAI SDK

import openai

client = openai.OpenAI(
    api_key="anything",
    base_url="http://localhost:4000/v1"
)

response = client.chat.completions.create(
    model="gpt-6-luna",
    messages=[{"role": "user", "content": "Write a short poem"}],
    extra_headers={"x-litellm-priority": "0"},
)

print(response)

Responses can also carry a header named x-litellm-priority. On /queue/chat/completions it holds the priority the request was queued at. With the dynamic rate limiter on, every non-streaming response, 429s included, holds the key's priority tier, such as premium or default. That tier comes from the key's metadata priority and is separate from this scheduler queue priority, so don't send it back as the request header.

Advanced - Redis Caching

Use redis caching to do request prioritization across multiple instances of LiteLLM.

SDK

from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "gpt-6-luna",
            "litellm_params": {
                "model": "gpt-6-luna",
                "mock_response": "Hello world this is Macintosh!", # fakes the LLM API call
                "rpm": 1,
            },
        },
    ],
    ### REDIS PARAMS ###
    redis_host=os.environ["REDIS_HOST"], 
    redis_password=os.environ["REDIS_PASSWORD"], 
    redis_port=os.environ["REDIS_PORT"], 
)

try:
    _response = await router.acompletion( # ADDS TO QUEUE + POLLS + MAKES CALL
        model="gpt-6-luna",
        messages=[{"role": "user", "content": "Hey!"}],
        priority=0, # LOWER IS BETTER
    )
except Exception as e:
    print("didn't make request")

PROXY

model_list:
    - model_name: gpt-6-luna-fake-model
      litellm_params:
        model: gpt-6-luna
        mock_response: "hello world!" 
        api_key: my-good-key

litellm_settings:
    request_timeout: 600 # Will keep retrying until timeout occurs

router_settings:
    redis_host: os.environ/REDIS_HOST
    redis_password: os.environ/REDIS_PASSWORD
    redis_port: os.environ/REDIS_PORT

$ litellm --config /path/to/config.yaml 

# RUNNING on http://0.0.0.0:4000s

curl -X POST 'http://localhost:4000/queue/chat/completions' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-D '{
    "model": "gpt-6-luna-fake-model",
    "messages": [
        {
        "role": "user",
        "content": "what is the meaning of the universe? 1234"
        }],
    "priority": 0 SET VALUE HERE
}'