Skip to main content

Server Tuning

Reach for this page only when the defaults are not enough. Most deployments should run one Uvicorn worker per pod and scale horizontally, as described in the production checklist. The options below matter when you pack multiple workers into one container, terminate TLS at the proxy, serve HTTP/2, or cannot mount a config file on your host. See the CLI reference for every flag.

Uvicorn vs. Gunicorn​

LiteLLM Proxy runs on Uvicorn by default. Passing --run_gunicorn instead starts Gunicorn as a process manager that supervises Uvicorn worker processes (uvicorn.workers.UvicornWorker). In both cases your application code still runs on Uvicorn; the difference is which process manages and recycles the workers.

Uvicorn (default)Gunicorn (--run_gunicorn)
When to useRecommended for almost all deployments, especially Kubernetes with one worker per pod.Choose when you run multiple workers in a single container and want a mature process manager to supervise and recycle them.
Worker recyclingUvicorn's limit_max_requests.Gunicorn's max_requests, the battle-tested mechanism Gunicorn has shipped for years.
Process supervisionUvicorn's built-in multiprocess manager.Gunicorn's arbiter, which restarts workers one at a time as they exit.
Recommendation

On Kubernetes, run one Uvicorn worker per pod and scale horizontally (more pods) rather than vertically (more workers per pod). One process per pod keeps latency predictable under load, lets the Horizontal Pod Autoscaler use the thresholds in the production checklist accurately, and makes rolling restarts hitless because Kubernetes drains one pod at a time. Reach for Gunicorn only when you must pack multiple workers into one container.

Recycle workers​

If you observe gradual memory growth under sustained load, recycle each worker after a fixed number of requests to bound memory usage. --max_requests_before_restart maps to Uvicorn's limit_max_requests (default server) and to Gunicorn's max_requests under --run_gunicorn. Configure it via CLI flag or environment variable:

# CLI
CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--num_workers", "1", "--max_requests_before_restart", "10000"]

# or ENV (for deployment manifests / containers)
export MAX_REQUESTS_BEFORE_RESTART=10000
tip

When you run multiple workers in one container and rely on --max_requests_before_restart, prefer --run_gunicorn. Gunicorn's max_requests recycling is more mature than Uvicorn's, and its arbiter restarts workers one at a time so the pod keeps serving traffic while a worker is replaced.

# Multiple workers in one container, with Gunicorn-managed recycling
CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--num_workers", "4", "--run_gunicorn", "--max_requests_before_restart", "10000"]

When several workers boot together and serve a similar amount of traffic, they reach the request threshold at almost the same time and recycle in lockstep, dropping a chunk of capacity at once. Add --max_requests_before_restart_jitter to offset each worker's threshold by a random amount in [0, jitter] so restarts stagger instead of synchronizing. It maps to Uvicorn's limit_max_requests_jitter (requires uvicorn>=0.41.0) and Gunicorn's max_requests_jitter, and has no effect without --max_requests_before_restart.

# Stagger recycling so workers don't all restart at once
CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--num_workers", "4", "--run_gunicorn", "--max_requests_before_restart", "10000", "--max_requests_before_restart_jitter", "1000"]

Keep restarts hitless​

A restart is "hitless" when in-flight requests finish before the process exits, so no client sees a dropped connection. Two cases matter in production:

Worker recycling (from --max_requests_before_restart). Both servers stop accepting new connections on the recycled worker and let outstanding requests drain before it exits, then a replacement worker starts. Gunicorn additionally guarantees in-flight requests up to its graceful_timeout (30s by default) on SIGTERM. With one worker per pod, recycling briefly reduces that pod's capacity, which is why we recommend scaling horizontally so the load balancer can route around it.

Rolling deploys and pod restarts (Kubernetes). Make restarts hitless at the orchestration layer rather than relying on the server alone:

  • Use a RollingUpdate strategy (the Deployment default) so new pods become Ready before old pods are terminated.
  • Keep a readiness probe on /health/readiness so Kubernetes only sends traffic to pods that can serve it, and stops routing to a pod as soon as termination begins.
  • Set terminationGracePeriodSeconds to comfortably exceed your longest expected request (LiteLLM's request timeout defaults to 600s; see the recommended config). On termination Kubernetes sends SIGTERM, and both Uvicorn and Gunicorn shut down gracefully by draining in-flight requests before exiting.
  • Optionally add a small preStop hook (for example sleep 5) to give the load balancer time to deregister the pod before the server begins shutting down, eliminating the brief window where traffic can still arrive at a terminating pod.
Kubernetes Deployment snippet for hitless rolling restarts
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0   # never drop below desired replica count
      maxSurge: 1         # add one new pod at a time
  template:
    spec:
      terminationGracePeriodSeconds: 620   # > your longest request (request_timeout: 600)
      containers:
        - name: litellm
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: 4000
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 5"]

TLS at the proxy​

For TLS terminated by the proxy itself (rather than your load balancer), pass the key and cert paths:

docker run docker.litellm.ai/berriai/litellm:latest \
    --ssl_keyfile_path ssl_test/keyfile.key \
    --ssl_certfile_path ssl_test/certfile.crt

HTTP/2 with Hypercorn​

To serve HTTP/2, build an image with hypercorn installed and pass --run_hypercorn:

FROM docker.litellm.ai/berriai/litellm:latest
WORKDIR /app
COPY config.yaml .
RUN chmod +x ./docker/entrypoint.sh
EXPOSE 4000/tcp
RUN uv add hypercorn
CMD ["--port", "4000", "--config", "config.yaml"]
docker run \
    -v $(pwd)/proxy_config.yaml:/app/config.yaml \
    -p 4000:4000 \
    -e DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname> \
    -e LITELLM_MASTER_KEY="sk-1234" \
    your_custom_docker_image \
    --config /app/config.yaml \
    --run_hypercorn

Granian ASGI server [Beta]​

Beta feature

--run_granian is in beta. Uvicorn is still the default server. Try Granian when you need more gateway throughput or see instability under load with uvicorn; report issues on GitHub.

Granian is a Rust-backed ASGI server. In LiteLLM benchmarks it showed a 10 to 20 RPS improvement over uvicorn with the same worker count, steadier latency under sustained load, and lower error rates (see PR #26027). Scale throughput with --num_workers.

docker run docker.litellm.ai/berriai/litellm:latest \
    --config /app/config.yaml \
    --port 4000 \
    --run_granian \
    --num_workers 4

Both --ssl_certfile_path and --ssl_keyfile_path are required when enabling TLS with Granian. Not supported with Granian: --max_requests_before_restart (use Gunicorn for per-request worker recycling) and --ciphers (Hypercorn only). See CLI server backend options.

Per-worker admission control​

A worker whose event loop is saturated keeps accepting connections, so during a load spike callers wait for seconds with no overload signal, and the liveness probe (which runs on the same loop) slows down enough that Kubernetes restarts the pod and pushes the load onto the remaining replicas. Admission control puts a hard cap on how much work each worker process takes on and turns the excess into an explicit, fast 503 that clients can retry against.

general_settings:
  max_in_flight_requests_per_worker: 64   # requests being processed at once, per worker process
  max_queued_requests_per_worker: 64      # requests waiting for a slot; defaults to the in-flight cap
  admission_queue_timeout_seconds: 1.0    # a queued request is rejected after waiting this long

The feature is off until max_in_flight_requests_per_worker is set. When a request arrives and the worker has a free slot it runs immediately. Otherwise it waits in the queue until a slot frees or the timeout elapses. If the queue is already full, or the wait times out, the client gets:

HTTP/1.1 503 Service Unavailable
retry-after: 1

{"error":{"message":"Worker at capacity: 64 in-flight, 64 queued requests. Retry later.","type":"overloaded_error","code":"503"}}

A slot is held for the whole response, so a streaming completion counts as one in-flight request until its last chunk is sent, and a client disconnect (queued or in flight) releases the slot immediately. The probe and metrics paths (/health/liveliness, /health/liveness, /health/readiness, /health/readiness/details, /health/backlog, /health/drain, /metrics) bypass the gate, so an overloaded worker still answers its liveness probe quickly while a wedged process does not. Rejections happen before authentication, so they are not attributed to a key in spend logs. The limits are read on the first request and need a restart to change.

The cap is per worker process and works the same on uvicorn and Granian. A pod started with --num_workers 4 and max_in_flight_requests_per_worker: 64 admits up to 256 concurrent requests, and a deployment of N replicas admits N times that, so size it from the per-worker throughput you measured rather than the deployment total. It is a good fit with an HPA: replicas that are already saturated shed load with 503s instead of accumulating latency while new replicas come up. It complements global_max_parallel_requests, which is a deployment-wide limit coordinated through Redis: use the global limit to bound total load on your providers and the per-worker limit to keep any single event loop from drowning without depending on Redis.

Monitor it with /health/backlog (fields in_flight_requests, admitted_requests, queued_requests, rejected_requests) or the Prometheus metrics litellm_admission_admitted_requests, litellm_admission_queued_requests, and litellm_admission_rejected_requests_total{reason="queue_full"|"queue_timeout"}; see Pod health metrics. A steady stream of queue_timeout rejections means the worker is at capacity and needs more replicas; queue_full rejections mean spikes are arriving faster than the queue can absorb, so raise the queue size or add capacity.

Keepalive timeout​

Defaults to 5 seconds; between requests, connections must receive new data within this period or be disconnected.

docker run docker.litellm.ai/berriai/litellm:latest \
    --keepalive_timeout 75

Or set KEEPALIVE_TIMEOUT=75 as an env var.

Load config.yaml from S3 or GCS​

Use this if you cannot mount a config file on your deployment service (AWS Fargate, Railway, etc.). LiteLLM reads config.yaml from the bucket at startup.

docker run --name litellm-proxy \
   -e DATABASE_URL=<database_url> \
   -e LITELLM_CONFIG_BUCKET_TYPE="gcs" \
   -e LITELLM_CONFIG_BUCKET_NAME="litellm-proxy" \
   -e LITELLM_CONFIG_BUCKET_OBJECT_KEY="proxy_config.yaml" \
   -p 4000:4000 \
   docker.litellm.ai/berriai/litellm:latest

Disable pulling live model prices​

Set LITELLM_LOCAL_MODEL_COST_MAP="True" to use the bundled model prices file instead of fetching it at startup, if you see long cold starts or have network egress restrictions.

🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →