Skip to main content

Production Deployment

Production deployment guide for AWS, Google Cloud, Azure, or any Kubernetes cluster. For a first deployment on a single machine, start with the Quickstart; this page picks up where it ends.

There are two supported paths. If you run Kubernetes, deploy with Helm on EKS, GKE, or AKS; the install is the same on every cloud, only the data stores and ingress differ. If you do not run Kubernetes, AWS and GCP have official Terraform modules that stand up the entire stack; Azure has no Terraform module, so AKS with Helm is the supported path there.

Architecture​

Clients (OpenAI SDK, LangChain, curl)
Application Load BalancerHTTPS, path-based routing
ECS Fargategateway :4000, backend :4001, ui :3000
gatewaybackendui
Aurora PostgreSQLwriter + reader, IAM auth
ElastiCache Redismulti-AZ, TLS
Secrets Managermaster key, provider keys

LiteLLM provides two deployment modes:

  • Monolithic: one litellm image serves LLM traffic, management APIs, and the UI. This is what the litellm-helm chart runs, and the simplest to operate.
  • Microservices: a gateway (LLM traffic, port 4000), backend (management APIs and UI backend, port 4001), and ui (port 3000), each deployed and scaled independently. This is what the componentized litellm chart and both Terraform modules run; see the chart values for the full reference.

The supporting infrastructure is identical in either mode:

ComponentPurposeNotes
LiteLLM servicesOne proxy deployment (monolithic) or gateway + backend + ui (microservices)Stateless; run 2+ replicas behind a load balancer
PostgreSQLKeys, teams, users, spend logs, configRequired for the proxy's auth and tracking features
RedisRate limiting, router state, caching across instancesRequired once you run more than one instance
Migrations jobApplies schema migrations against PostgresRuns once per upgrade; proxy instances set DISABLE_SCHEMA_UPDATE=true

Core configuration​

DATABASE_URL="postgresql://user:password@host:5432/litellm"
LITELLM_MASTER_KEY="sk-..."   # admin key for the proxy
LITELLM_SALT_KEY="sk-..."     # encrypts provider credentials stored in the DB. Set once, never change it
DISABLE_SCHEMA_UPDATE="true"  # proxy instances never run migrations; the migrations job does
STORE_MODEL_IN_DB="True"      # manage models from the Admin UI instead of config files

LITELLM_SALT_KEY cannot be rotated after you add models: it encrypts the provider credentials stored in your database, and changing it makes them unreadable. Generate a strong random value and store both keys in your cloud's secret manager.

Official images are published to ghcr.io/berriai and mirrored at docker.litellm.ai/berriai. Use ghcr.io/berriai/litellm-database for monolithic deployments with Postgres (it bundles the Prisma toolchain), and pin a version tag rather than latest or a moving tag, so rollbacks are deterministic. All images are signed; see the Docker Image Security Guide for verification and the non-root variant.

Provision the data stores​

The Helm path needs a PostgreSQL database and a Redis reachable from your cluster. Use the managed services:

Provision RDS PostgreSQL and ElastiCache Redis in the same VPC as your EKS cluster, with security groups permitting the cluster's nodes on ports 5432 and 6379.

Deploy with Helm​

First create the secrets both charts consume:

kubectl create secret generic litellm-masterkey \
  --from-literal=masterkey="sk-$(openssl rand -hex 24)"

kubectl create secret generic litellm-db \
  --from-literal=username=litellm \
  --from-literal=password="<database-password>"

kubectl create secret generic litellm-env \
  --from-literal=LITELLM_SALT_KEY="sk-$(openssl rand -hex 24)" \
  --from-literal=REDIS_PASSWORD="<redis-password>" \
  --from-literal=OPENAI_API_KEY="<provider-key>"

Then pick a deployment mode:

values.yaml
replicaCount: 3

image:
  repository: ghcr.io/berriai/litellm-database
  tag: "v1.90.2"          # pin your version

masterkeySecretName: litellm-masterkey
masterkeySecretKey: masterkey

db:
  useExisting: true
  deployStandalone: false
  endpoint: "<postgres-endpoint>"
  database: litellm
  secret:
    name: litellm-db
    usernameKey: username
    passwordKey: password

environmentSecrets:
  - litellm-env

proxy_config:
  model_list:
    - model_name: gpt-4o
      litellm_params:
        model: openai/gpt-4o
        api_key: os.environ/OPENAI_API_KEY
  router_settings:
    redis_host: "<redis-endpoint>"
    redis_port: 6379
    redis_password: os.environ/REDIS_PASSWORD
helm install litellm oci://ghcr.io/berriai/litellm-helm -f values.yaml

The chart lives at deploy/charts/litellm-helm; the published chart versions carry LiteLLM release numbers (for example 1.90.2), and helm show values oci://ghcr.io/berriai/litellm-helm lists every knob. Beyond the values above it supports autoscaling (autoscaling.* or keda.*), PodDisruptionBudgets (pdb.*), a Prometheus ServiceMonitor (serviceMonitor.*), read replica routing (db.readReplicaUrl, see Database Read Replica), graceful drain on shutdown (lifecycle), and ArgoCD or Helm hooks for the migrations job (migrationJob.hooks.*, see Helm PreSync hooks).

Both charts run the migrations job automatically and keep DISABLE_SCHEMA_UPDATE=true on the proxy pods. Expose the service through your cloud's ingress: the AWS Load Balancer Controller on EKS, GKE Ingress on GKE, or Application Gateway Ingress (AGIC) on AKS, with health checks on /health/readiness, then point your DNS record at the resulting load balancer. For secrets, prefer your cloud's secret manager over plain Kubernetes secrets (Key Vault CSI driver on AKS, for example); the charts consume whatever secret you mount.

Kubernetes without Helm​

If you manage raw manifests, the equivalent deployment is a ConfigMap for config.yaml, a Secret for keys, a Deployment with health probes, and a Service.

Full manifest (ConfigMap, Secret, Deployment, Service)
apiVersion: v1
kind: ConfigMap
metadata:
  name: litellm-config-file
data:
  config.yaml: |
      model_list:
        - model_name: gpt-4o
          litellm_params:
            model: openai/gpt-4o
            api_key: os.environ/OPENAI_API_KEY
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: litellm-secrets
data:
  OPENAI_API_KEY: bWVvd19pbV9hX2NhdA== # your api key in base64
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-deployment
  labels:
    app: litellm
spec:
  replicas: 2
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
      - name: litellm
        image: docker.litellm.ai/berriai/litellm:main-v1.90.2 # pin a version, do not use :latest
        args:
          - "--config"
          - "/app/proxy_server_config.yaml"
        ports:
        - containerPort: 4000
        volumeMounts:
        - name: config-volume
          mountPath: /app/proxy_server_config.yaml
          subPath: config.yaml
        envFrom:
        - secretRef:
            name: litellm-secrets
        livenessProbe:
          httpGet:
            path: /health/liveliness
            port: 4000
          initialDelaySeconds: 120
          periodSeconds: 15
        readinessProbe:
          httpGet:
            path: /health/readiness
            port: 4000
          initialDelaySeconds: 120
          periodSeconds: 15
      volumes:
        - name: config-volume
          configMap:
            name: litellm-config-file
---
apiVersion: v1
kind: Service
metadata:
  name: litellm-service
spec:
  selector:
    app: litellm
  ports:
    - protocol: TCP
      port: 4000
      targetPort: 4000
  type: NodePort

To connect the database, switch the image to docker.litellm.ai/berriai/litellm-database and add DATABASE_URL and LITELLM_MASTER_KEY to the Secret; nothing else in the manifest changes.

Deploy with Terraform (AWS and GCP)​

The official modules deploy the full microservices stack (network, database, Redis, object storage, secrets, compute, load balancer, and a migrations job that runs before the services start) and are published to the Terraform Registry:

Provisions a VPC with public and private subnets, an Aurora PostgreSQL cluster (writer plus reader, IAM database auth), ElastiCache Redis (multi-AZ, encrypted), an S3 bucket, Secrets Manager entries, an Application Load Balancer, and ECS Fargate services.

main.tf
module "litellm" {
  source  = "BerriAI/litellm/aws"
  version = "~> 1.90"

  region = "us-east-1"
  azs    = ["us-east-1a", "us-east-1b"]
  tenant = "acme"
  env    = "prod"

  ui_password         = var.ui_password
  litellm_license     = var.litellm_license      # optional, omit for open source
  acm_certificate_arn = var.acm_certificate_arn  # TLS is required by default

  proxy_config = {
    model_list = [{
      model_name = "gpt-4o"
      litellm_params = {
        model   = "openai/gpt-4o"
        api_key = "os.environ/OPENAI_API_KEY"
      }
    }]
  }
  gateway_extra_secrets = {
    OPENAI_API_KEY = var.openai_key_secret_arn
  }
}

Before you apply: provision the TLS certificate in AWS Certificate Manager (the module refuses a plaintext ALB unless you explicitly set allow_plaintext_alb = true), and create any provider-key secrets in Secrets Manager first, since gateway_extra_secrets takes their ARNs. After apply, point your DNS record at the ALB hostname.

The module auto-generates the master key into Secrets Manager if you do not supply one. The application connects to Aurora with short-lived IAM tokens, so its DATABASE_URL carries no password (the database master password itself is generated into Secrets Manager and never touches the application). Every resource is named <tenant>-litellm-<env>, and the module declares no provider, so you can for_each it to run one stack per tenant.

To manage LiteLLM resources (keys, teams, models) as code once the stack is up, use terraform-provider-litellm.

Other platforms​

Deploy on Render:

Verify the deployment​

Confirm the proxy is up and can reach its database:

curl -s https://llm.example.com/health/readiness

Then open the Admin UI at https://llm.example.com/ui, log in with your master key, add a model, create a virtual key, and send a Playground message; a response proves the full path through the load balancer, proxy, database, and provider credentials. The Quickstart walks through each of those clicks with screenshots; the flow is identical on a production deployment.

Next steps​

Harden the deployment with the production checklist (worker counts, machine sizing, Redis settings, server tuning, graceful degradation). Verify image signatures with the Docker Image Security Guide. Add regions with Multi-Region Deployment. For very high throughput (1000+ RPS), enable the Redis transaction buffer.

🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →