Azure

Connect corporate users and applications through VPN, ExpressRoute, or an approved private access service. Use private DNS and a private Application Gateway frontend. Keep AKS and data connections private. NAT Gateway serves outbound provider calls.

Helm on AKS

Deploy separate gateway, backend, and UI services with Helm. Provide the VNet, private AKS cluster, Application Gateway, databases, private endpoints, and corporate network routes.

Azure AKS: separate services

RequestsPrivate trafficEgressOptionalDots join paths. Bridges cross them.

Corporate clients use private network access to reach a private Application Gateway frontend. AKS and data stores stay private. NAT Gateway handles outbound provider requests.

Architecture details and references

One Helm chart

The litellm chart uses one image for all components. This layout uses the default monolith.enabled=false.

Infrastructure you provide

This example uses a customer-managed AKS cluster and virtual network. Provision Application Gateway, subnet policies, NAT Gateway, databases, private endpoints, and DNS separately. The LiteLLM Helm chart deploys the application workloads.

Private Application Gateway

Use a private-only frontend in a dedicated subnet delegated to Microsoft.Network/applicationGateways. Use AGIC v1.7 or later and appgw.ingress.kubernetes.io/use-private-ip: true. Allow corporate traffic through VPN, ExpressRoute, or an approved private access service. Configure private DNS. AGIC routes requests to pod private IPs. Set backend routes and health probes for your service layout.

Public outbound traffic

For your own virtual network, use AKS outboundType userAssignedNATGateway and associate NAT Gateway with the AKS subnet. NAT handles outbound public API calls. Private endpoint traffic stays private and bypasses NAT. NAT translates addresses. It does not inspect traffic.

Private data connections

This diagram uses PostgreSQL Private Link. Configure endpoint DNS and disable public access. Creating a private endpoint does not disable public access. Use the private PostgreSQL and Redis connections with TLS.

Optional private model provider

The dashed Azure OpenAI path needs its own private endpoint and private DNS records linked to the workload VNet. Create this endpoint separately. Public providers continue to use the separate NAT egress path.

Design references: LiteLLM chart and layouts (pinned) · Private Application Gateway deployment · AGIC private frontend · Application Gateway subnet requirements · Application Gateway Ingress Controller · AKS NAT Gateway options · PostgreSQL Private Link · Redis Private Link · Azure AI private network access

Azure AKS: separate services

RequestsPrivate trafficEgressOptionalDots join paths. Bridges cross them.

Use Application Gateway v2 with a private-only frontend in its dedicated subnet. Delegate that subnet to Microsoft.Network/applicationGateways. Allow corporate client ranges to reach HTTPS on the private frontend. Configure private DNS for the gateway hostname. See private Application Gateway deployment.

Use AGIC v1.7 or later. Set appgw.ingress.kubernetes.io/use-private-ip: "true" on the LiteLLM Ingress. AGIC ignores this annotation when the gateway has no private IP. It sends requests directly to pod private IPs. Set routes and health probes for your service layout. See AGIC private IP configuration.

Associate NAT Gateway with the AKS subnet for outbound public API calls. Keep PostgreSQL, Redis, and optional Azure OpenAI private endpoints on their private paths. Disable public access to these data services and link their private DNS zones to the VNet.

For identity checks, see Microsoft Entra Application Proxy. For client certificates, see Client Authentication with mTLS. For replicas and rollouts, see High Availability.