High Availability
Run replicas across availability zones to handle instance failures and upgrades. Add another region to meet recovery or latency requirements.
| Topic | Use it for |
|---|---|
| Blue/Green | Test a complete replacement deployment, then switch traffic. |
| Canary | Send a small share of requests to a new version before you promote it. |
| Multi-Region | Run regional gateways with a shared database and regional Redis. |
| Routing | Choose which region receives a request. |
| Failover | Decide when to remove a region and how to recover. |
Make one region resilient
Run at least two serving replicas on separate nodes and spread them across zones. Leave enough spare capacity for the remaining replicas to carry traffic after a failure. See Sizing & Scaling and the cloud-specific Reference Architectures.
On Kubernetes, use topology spread constraints to place replicas. A PodDisruptionBudget limits voluntary evictions, such as a node drain. It does not prevent node or zone failures. Set the Deployment rollout strategy to keep enough replicas available during upgrades.
Protect the data services
All replicas in a deployment use the same logical PostgreSQL database. Use database replication and automated failover across zones. Test connection recovery after a writer change and test backup restoration. Size the connection pool for the maximum replica count, including extra pods during a rollout. See Database Sizing.
Use shared Redis for replicas that need common rate-limit counters, router state, and caches. Configure Redis replication and failover. Test how your features behave when Redis is unavailable. See What Needs Redis and Redis Sizing.
Configure and test model fallbacks for provider outages.
Use the right health signal
Use /health/liveliness for process liveness and /health/readiness to decide whether an instance can receive traffic. Neither makes an LLM request. Add a separate, controlled synthetic request to check authentication, model access, and provider connectivity. See Health Checks.
By default, readiness returns 503 when the configured database is unreachable. If you enable allow_requests_on_db_unavailable, readiness can remain 200 during a database outage. This setting changes authentication and enforcement during the outage. Read graceful DB unavailability before enabling it. Use the multi-region health policy when several regions share the database.
Verify recovery
In a test environment, remove one replica, drain a node, trigger database failover, and interrupt Redis access. Confirm that the remaining capacity meets your latency and error targets. Check that streams finish during planned drains and spend queues recover. Record the time to detect a failure and restore service. If your recovery plan covers a whole-region outage, continue with Multi-Region.
