Vertex AI Live API WebSocket Passthrough
LiteLLM now supports WebSocket passthrough for the Vertex AI Live API, enabling real-time bidirectional communication with Gemini models.
Overview
The Vertex AI Live API WebSocket passthrough allows you to:
- Connect to Vertex AI Live API through LiteLLM proxy
- Use existing Vertex AI authentication methods
- Pass through all WebSocket messages bidirectionally
- Support text, audio, video, and multimodal interactions
- Track costs automatically for all usage types
Configuration
Environment Variables
Set the following environment variables for Vertex AI authentication:
# Required
DEFAULT_VERTEXAI_PROJECT=your-project-id
DEFAULT_VERTEXAI_LOCATION=us-central1
# Optional - use one of these for authentication
DEFAULT_GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
# OR run: gcloud auth application-default login
Configuration File
Alternatively, configure in your config.yaml:
default_vertex_config:
vertex_project: "your-project-id"
vertex_location: "us-central1"
vertex_credentials: "os.environ/GOOGLE_APPLICATION_CREDENTIALS"
default_vertex_config is a top-level key, not a child of litellm_settings.
Usage
WebSocket Endpoint
ws://your-proxy-host/vertex_ai/live
Query Parameters
vertex_project(optional): Google Cloud project ID (can be set in config)vertex_location(optional): Vertex AI location (can be set in config, default: us-central1)model(optional): model name, used to determine the Vertex region for global models and to pick ause_in_pass_throughdeployment
Pinned deployments
When model names a use_in_pass_through Vertex AI deployment and the proxy has no default_vertex_config or DEFAULT_VERTEXAI_PROJECT, the session uses that deployment's project and region. A vertex_project or vertex_location query param that differs from the deployment's value closes the connection with code 1008 and the reason Rejected Request: vertex_location is pinned by the deployment, see configurable_clientside_auth_params. A vertex_project and vertex_location pair that matches another use_in_pass_through deployment connects through that deployment instead.
To let callers pick the region or project, list the param in the deployment's configurable_clientside_auth_params, or set general_settings.allow_client_side_credentials: true proxy-wide.
When the proxy sets default_vertex_config, each field it sets pins the session the same way. A vertex_project or vertex_location query param that differs from it closes the connection with code 1008 and the reason Rejected Request: vertex_location is pinned by default_vertex_config, see allow_client_side_credentials. A region that default_vertex_config leaves unset is still the caller's choice, since it only picks which Google endpoint serves the session. A project that nothing pins is not, since it decides whose quota and billing the proxy's credentials are spent on. A vertex_project other than the proxy's own VERTEXAI_PROJECT or DEFAULT_VERTEXAI_PROJECT then closes the connection with code 1008 and the reason Rejected Request: vertex_project must come from the proxy's configuration, see allow_client_side_credentials. default_vertex_config has no per-field opt-in, so only general_settings.allow_client_side_credentials: true lifts its pin.
Google's SDKs send the setup message's model as a full projects/<project>/locations/<location>/publishers/google/models/<model> path. The proxy moves the project and region in that path onto the session's own before it forwards the message, so the session stays on the project and region the admin chose. A setup message sent as a binary frame gets the same rewrite, and one that is not UTF-8 text closes the connection with code 1008. With allow_client_side_credentials: true the path is forwarded as sent.
model_list:
- model_name: gemini-live
litellm_params:
model: vertex_ai/gemini-3.8-live
vertex_project: my-project
vertex_location: us-central1
vertex_credentials: os.environ/GOOGLE_APPLICATION_CREDENTIALS
use_in_pass_through: true
configurable_clientside_auth_params: ["vertex_location"] # optional, lets callers pick the region
const ws = new WebSocket('ws://localhost:4000/vertex_ai/live?model=gemini-live');
Example Connection
// If vertex_project and vertex_location are set in config, you can connect without query params
const ws = new WebSocket('ws://localhost:4000/vertex_ai/live');
// Or specify them explicitly
const ws = new WebSocket('ws://localhost:4000/vertex_ai/live?vertex_project=your-project-id&vertex_location=us-central1');
Cost Tracking
The WebSocket passthrough automatically tracks costs for all usage types based on the Vertex AI pricing:
Supported Cost Tracking
- Text: Character-based or token-based pricing depending on model
- Audio: Per-second pricing for audio input/output
- Video: Per-second pricing for video input
- Images: Per-image pricing for image input
Cost Calculation
Costs are calculated using the same methods as other Vertex AI models in LiteLLM:
- Uses
cost_per_characterfor Gemini models - Uses
cost_per_tokenfor partner models (Claude, Llama, etc.) - Includes audio, video, and image costs when applicable
Cost Logging
Costs are automatically logged to:
- LiteLLM proxy logs
- Database (if configured)
- Spend tracking system
- Admin dashboard
Example log output:
Vertex AI Live WebSocket session cost: $0.001234 (input: $0.000800, output: $0.000434) tokens: 150, characters: 1200, duration: 45.2s
API Reference
Setup Message
Send this message first to initialize the session:
{
"setup": {
"model": "projects/your-project-id/locations/us-central1/publishers/google/models/gemini-2.0-flash-live-preview-04-09",
"generation_config": {
"response_modalities": ["TEXT"]
}
}
}
Text Input
{
"client_content": {
"turns": [
{
"role": "user",
"parts": [{"text": "Hello! How are you?"}]
}
],
"turn_complete": true
}
}
Audio Input
{
"realtime_input": {
"media_chunks": [
{
"data": "base64-encoded-audio-data",
"mime_type": "audio/pcm"
}
]
}
}
Supported Features
Response Modalities
- TEXT: Text responses
- AUDIO: Audio responses with voice synthesis
Tools
- Function Calling: Define and use custom functions
- Code Execution: Execute Python code
- Google Search: Search the web
- Voice Activity Detection: Detect when user is speaking
Advanced Features
- Audio Transcription: Transcribe input and output audio
- Proactive Audio: Model responds only when relevant
- Affective Dialog: Understand emotional expressions
Examples
Python Client
import asyncio
import json
import websockets
async def chat_with_gemini():
uri = "ws://localhost:4000/vertex_ai/live?vertex_project=your-project-id"
headers = {"Authorization": "Bearer sk-<your-litellm-api-key>"}
async with websockets.connect(uri, additional_headers=headers) as websocket:
# Setup
setup = {
"setup": {
"model": "projects/your-project-id/locations/us-central1/publishers/google/models/gemini-2.0-flash-live-preview-04-09",
"generation_config": {"response_modalities": ["TEXT"]}
}
}
await websocket.send(json.dumps(setup))
# Wait for setup response
response = await websocket.recv()
print(f"Setup: {response}")
# Send message
message = {
"client_content": {
"turns": [{"role": "user", "parts": [{"text": "Hello!"}]}],
"turn_complete": True
}
}
await websocket.send(json.dumps(message))
# Receive response
async for response in websocket:
print(f"Response: {response}")
# Check if turn is complete
data = json.loads(response)
if data.get("serverContent", {}).get("turnComplete"):
break
asyncio.run(chat_with_gemini())
JavaScript Client
const ws = new WebSocket('ws://localhost:4000/vertex_ai/live?vertex_project=your-project-id');
ws.onopen = function() {
// Send setup
const setup = {
setup: {
model: "projects/your-project-id/locations/us-central1/publishers/google/models/gemini-2.0-flash-live-preview-04-09",
generation_config: { response_modalities: ["TEXT"] }
}
};
ws.send(JSON.stringify(setup));
};
ws.onmessage = function(event) {
const data = JSON.parse(event.data);
console.log('Received:', data);
// Check if setup is complete
if (data.setupComplete) {
// Send a message
const message = {
client_content: {
turns: [{ role: "user", parts: [{ text: "Hello!" }] }],
turn_complete: true
}
};
ws.send(JSON.stringify(message));
}
};
Error Handling
If the proxy cannot resolve Vertex AI credentials or a project ID (no use_in_pass_through Vertex model, no default_vertex_config, and no DEFAULT_VERTEXAI_* env vars), it accepts the connection and then closes it with code 1011 and the reason:
Vertex AI auth failed: set a use_in_pass_through vertex model, default_vertex_config, or DEFAULT_VERTEXAI_* env
A vertex_project or vertex_location query param that moves a pinned deployment or a field default_vertex_config sets, or a vertex_project the proxy has no configuration for, closes the connection with code 1008 before any call to Vertex AI. See Pinned deployments.
Authentication
The WebSocket passthrough uses the same authentication as other LiteLLM endpoints:
- API Key: Pass
Authorization: Bearer your-api-keyheader - Vertex AI Credentials: Set environment variables or config file
Limitations
- Requires valid Google Cloud project with Vertex AI API enabled
- WebSocket connections are not persistent across server restarts
- Rate limits apply based on your Google Cloud quotas
Troubleshooting
Common Issues
- Authentication Error: Ensure Vertex AI credentials are properly configured
- Project Not Found: Verify the project ID exists and has Vertex AI enabled
- Connection Refused: Check that the LiteLLM proxy server is running
Debug Mode
Enable debug logging to see detailed connection information:
export LITELLM_LOG=DEBUG