Skip to main content

Vertex AI - Anthropic, DeepSeek, Model Garden

Supported Partner Providersโ€‹

ProviderLiteLLM RouteVertex Documentation
Anthropic (Claude)vertex_ai/claude-*Vertex AI - Anthropic Models
DeepSeekvertex_ai/deepseek-ai/{MODEL}Vertex AI - DeepSeek Models
Meta/Llamavertex_ai/meta/{MODEL}Vertex AI - Meta Models
Mistralvertex_ai/mistral-*Vertex AI - Mistral Models
AI21 (Jamba)vertex_ai/jamba-*Vertex AI - AI21 Models
Model Gardenvertex_ai/openai/{MODEL_ID} or vertex_ai/{MODEL_ID}Vertex Model Garden

Vertex AI - Anthropic (Claude)โ€‹

Model NameFunction Call
claude-3-opus@20240229completion('vertex_ai/claude-3-opus@20240229', messages)
claude-3-5-sonnet@20240620completion('vertex_ai/claude-3-5-sonnet@20240620', messages)
claude-3-sonnet@20240229completion('vertex_ai/claude-3-sonnet@20240229', messages)
claude-3-haiku@20240307completion('vertex_ai/claude-3-haiku@20240307', messages)
claude-3-7-sonnet@20250219completion('vertex_ai/claude-3-7-sonnet@20250219', messages)

Usageโ€‹

from litellm import completion
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ""

model = "claude-3-sonnet@20240229"

vertex_ai_project = "your-vertex-project" # can also set this as os.environ["VERTEXAI_PROJECT"]
vertex_ai_location = "your-vertex-location" # can also set this as os.environ["VERTEXAI_LOCATION"]

response = completion(
model="vertex_ai/" + model,
messages=[{"role": "user", "content": "hi"}],
temperature=0.7,
vertex_ai_project=vertex_ai_project,
vertex_ai_location=vertex_ai_location,
)
print("\nModel Response", response)

Usage - thinking / reasoning_contentโ€‹

from litellm import completion

resp = completion(
model="vertex_ai/claude-3-7-sonnet-20250219",
messages=[{"role": "user", "content": "What is the capital of France?"}],
thinking={"type": "enabled", "budget_tokens": 1024},
)

Expected Response

ModelResponse(
id='chatcmpl-c542d76d-f675-4e87-8e5f-05855f5d0f5e',
created=1740470510,
model='claude-3-7-sonnet-20250219',
object='chat.completion',
system_fingerprint=None,
choices=[
Choices(
finish_reason='stop',
index=0,
message=Message(
content="The capital of France is Paris.",
role='assistant',
tool_calls=None,
function_call=None,
provider_specific_fields={
'citations': None,
'thinking_blocks': [
{
'type': 'thinking',
'thinking': 'The capital of France is Paris. This is a very straightforward factual question.',
'signature': 'EuYBCkQYAiJAy6...'
}
]
}
),
thinking_blocks=[
{
'type': 'thinking',
'thinking': 'The capital of France is Paris. This is a very straightforward factual question.',
'signature': 'EuYBCkQYAiJAy6AGB...'
}
],
reasoning_content='The capital of France is Paris. This is a very straightforward factual question.'
)
],
usage=Usage(
completion_tokens=68,
prompt_tokens=42,
total_tokens=110,
completion_tokens_details=None,
prompt_tokens_details=PromptTokensDetailsWrapper(
audio_tokens=None,
cached_tokens=0,
text_tokens=None,
image_tokens=None
),
cache_creation_input_tokens=0,
cache_read_input_tokens=0
)
)

VertexAI DeepSeekโ€‹

PropertyDetails
Provider Routevertex_ai/deepseek-ai/{MODEL}
Vertex DocumentationVertex AI - DeepSeek Models

Usageโ€‹

LiteLLM Supports all Vertex AI DeepSeek Models. Ensure you use the vertex_ai/deepseek-ai/ prefix for all Vertex AI DeepSeek models.

Model NameUsage
vertex_ai/deepseek-ai/deepseek-r1-0528-maascompletion('vertex_ai/deepseek-ai/deepseek-r1-0528-maas', messages)

VertexAI Meta/Llama APIโ€‹

Model NameFunction Call
meta/llama-3.2-90b-vision-instruct-maascompletion('vertex_ai/meta/llama-3.2-90b-vision-instruct-maas', messages)
meta/llama3-8b-instruct-maascompletion('vertex_ai/meta/llama3-8b-instruct-maas', messages)
meta/llama3-70b-instruct-maascompletion('vertex_ai/meta/llama3-70b-instruct-maas', messages)
meta/llama3-405b-instruct-maascompletion('vertex_ai/meta/llama3-405b-instruct-maas', messages)
meta/llama-4-scout-17b-16e-instruct-maascompletion('vertex_ai/meta/llama-4-scout-17b-16e-instruct-maas', messages)
meta/llama-4-scout-17-128e-instruct-maascompletion('vertex_ai/meta/llama-4-scout-128b-16e-instruct-maas', messages)
meta/llama-4-maverick-17b-128e-instruct-maascompletion('vertex_ai/meta/llama-4-maverick-17b-128e-instruct-maas',messages)
meta/llama-4-maverick-17b-16e-instruct-maascompletion('vertex_ai/meta/llama-4-maverick-17b-16e-instruct-maas',messages)

Usageโ€‹

from litellm import completion
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ""

model = "meta/llama3-405b-instruct-maas"

vertex_ai_project = "your-vertex-project" # can also set this as os.environ["VERTEXAI_PROJECT"]
vertex_ai_location = "your-vertex-location" # can also set this as os.environ["VERTEXAI_LOCATION"]

response = completion(
model="vertex_ai/" + model,
messages=[{"role": "user", "content": "hi"}],
vertex_ai_project=vertex_ai_project,
vertex_ai_location=vertex_ai_location,
)
print("\nModel Response", response)

VertexAI Mistral APIโ€‹

Supported OpenAI Params

Model NameFunction Call
mistral-large@latestcompletion('vertex_ai/mistral-large@latest', messages)
mistral-large@2407completion('vertex_ai/mistral-large@2407', messages)
mistral-nemo@latestcompletion('vertex_ai/mistral-nemo@latest', messages)
codestral@latestcompletion('vertex_ai/codestral@latest', messages)
codestral@@2405completion('vertex_ai/codestral@2405', messages)

Usageโ€‹

from litellm import completion
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ""

model = "mistral-large@2407"

vertex_ai_project = "your-vertex-project" # can also set this as os.environ["VERTEXAI_PROJECT"]
vertex_ai_location = "your-vertex-location" # can also set this as os.environ["VERTEXAI_LOCATION"]

response = completion(
model="vertex_ai/" + model,
messages=[{"role": "user", "content": "hi"}],
vertex_ai_project=vertex_ai_project,
vertex_ai_location=vertex_ai_location,
)
print("\nModel Response", response)

Usage - Codestral FIMโ€‹

Call Codestral on VertexAI via the OpenAI /v1/completion endpoint for FIM tasks.

Note: You can also call Codestral via /chat/completion.

from litellm import completion
import os

# os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ""
# OR run `!gcloud auth print-access-token` in your terminal

model = "codestral@2405"

vertex_ai_project = "your-vertex-project" # can also set this as os.environ["VERTEXAI_PROJECT"]
vertex_ai_location = "your-vertex-location" # can also set this as os.environ["VERTEXAI_LOCATION"]

response = text_completion(
model="vertex_ai/" + model,
vertex_ai_project=vertex_ai_project,
vertex_ai_location=vertex_ai_location,
prompt="def is_odd(n): \n return n % 2 == 1 \ndef test_is_odd():",
suffix="return True", # optional
temperature=0, # optional
top_p=1, # optional
max_tokens=10, # optional
min_tokens=10, # optional
seed=10, # optional
stop=["return"], # optional
)

print("\nModel Response", response)

VertexAI AI21 Modelsโ€‹

Model NameFunction Call
jamba-1.5-mini@001completion(model='vertex_ai/jamba-1.5-mini@001', messages)
jamba-1.5-large@001completion(model='vertex_ai/jamba-1.5-large@001', messages)

Usageโ€‹

from litellm import completion
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ""

model = "meta/jamba-1.5-mini@001"

vertex_ai_project = "your-vertex-project" # can also set this as os.environ["VERTEXAI_PROJECT"]
vertex_ai_location = "your-vertex-location" # can also set this as os.environ["VERTEXAI_LOCATION"]

response = completion(
model="vertex_ai/" + model,
messages=[{"role": "user", "content": "hi"}],
vertex_ai_project=vertex_ai_project,
vertex_ai_location=vertex_ai_location,
)
print("\nModel Response", response)

Model Gardenโ€‹

tip

All OpenAI compatible models from Vertex Model Garden are supported.

Using Model Gardenโ€‹

Almost all Vertex Model Garden models are OpenAI compatible.

PropertyDetails
Provider Routevertex_ai/openai/{MODEL_ID}
Vertex DocumentationSDK for Deploy & OpenAI Chat Completions, Vertex Model Garden
Supported Operations/chat/completions, /embeddings
from litellm import completion
import os

## set ENV variables
os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811"
os.environ["VERTEXAI_LOCATION"] = "us-central1"

response = completion(
model="vertex_ai/openai/<your-endpoint-id>",
messages=[{ "content": "Hello, how are you?","role": "user"}]
)