Skip to main content

Baseten

LiteLLM supports both Baseten Model APIs and dedicated deployments with automatic routing.

API Types​

Model API (Default)​

  • URL: https://inference.baseten.co/v1
  • Format: baseten/<model-name> (e.g., baseten/openai/gpt-oss-120b)
  • Best for: Quick access to popular models

Dedicated Deployments​

  • URL: https://model-{id}.api.baseten.co/environments/production/sync/v1
  • Format: baseten/{8-digit-alphanumeric-code} (e.g., baseten/abcd1234)
  • Best for: Custom models, latency SLAs
tip

Automatic Routing: LiteLLM detects the type based on model format:

  • 8-digit alphanumeric codes → Dedicated deployment
  • All other formats → Model API

Quick Start​

import os
from litellm import completion

os.environ['BASETEN_API_KEY'] = "your-api-key"

# Model API (default)
response = completion(
model="baseten/openai/gpt-oss-120b",
messages=[{"role": "user", "content": "Hello!"}]
)

# Dedicated deployment (8-digit ID)
response = completion(
model="baseten/abcd1234",
messages=[{"role": "user", "content": "Hello!"}]
)

Examples​

Basic Usage​

# Model API
response = completion(
model="baseten/openai/gpt-oss-120b",
messages=[{"role": "user", "content": "Explain quantum computing"}],
max_tokens=500,
temperature=0.7
)

# Dedicated deployment
response = completion(
model="baseten/abcd1234",
messages=[{"role": "user", "content": "Explain quantum computing"}],
max_tokens=500,
temperature=0.7
)

Streaming (Model API only)​

response = completion(
model="baseten/openai/gpt-oss-120b",
messages=[{"role": "user", "content": "Write a poem"}],
stream=True,
stream_options={"include_usage": True}
)

for chunk in response:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")

Usage with LiteLLM Proxy​

  1. Config:
model_list:
- model_name: baseten-model
litellm_params:
model: baseten/openai/gpt-oss-120b
api_key: your-baseten-api-key
  1. Request:
import openai
client = openai.OpenAI(
api_key="sk-<your-litellm-api-key>",
base_url="http://0.0.0.0:4000"
)

response = client.chat.completions.create(
model="baseten-model",
messages=[{"role": "user", "content": "Hello!"}]
)

Latest model aliases​

Since v1.109.0

baseten/<family>-latest names follow a model family as LiteLLM moves them to newer models, so your application keeps one model string. Use them like any other Baseten model name. LiteLLM sends Baseten the current target model and bills at that model's price. See Latest model aliases for how aliases work and the full list across providers.

AliasModel family
baseten/kimi-latestKimi
baseten/glm-latestGLM
baseten/glm-flash-latestGLM Flash
baseten/deepseek-pro-latestDeepSeek Pro
baseten/deepseek-flash-latestDeepSeek Flash

Responses API (/v1/responses)​

Baseten's Model API serves /v1/responses itself. To forward /v1/responses requests to it untranslated instead of bridging them through /v1/chat/completions, add supported_endpoints: ["/v1/responses"] under the deployment's model_info, see Native /v1/responses passthrough.