Skip to main content

v1.94.0rc1 - Router Plugins, MCP Client-Held Credentials & Shared DataTable UI

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:1.94.0-rc.1

Key Highlights​

  • Breaking change: user budgets now apply on team keys by default - a user's personal max_budget is now enforced whenever they call through a team key, so it stacks with the team and team-member budgets instead of being skipped as before. If you relied on personal budgets being ignored on team keys, set general_settings.skip_user_budget_on_team_key: true to restore the previous behavior - PR #32005.
  • Router plugin pipeline and Auto-Router v2 - a new Router(plugins=[...]) extension point, resolvable from proxy YAML config, plus soft-floor adaptive mode, opt-in (now default) session affinity, multi-model tier random-pick, and user-triggered escalation keywords for the complexity router.
  • MCP client-held credentials mature - interactive SSO sign-in for dcr_bridge oauth_delegate DCR clients, client-held refresh envelopes, gateway-bound envelopes minted at the token endpoint, issuer-anchored OAuth discovery (RFC 8414 §3.3) to close the authorization-server mix-up, and ID-JAG support for MCP egress.
  • Shared DataTable dashboard migration - Virtual Keys, Teams, Guardrails, Tags, Vector Stores, Prompts, Skills, AI Hub, MCP Toolsets, and Policy Attachments all move onto the shared composable DataTable.
  • Python 3.14 support - the requires-python cap moves to <3.15, pyo3 rises to 0.29 so the native Rust bridge compiles, and redisvl / pypdf / openapi-core are unblocked on 3.14.
  • Per-model prompt cache minimums - prompt_cache_min_tokens is now recorded across the Anthropic and Bedrock Claude cost map entries, and the router resolves the real per-model minimum instead of a flat 1024.

New Models / Updated Models​

New Model Support (5 new pricing entries)​

ProviderModelContext WindowInput ($/1M tokens)Output ($/1M tokens)Features
Google AI Studiogemini/gemini-omni-flash-preview1.05M$1.50$9.00 (video output $17.50)Reasoning, vision, audio input, video input/output
Google Vertex AIgemini-omni-flash-preview1.05M$1.50$9.00 (video output $17.50)Reasoning, vision, audio input, video input/output
Amazon Bedrock (Mantle)bedrock_mantle/openai.gpt-5.6-sol272K$5.50$33.00Reasoning, vision, function calling, prompt caching, Responses API
Amazon Bedrock (Mantle)bedrock_mantle/openai.gpt-5.6-terra272K$2.75$16.50Reasoning, vision, function calling, prompt caching, Responses API
Amazon Bedrock (Mantle)bedrock_mantle/openai.gpt-5.6-luna272K$1.10$6.60Reasoning, vision, function calling, prompt caching, Responses API

Beyond the new entries, this release records prompt_cache_min_tokens on the Anthropic and Bedrock Claude families (512-4096 depending on the model), marks the gpt-realtime family as mode: realtime rather than chat, flags Gemini image-generation models as non-reasoning, and corrects the Fireworks glm-5p2 prompt-cache read price.

Features​

Bug Fixes​

  • Anthropic
    • Emit message_start exactly once in the Responses stream adapter - PR #32667
    • Translate raw adaptive thinking for pre-4.6 models on Chat Completions and Bedrock Converse - PR #32944
    • Drop the incompatible temperature param when downgrading adaptive thinking for pre-4.6 models on pass-through - PR #33244
    • Stop a 500 on a combined thinking-plus-signature streaming chunk - PR #33505
    • Self-heal on missing thinking-signature errors from Bedrock and Vertex - PR #33719
    • Drop empty content_block_delta events in the Anthropic adapter - PR #33315
    • Honor the messages request timeout - PR #33418
    • Unblock lite autoroute proxy deps, adaptive thinking, and thinking-plus-signature streaming in the CLI - PR #33507
  • Google Vertex AI
    • Surface Gemini grounding toolUsePromptTokenCount in Usage - PR #33533
    • Exclude Google Search grounding tokens from input token billing - PR #33742
    • Mark Gemini image generation models with supports_reasoning: false - PR #32836
  • Amazon Bedrock
    • Forward AWS credential kwargs into litellm_params so the responses bridge keeps WIF auth - PR #32956
  • Fireworks AI
    • Bill prompt-cache hits at the cache_read rate - PR #33714
    • Correct the glm-5p2 prompt-cache read price to $0.14/1M - PR #33796
  • OpenAI
    • Mark realtime-only gpt-realtime models as mode: realtime - PR #33728

LLM API Endpoints​

Features​

  • General
    • Add an x-litellm-model-name response header carrying the deployment model string - PR #33698

Bugs​

  • Responses API
    • Continue MCP gateway tool turns from the final response and surface failures - PR #33025
    • Clamp max_output_tokens below the API minimum - PR #33098
    • Intercept web search on the Responses API - PR #33129
  • Pass-through
    • Stop classifying plain predict / search paths as Vertex - PR #33658
    • Stop treating an upstream model body field as a LiteLLM model on auth-enforced pass-through routes - PR #33710
  • General
    • Surface upstream connection resets instead of an empty 200 stream - PR #33222
    • Source /v1/models token limits from the cost map instead of Router.get_model_group_info - PR #33721
    • Treat malformed configured token limits as absent on /v1/models - PR #33864

Management Endpoints / UI​

Features​

  • UI (shared DataTable migration)
    • Rebuild the Virtual Keys and Teams tables on the shared DataTable - PR #32991, PR #33128
    • Migrate the guardrails, tags, and policy attachments tables onto the shared DataTable - PR #33303, PR #33314, PR #33827
    • Migrate the vector stores, prompts, and skills tables, plus five more simple tables - PR #33343, PR #33548
    • Migrate the AI Hub, public hub, and MCP Toolsets tables - PR #33629
  • UI
    • Convert the endpoint usage charts to shadcn/recharts - PR #32723
    • Adopt openapi-react-query ($api) and convert useCustomers - PR #32949
    • Working Test Connection for the complexity auto router, and adaptive routing settings in Auto-Router v2 - PR #32950, PR #33146
    • Require an embedding model for the semantic auto router - PR #33313
    • Configure Anthropic automatic prompt caching from the Admin UI - PR #33581
    • Show the exact license expiration date in the usage cards - PR #33478
    • Move Caching out of Experimental into Developer Tools - PR #33432
    • Add a reusable BetaBadge and use it for the Projects sidebar item - PR #33449
    • Add a personal Logs view scoped to the current user in the chat UI - PR #33829
    • Left-anchor the Create Key and Create Team CTAs - PR #33248
    • Consolidate the Add/Edit credential modals into one CredentialModal - PR #32572
    • Colocate the mcp-servers view while keeping the shared mcp_tools surface - PR #32968
    • Standardize debounce waits behind a shared DEBOUNCE_WAIT_MS constant and migrate value and callback debounces to react-pacer - PR #33040, PR #33042, PR #33043, PR #33041
    • Remove the unmounted UsageIndicator and the Hide Usage Indicator flag - PR #33482
  • Auth & Management
    • Ingest and round-trip SCIM entitlements and roles user attributes - PR #33587
    • Add a disable_auto_add_proxy_admin_to_teams flag - PR #33563
    • Add lite up / lite down to ambiently route Claude Code through the proxy - PR #33231

Bugs​

  • UI
    • Address Virtual Keys redesign review nits - PR #33112
    • Drop w-full from page-content wrappers to remove 32px horizontal overflow - PR #33118
    • Render the sidebar scrollbar with the shadcn ScrollArea - PR #33124
    • Show and allow editing team model aliases after team creation - PR #33047
    • Respect litellm_key_header_name in BYOK credential save and workflow-run fetches - PR #33103
    • Stop credential edit from persisting the masked API key - PR #33797
    • Resolve chat routes at render time so navigation works under server_root_path - PR #33446
    • Navigate to /ui/login/ with a trailing slash via hard navigation - PR #33561
    • Show all teams in the policy attachment form for admins - PR #33628
    • Remove the Chat item from the dashboard left nav - PR #33647
    • Migrate tag deletion to the shared DeleteResourceModal - PR #33795
    • Stop sending the complexity-router pseudo-model to /health/test_connection - PR #33498
    • Filter embedding models out of the complexity tab dropdowns, require all tiers, and validate inline - PR #32978
  • Auth & Management
    • Route the master key to team-scoped models - PR #32926
    • Stop unrecognized model namespaces slipping through provider wildcard keys - PR #32979
    • Persist key_type so the UI shows the correct key scope instead of "All Proxy Models" - PR #33115
    • Enforce a minimum custom key length and mask short keys in key_name - PR #33462
    • Paginate through all pages when fetching SSO service principal group assignments - PR #33149
    • Scope the JWT enterprise gate to actual JWTs - PR #33296
    • Stop enforcing the UI session budget on CLI login tokens - PR #33312
    • Surface actionable CLI SSO errors when the CLI and proxy versions skew - PR #33309
    • Resolve team wildcard credentials for vector store files - PR #33649
    • Make CLI output ASCII-only so it does not crash legacy Windows consoles - PR #33465

AI Integrations​

Logging​

  • OpenTelemetry
    • Push-based OTLP billable-request metering for enterprise deployments - PR #31592
    • Restore proxy-level error.* attributes on v2 failure spans - PR #33664
  • Langfuse
    • Build a per-request OTLP exporter from key- and team-level dynamic Langfuse credentials - PR #32437
  • Prometheus
    • Expose video duration and image count consumption metrics - PR #33138
    • Read v3 rate limiter remaining values for the per-key model gauges - PR #33119
  • S3
    • Sanitize slashes in the response-id-derived object key file name - PR #33271
  • General
    • Add user- and team-level spend and budget to StandardLoggingPayload metadata - PR #33459
    • Add structured budget fields to budget rejection failure logs - PR #33460
    • Preserve callback order in get_combined_callback_list - PR #33005
    • Classify async anthropic_messages and generate_content as async - PR #33589
    • Redact the async complete streaming response for custom callbacks - PR #33106
    • Redact assistant tool call arguments in spend logs - PR #33111
    • Never log raw virtual keys in key insertion debug output - PR #33268

Guardrails​

  • Straiker
    • Add the Straiker guardrail integration - PR #33781
  • Compresr
    • Add the Compresr guardrail for query-aware context compression - PR #33295
  • Bedrock Guardrails
    • Add a resource-less InvokeGuardrailChecks detect-only mode - PR #33299
  • Model Armor
    • Restore reference attachments via skip_unscannable_attachments and remove the attachment count cap - PR #33554
  • Lasso
    • Send source.type for Used By attribution - PR #33090
  • LLM Guard
    • Apply the sanitized prompt returned by the moderation API to the request - PR #33331
  • General
    • Streaming text transformation in generic_guardrail_api - PR #33110
    • Forward optional metadata on POST /guardrails/apply_guardrail - PR #33067
    • Run apply_guardrail-style model-level pre_call guardrails at the deployment hook - PR #33136
    • Show YAML-defined guardrails in the Guardrail Monitor - PR #32853
    • Walk custom_tool_call_output items in the shared content helpers - PR #32969
    • Use StandardLoggingGuardrailInformation in the xecguard logging hook, and sanitize the scan result before recording it - PR #32911, PR #32935
    • Remove the stray docstring from the singulr module for consistency - PR #33800

Spend Tracking, Budgets and Rate Limiting​

  • Budgets
    • Enforce user budget on team keys at read time and on reservation, with a UI opt-out - PR #32005
    • Coerce default_internal_user_params.max_budget to a float on config load - PR #32434
  • Rate Limiting
    • Enforce max_parallel_requests as a per-slot concurrency gauge - PR #32441
  • Cost Tracking
    • Use the provider-reported usage cost for OpenRouter streams - PR #32255
    • Track cost for unmanaged Bedrock batches and generalize the flag - PR #32315
    • Track unauthenticated pass-through requests in spend logs - PR #32410
    • Track LLM completion usage and spend for /v1/rag/query - PR #32438
    • Bill partial streamed spend when the client disconnects mid-stream - PR #33736
    • Remove the dead user-cache lookup with a None key in the spend-update path - PR #33555

MCP Gateway​

  • Client-Forwarded Credentials (dcr_bridge / oauth_delegate)
    • Interactive SSO sign-in for dcr_bridge oauth_delegate DCR clients - PR #32946
    • Mint a gateway-bound envelope at the token endpoint, and add a client-held refresh envelope for the flow - PR #32828, PR #32980
    • Extract the dcr_bridge token flow into bridge_token_flow.py - PR #33141
    • Persist config.yaml DCR clients in a server-scoped store so refresh survives token expiry - PR #33768
    • Surface rejected delegate-auth upstream tokens as a connect-time 401, and relay upstream OAuth token and DCR rejections instead of a generic 500 - PR #32741, PR #33113
    • Make the preemptive-401 OAuth challenge decision mode-aware - PR #33586
  • OAuth & Identity
    • Issuer-anchored OAuth discovery (RFC 8414 §3.3) to close the authorization-server mix-up - PR #33450
    • Add ID-JAG (identity assertion authorization grant) support for MCP egress - PR #31516
    • Persist discovered OAuth endpoints and keep the last known good set on a failed re-discovery - PR #33286
    • Discover missing OAuth scopes and token_url when authorization_url is set manually - PR #33317
    • Cap the per-user OAuth token cache TTL at the token's own lifetime - PR #33346
  • Tools & Permissions
    • Per-server outcomes for aggregate tools/list and truthful single-server REST statuses - PR #33153
    • Expand toolset grants in the shared permission primitives so tools/call honors them - PR #33612
    • Index authed request-time tools missing from the semantic filter startup index - PR #33318
    • Keep the MCP reference intact when the semantic filter narrows tools - PR #33584

Performance / Loadbalancing / Reliability improvements​

  • Routing
    • Add a Router(plugins=[...]) routing-plugin pipeline, resolve auto-router routing plugins from proxy YAML config, and resolve router_settings.plugins dotted paths from installed packages - PR #32972, PR #33251, PR #33644
    • Soft-floor adaptive mode and random-pick multi-model tiers for the complexity router - PR #32947, PR #32967
    • Opt-in session affinity for the complexity router, then enabled by default, with the session id derived from Anthropic metadata.user_id - PR #33126, PR #33500, PR #33723
    • User-triggered escalation keywords, and per-tier semantic keyword prompts in the configure wizard - PR #33656, PR #33508
    • Honor a per-request routing_strategy from key/team router_settings, and apply team/key enable_tag_filtering to tag routing - PR #33429, PR #33436
    • Tag-aware pre-routing strategy selection for a shared model_name - PR #33691
    • Resolve the prompt cache minimum per model instead of a flat 1024 - PR #33637
    • Enforce context-window pre-call checks for Responses API input - PR #33706
    • Correct the Responses API tool_choice shape and propagate alias litellm_params in the auto router - PR #32974
    • Return an empty dict from _classifier_call_metadata when metadata is absent - PR #33452
    • Cast model_info cost values to float in _set_model_group_info - PR #33556
  • Reliability
    • Route Azure Anthropic /messages through Rust behind rust:true - PR #33616
    • Stop pinning large request payloads past request end - PR #33455
    • Stop a stale auth cache re-publish to Redis so key updates propagate across replicas - PR #33565
    • Honor the ssl value instead of key presence when building the async Redis connection pool - PR #32590
    • Reap orphaned Prisma query-engine processes when a worker dies - PR #33424
  • Deployment
    • Add per-component PodDisruptionBudget and topologySpreadConstraints to the componentized Helm chart - PR #33430
    • Bake the Prisma CLI and engines at a fixed path so fresh-DB migrations work for any uid offline - PR #33853
    • Restore the litellm-proxy-extras source dir in the runtime images - PR #33592

Documentation Updates​

  • Add a QA runbook section to the PR template - PR #32965
  • Add a router plugin reference catalog - PR #33746
  • Align the e2e skip-vs-fail docs with the hard-fail contract and scope the no-unit-tests rule - PR #33755
  • Add provider coding standards and provider abstraction standards for litellm-rust, and require the official Rust Style Guide in the agent rules - PR #33833, PR #33865, PR #33867

PR roll-up by ownership area​

PRs by ownership area (total: 236)

  • Other (CI / chore / tests / build / version bumps): 69
  • UI: 39
  • Performance / Routing / Reliability: 26
  • Models & Providers: 21
  • MCP: 17
  • Guardrails: 14
  • Logging: 13
  • Auth & Management: 13
  • Spend / Budgets / Rate Limits: 9
  • LLM API Endpoints: 9
  • Docs: 6

End-to-End Testing​

We are investing heavily in end-to-end testing to cut regressions and make LiteLLM more stable release over release. Every version is now exercised by a live suite that runs against a real deployed proxy and hits real provider endpoints, not mocks, so the behavior we validate is the behavior you get in production. Our goal is to reach 95% coverage this week and hold that bar going forward, so that fewer regressions ever reach a release.

This run executed 264 tests over roughly 58 minutes against a live gateway spanning Anthropic, Azure, Azure OpenAI, Amazon Bedrock (Converse and Invoke), Google Vertex AI, and OpenAI.

ResultCount
Passed263
Failed1

Coverage spans access control, batches, the Claude Code surface (streaming and non-streaming messaging, tool use, vision, thinking, prompt caching, structured outputs, PDF input, and web search), the Realtime API, embeddings, rerank, image generation, OCR, the Responses API and /v1/messages, logging into Datadog, OpenTelemetry, and Prometheus, key/team/user/organization management, the MCP gateway, budgets, rate limits, spend tracking, and routing. Against our internal coverage registry the suite currently exercises 43.0% of tracked cells (168/391), with quota management already at 95.0% and core LLM endpoints at 72.4%.

The one failure, test_sustains_throughput_slo_under_load, is a load-generation scenario that intentionally pushes the proxy past its configured throughput ceiling. It is unrelated to functional correctness and is expected to trip under that traffic.

The full run is attached here: v1.94.0rc1 e2e report.

New Contributors​

  • @Napuh made their first contribution in PR #32667

Full Changelog​

https://github.com/BerriAI/litellm/compare/v1.93.0-rc.1...v1.94.0-rc.1

🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →