Skip to main content

AI Gateway

Engineering

How we build the world's most widely used open-source AI Gateway. Routing, reliability, observability, and what we learn along the way.

We're hiring!

Routing to 100+ providers

OpenAIOpenAI|AnthropicAnthropic|Google GeminiGoogle Gemini|AWS BedrockAWS Bedrock|Azure OpenAIAzure OpenAI|Mistral AIMistral AI|Meta LlamaMeta Llama|GroqGroq|Hugging FaceHugging Face|PerplexityPerplexity|DeepSeekDeepSeek|CohereCohere|Together AITogether AI|Vertex AIVertex AI|OpenAIOpenAI|AnthropicAnthropic|Google GeminiGoogle Gemini|AWS BedrockAWS Bedrock|Azure OpenAIAzure OpenAI|Mistral AIMistral AI|Meta LlamaMeta Llama|GroqGroq|Hugging FaceHugging Face|PerplexityPerplexity|DeepSeekDeepSeek|CohereCohere|Together AITogether AI|Vertex AIVertex AI|

TypeSafe Jev on LiteLLM

TypeSafe AI's Jev lands in LiteLLM v1.103.0-rc: call it through the proxy with logging and cost tracking.

Kerry Lu —

Subtask-Specific Routing: Same Quality, 46% Less Cost

Routing each agent turn by what the agent is doing, exploring, implementing, or verifying, instead of by how hard the prompt looks. On a SWE-bench Verified subset it matched fixed Claude Opus-5 quality for 46% less, with 86% of turns never touching a frontier model.

Moe Khalil —

AutoRouter Per-Hop Compression: Cut LLM Classifier Costs Another 32%

The complexity router's LLM classifier can now use different compression than the model call it routes to. The classifier only needs enough context to route correctly, not to generate an answer. In internal testing, compressing it aggressively cut classification costs a further 32% beyond shared compression, with no change in routing accuracy.

Moe Khalil —

Auto-Router: Escalate a Task That Gets Stuck

The Auto-Router now reads the assistant's own recent tool calls, notices when an agentic task is stuck in a retry loop, and escalates it one tier automatically, the same bump escalation_keywords already offers.

Moe Khalil —

Introducing AutoRouter Heuristic v2: 27% More Tasks Solved at 45% Lower Cost

Heuristic v2 is a new classifier for LiteLLM's AutoRouter, pretrained across multiple rounds of graded response data so it ships with zero cold start. On a 21-task Terminal-Bench 2.0 subset it solved 3 more tasks than Heuristic v1, at 45% lower cost per solved task and lower latency.

Tin Lo —

Introducing LiteLLM Fusion: 56% More Tasks Solved Than Fable 5

LiteLLM Auto Router Fusion ran three models on the same task and synthesized their work, solving 14 of 21 Terminal-Bench tasks against 9 for Claude Fable-5 alone. Total spend rose 36%, cost per solved task fell 12%, and turn latency went up 5x.

Tin Lo —

Auto-Router: Route on Context Size and Modality

The Auto-Router now supports more routing configurations: context-window escalation moves oversized prompts to the cheapest tier that fits them before dispatch, modality routing sends image requests to tiers that can see, classification can run on user turns only, and shadow evaluations can compare several router configs on a team's live traffic.

Tin Lo —

Shadow Evaluations: Test the Auto-Router on Your Own Production Traffic

Shadow evaluations duplicate a sampled slice of one key's live traffic through an auto-router and have an LLM judge blindly compare the answers. On our own traffic the router matched or beat the current model on 88.1% of judged responses, measured before a single user-facing response changed.

Tin Lo —

Auto Router: Opus level quality at up to 27% lower cost

On a 21 task subset of Terminal-Bench 2.0, an auto router matched Claude Opus-5 solve rate at 27% lower cost. Adding the last 3 user messages as classifier context raised relative quality 14% but cost 44% more; adding assistant replies made both worse.

Tin Lo —

AutoRouter: Easy Visibility to Your Savings

A new Auto-Router Usage tab in Cost Optimization, per-request classifier cost reporting, preset matching against your own deployments, and two routing fixes.

Tin Lo —

AutoRouter: 1 Click Deploy

Six changes to the Auto-Router: 1-click Anthropic and OpenAI presets, a one-line agent setup skill, Test Routing in the UI, a replaceable classifier prompt, customizable tiers, and configurable reminder markers.

Tin Lo —

Auto Router v1.97: usage benchmarks and better quality for lower cost

v1.97 adds cost and usage benchmarks for the auto router, gives the LLM classifier a window of prior turns, and turns session affinity off by default. Across 5,600 live classifier calls, prior turns raised tier agreement on referential follow-ups from 14% to 78% at under a tenth of a cent per request, with no measurable latency change.

Tin Lo —

Prompt Caching Works with Auto Router

The most common objection to auto-routing is that switching models throws away your prompt cache. We measured it across five datasets, including real gateway traffic with the provider's own cache accounting, and the answer is no.

Tin Lo —

Cut 75% Claude Code cost with near frontier model quality

Two independent evaluations of a four-tier Auto Router config against an all-frontier baseline: 8,619 graded prompts, 14,000 simulated real conversations, and what the cost and quality numbers actually depend on.

Tin Lo —