Auto Router [Add-on]

🚀 Help shape the Auto-Router

Get early access, work directly with the LiteLLM team, and influence the roadmap with your production traffic.

Apply to Become a Design Partner

Already testing it? Share your results in discussion #32168.

• One-click setup. Configure automatically checks the models your proxy already serves and fills all four tiers for you; no template to pick.
• One model name in your clients. The gateway classifies each request and picks the model.
• Any model, any provider, per tier. A single model, a random pool, or a Thompson-sampled pool.
• Classifier choices. Heuristics, an LLM, JEV through TypeSafe System One Choice, keyword rules, or a custom plugin
• Savings reported per request. Against an all-frontier baseline, in the logs and in Cost Optimization.
• Agent-safe. Prompt caching, context-window escalation, modality routing, mid-task stall escalation, and optional session pinning.

Results

Result	Measured on	Read more
JEV: 5.43x as fast as Haiku by median classification latency and 96.12% lower registry-priced classifier cost	80 authored synthetic cases, three paired repeats, 95.00% vs 73.75% match with authored expected tiers under one fixed rubric	JEV comparison
Claude Opus-5 solve rate at 27% lower cost	21-task subset of Terminal-Bench 2.0, 16/21 solved by both	Terminal-Bench
Heuristic v2: 27% more tasks solved at 45% lower cost per task than v1	Same 21-task subset, no LLM classifier call	Heuristic v2
74.5% cheaper at 87.3% of frontier quality	RouterArena, 8,399 graded queries	Cost and quality
51.1% saved, $12,249 over four months	272,876 production requests, 450+ users	Production case study
37% to 69% cheaper than caching alone	Five datasets including live gateway traffic	Prompt caching
Matched or beat the current model on 88.1% of responses	Shadow evaluation on live traffic, 143 judged turns	Shadow evaluations

Quick start

• Dashboard: Models + Endpoints, Add Model, Auto Router tab, enter a name, then click Configure automatically or pick a template. Review the tiers, Test Routing, and save.
• Agent: tell it run curl -fsSL https://docs.litellm.ai/skills/auto-router and follow the instructions.
• config.yaml: one router entry whose tiers name other models in the same file.

config.yaml

model_list:
  - model_name: claude-haiku-4-5
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-opus-5
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE:    claude-haiku-4-5
          MEDIUM:    claude-sonnet-5
          COMPLEX:   claude-opus-5
          REASONING: claude-opus-5
        classifier_type: heuristic
      complexity_router_default_model: claude-sonnet-5

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "What is 2+2?"}]}'

Explore

Use the OSS Classifier with hosted Jev, self-hosted Nimble and Laya, or the Databricks ai_decide AI Function. The OSS classifier guide covers setup, required gateway and dashboard builds, and migration from the existing Jev configuration. See the classifier reference for context, fallback and accounting.

Admin Setup

Create an Auto Router with dashboard presets, an agent skill, config.yaml, or the CLI.

User Setup

Connect Claude Code or Codex with lite configure and see live session savings in Claude Code.

Recommended Configurations

1M Context, Anthropic, OpenAI, Gemini, and Lite ladders as config.yaml, plus the benchmark and production configs.

Public Benchmarks

Terminal-Bench 2.0, Heuristic v2, RouterArena, classifier context, a production case study, and Fusion.

Prompt Caching

Switching models keeps the cache warm. Measured on five datasets.

Evaluate on Your Traffic

Shadow evaluations before you switch, savings accounting after.

OTEL Telemetry

Trace the selected configuration, routing reason, classifier failures, and retries in Lens or an OTLP backend.

Feature History

Which Auto Router features shipped in which release, with links to the stable GitHub releases.

Configuration Reference

Every complexity_router_config key, with defaults.

Release posts

• Harness-Aware Routing: Claude Code and Codex context handling, encrypted delegated tasks, and classifier logs
• Mid-Task Stall Escalation: bumps a request one tier when it's stuck in a retry loop
• Auto Router v2: one router for complexity, semantic, and adaptive routing
• 1-click presets and Test Routing
• Savings tab and per-request classifier cost
• Classifier context and usage benchmarks
• Shadow evaluations
• Context-size and modality routing