Auto Router: 5 Improvements for Cost, Speed, and Team Control

Predict cache costs, set speed by tier, let teams manage routers, and see the chosen model and savings in your CLI.

Help shape Auto Router

Test Auto Router on your production traffic with the LiteLLM team.

Apply to Become a Design Partner

1. Predict cache costs before switching models

Switching to a cheaper model can mean paying to rebuild the prompt cache. Estimate those costs from past requests before you switch.

• Compare input-token costs for your current and candidate models, including cache reads and writes.
• Check cache status and compare cold-cache and warm-cache costs.
• See the past requests behind each estimate. You get an unknown estimate if token counts or prices are missing.

Call POST /cost/predict-cache with both deployment IDs and a supported native Anthropic Messages request.

2. Set Fast Mode per tier

Turn on Fast beside reasoning effort in the Auto Router editor.

• Enable it for a model in one tier, such as Reasoning, while leaving the other tiers unchanged.
• Choose speed and reasoning effort as separate settings.
• Use Fast Mode with supported models on direct Anthropic.

You need Anthropic Fast Mode access and quota, and you pay a higher rate. Turning Fast off clears the tier setting; caller and provider defaults apply.

3. Let team members manage Auto Routers

Admins can give a team permission to create Auto Routers without granting proxy-admin access.

• Admins enable /auto_router/manage under Teams → Member Permissions.
• Members create a router under Models + Endpoints → Auto-Routers, choose a team, and add their models.
• Teammates can use the router. Its creator or an admin can edit it; existing model and dependency permissions apply.

4. See models and savings in your CLI

Track Auto Router usage without leaving your coding agent.

• See the model that handled your last turn.

• Compare session spend with an estimate for the most expensive model in your router's highest tier.

• Use your existing LiteLLM API key to read your own session stats.

• Set it up with lite configure. Use your proxy URL, set LITELLM_PROXY_API_KEY to your key, and replace production-auto-router with your router name:

  lite configure \
    --gateway-url http://localhost:4000 \
    claude \
    --api-key "$LITELLM_PROXY_API_KEY" \
    --model production-auto-router

In this example, the session cost $3.03 against a $14.00 estimated baseline, a 78% saving. Your results depend on your session. Stats can lag a completed turn while the proxy records it.

5. Get more accurate Anthropic savings

We fixed the baseline calculation to include Anthropic's Fast Mode and regional pricing.

• Compare costs that include the same Fast Mode and regional charges, such as inference_geo: us.
• Keep any custom prices you've set for a deployment.
• See zero savings when the routed request and baseline have identical usage, cache treatment, and prices.

See the fix in PR #41341.

Try it

Use Sonnet for Simple and Medium, Opus for Complex, and Fast Opus for Reasoning. Choose proxy and CLI builds that include these updates from main.

Set ANTHROPIC_API_KEY, DATABASE_URL, and LITELLM_MASTER_KEY before using this config. Use PostgreSQL for session stats and a master key starting with sk-.

Download config.yaml.

config.yaml

model_list:
  - model_name: sonnet-standard
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: opus
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: production-auto-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        classifier_type: heuristic
        session_affinity: true
        tiers:
          SIMPLE:
            - model_name: sonnet-standard
          MEDIUM:
            - model_name: sonnet-standard
          COMPLEX:
            - model_name: opus
          REASONING:
            - model_name: opus
              litellm_params:
                speed: fast

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Become an Auto Router design partner

Test Auto Router on your workload and help us decide what to build next.

Apply to Become a Design Partner