Skip to main content

UI - Router Settings for Keys and Teams

Configure router settings at the key and team level to achieve granular control over routing behavior, fallbacks, retries, and other router configurations. This enables you to customize routing behavior for specific keys or teams without affecting global settings.

Overview​

Router Settings for Keys and Teams allows you to configure router behavior at different levels of granularity. Previously, router settings could only be configured globally, applying the same routing strategy, fallbacks, timeouts, and retry policies to all requests across your entire proxy instance.

With key-level and team-level router settings, you can now:

  • Customize routing strategies per key or team (e.g., use least-busy for high-priority keys, latency-based-routing for others)
  • Configure different fallback chains for different keys or teams
  • Split traffic across deployments with deployment weights per key or team
  • Set key-specific or team-specific timeouts and retry policies
  • Apply different reliability settings (cooldowns, allowed failures) per key or team
  • Override global settings when needed for specific use cases

Summary​

Router settings follow a hierarchical resolution order: Keys > Teams > Global. When a request is made:

  1. Key-level settings are checked first. If router settings are configured for the API key being used, those settings are applied.
  2. Team-level settings are checked next. If the key belongs to a team and that team has router settings configured, those settings are used (unless key-level settings exist).
  3. Global settings are used as the final fallback. If neither key nor team settings are found, the global router settings from your proxy configuration are applied.

This hierarchical approach ensures that the most specific settings take precedence, allowing you to fine-tune routing behavior for individual keys or teams while maintaining sensible defaults at the global level.

How Router Settings Resolution Works​

Router settings are resolved in the following priority order:

Resolution Order: Key > Team > Global​

  1. Key-level router settings (highest priority)

    • Applied when router settings are configured directly on an API key
    • Takes precedence over all other settings
    • Useful for individual key customization
  2. Team-level router settings (medium priority)

    • Applied when the API key belongs to a team with router settings configured
    • Only used if no key-level settings exist
    • Useful for applying consistent settings across multiple keys in a team
  3. Global router settings (lowest priority)

    • Applied from your proxy configuration file or database
    • Used as the default when no key or team settings are found
    • Previously, this was the only option available

How to Configure Router Settings​

Configuring Router Settings for Keys​

Follow these steps to configure router settings for an API key:

  1. Navigate to http://localhost:4000/ui/?login=success

  1. Click "+ Create New Key" (or edit an existing key)

  1. Click "Optional Settings"

  1. Click "Router Settings"

  1. Configure your desired router settings. For example, click "Fallbacks" to configure fallback models:

  1. Click "Select a model to begin configuring fallbacks" and configure your fallback chain:

Configuring Router Settings for Teams​

Follow these steps to configure router settings for a team:

  1. Navigate to http://localhost:4000/ui/?login=success

  1. Click "Teams"

  1. Click "+ Create New Team" (or edit an existing team)

  1. Click "Router Settings"

  1. Configure your desired router settings. For example, click "Fallbacks" to configure fallback models:

  1. Click "Select a model to begin configuring fallbacks" and configure your fallback chain:

Deployment Weights​

Deployment weights split a model group's traffic across its deployments for one key or team. For example, a team can send 90% of its gpt-6.1-sol traffic to one deployment and 10% to another, while every other caller keeps the global split.

To set them in the UI, open the key or team Router Settings, select the simple-shuffle routing strategy on the Loadbalancing tab, and add a model group under Deployment Weights. Each deployment of that group is listed with a weight field and the share of traffic it receives. Weights are relative, so 9 and 1 give the same split as 90 and 10.

The same settings can be sent through the API on /team/new, /team/update, /key/generate or /key/update. The keys of weights are model group names, and the inner keys are deployment IDs (model_info.id):

{
"router_settings": {
"routing_strategy": "simple-shuffle",
"weights": {
"gpt-6.1-sol": {"deployment-id-a": 9, "deployment-id-b": 1}
}
}
}

Weights only apply with the simple-shuffle routing strategy, set either on the key or team or globally. The UI shows a warning when another strategy is selected. Once any deployment in a group has a weight, a deployment in that group without one receives no traffic from this key or team. A group with weights must have at least one weight above 0, and the proxy rejects deployment IDs that do not exist or that belong to a different team. Weights follow the same resolution order as the other router settings, so a key with its own router settings uses only those and the team's weights do not apply to it.

Use Cases​

Different Routing Strategies per Key​

Configure different routing strategies for different use cases:

  • High-priority production keys: Use latency-based-routing for optimal performance
  • Development keys: Use simple-shuffle for simplicity
  • Cost-sensitive keys: Use cost-based-routing to minimize expenses

Team-Level Consistency​

Apply consistent router settings across all keys in a team:

  • Set team-wide fallback chains for reliability, and with fallback_on_saturation: true walk them as soon as the requested model reaches the dynamic rate limiter's saturation_threshold (see Fall back before the preferred model saturates)
  • Configure team-specific timeout policies
  • Apply uniform retry policies across team members

Override Global Settings​

Override global settings for specific scenarios:

  • Production keys may need stricter timeout policies than development
  • Certain teams may require different fallback models
  • Individual keys may need custom retry policies for specific use cases

Gradual Rollout​

Test new router settings on specific keys or teams before applying globally:

  • Configure new routing strategies on a test key first
  • Validate fallback chains on a small team before global rollout
  • A/B test different timeout values across different keys