LiteLLM was counting every token in a long conversation to answer a yes-or-no routing question.
Removing that unnecessary work took median time to first byte from 553 ms to 35 ms in our local 440k-token benchmark: 94% lower.
This week, we cut the number of times a request to the LiteLLM proxy waits on Redis from 22 to 8.
A request from a key with a budget, in a team with a budget and TPM/RPM limits, against a model group with usage-based routing and a Redis response cache, made 22 Redis round trips: 12 before the model was called and 10 after. The same request, with the same checks and the same writes, now makes 5 before and 3 after.
The LiteLLM gateway course walks you through how the gateway, Router, and SDK work together. We built it for developers and platform teams deploying LiteLLM, and contributors who want to understand the code before opening a pull request.
Router plugins run on the proxy from v1.94.x. The design is still evolving; tell us how you'd use it and what you'd want next in the autorouter discussion on GitHub (#32168).
Router plugins are now available on LiteLLM. Each plugin receives the routing context, enriches it, and hands it to the next before the router makes the final decision.
The push came from the autorouter discussion (#32168): teams wanted to layer their own signals (language detection, domain classification, tenant policy, budget caps) onto routing without waiting for each one to land in core. This plugin extension lets teams make these changes while keeping LiteLLM's routing core stable.
LiteLLM now supoports videos character, edit and extension apis.
Connect to the Realtime API via WebRTC from browser/mobile clients. LiteLLM handles auth and key management.
Date: Feb 24, 2026
Duration: Ongoing (until fix deployed)
Severity: High (for users load balancing Responses API across different API keys)
Status: Resolved
When load balancing OpenAI's Responses API across deployments with different API keys (e.g., different Azure regions or OpenAI organizations), follow-up requests containing encrypted content items (like rs_... reasoning items) would fail with:
{
"error": {
"message": "The encrypted content for item rs_0d09d6e56879e76500699d6feee41c8197bd268aae76141f87 could not be verified. Reason: Encrypted content organization_id did not match the target organization.",
"type": "invalid_request_error",
"code": "invalid_encrypted_content"
}
}
Encrypted content items are cryptographically tied to the API key's organization that created them. When the router load balanced a follow-up request to a deployment with a different API key, decryption failed.
Date: Feb 23, 2026
Duration: ~3 hours
Severity: High (for users with provider wildcard access rules)
Status: Resolved
When a new Anthropic model (e.g. claude-sonnet-4-6) was added to the LiteLLM model cost map and a cost map reload was triggered, requests to the new model were rejected with:
key not allowed to access model. This key can only access models=['anthropic/*']. Tried to access claude-sonnet-4-6.
The reload updated litellm.model_cost correctly but never re-ran add_known_models(), so litellm.anthropic_models (the in-memory set used by the wildcard resolver) remained stale. The new model was invisible to the anthropic/* wildcard even though the cost map knew about it.
openai/*, gemini/*).