Skip to main content

25 posts tagged with "ai-gateway"

View All Tags

Evaluate agents on your own tasks

Ishaan Jaffer
CTO, LiteLLM
LiteLLMPYTHON SDK / AGENT EVALUATIONS

Compare agents on your tasks

Typed tasks, harness and model targets, and a report you can inspect.

evaluate.py
from pathlib import Path
from tempfile import TemporaryDirectory
import litellm
from litellm import EvalTask, EvalTarget, Harness, PredicateCheck, sandbox

harness = Harness.TOOL_LOOP
model = "litellm_proxy/model-a"
prompt = "Classify: I was charged twice. Reply only with billing, technical, or sales."

# Run the agent.
with TemporaryDirectory() as workspace:
result = litellm.agent(
harness, prompt, model=model, sandbox=sandbox.local(workspace),
)
print(result.text)

# Evaluate the same setup and a second model.
task = EvalTask(
id="billing-ticket",
prompt=prompt,
check=PredicateCheck(lambda result: result.text.strip().lower() == "billing"),
)
targets = [
EvalTarget(harness, model, label="model-a"),
EvalTarget(harness, "litellm_proxy/model-b", label="model-b"),
]

report = litellm.agent_eval(
tasks=[task], targets=targets, trials=3, max_concurrency=2,
sandbox=sandbox.local,
)
print(report.to_markdown())
Path("scorecard.json").write_text(report.to_json(), encoding="utf-8")
Terminalselected columns

$ python evaluate.py

Target    Passed   Rate     Errors
model-a   3/3      100.0%   0
model-b   0/3      0.0%     0

Selected run
Task     billing-ticket
Target   model-b
Trial    1
Result   fail
Detail   predicate failed
JSON exportscorecard.json

Recorded SDK example with mocked answers. No live model calls.

Run your agent, then evaluate the same setup and compare a second model.Set up the example

Run an agent on a support ticket, then evaluate the same setup across repeated attempts. With litellm.agent_eval(), you define typed tasks and success checks, compare harness and model targets, and read pass rate, cost and duration in one scorecard.

Preview

LiteLLM's main branch includes the evaluation implementation. Published PyPI releases do not include this API yet. The examples below show the upcoming SDK interface.

Connect to your AI Gateway​

Use a LiteLLM AI Gateway virtual key with access to two model groups. The examples call them model-a and model-b. Replace those names with your groups.

In Bash or Zsh, set the gateway root URL without /v1. Run read, then enter your virtual key. The terminal hides the input.

export LITELLM_PROXY_API_BASE=https://litellm.example.com
export LITELLM_PROXY_API_KEY
read -rs LITELLM_PROXY_API_KEY

Skip read if your secret manager has set the key. The first comparison uses Tool Loop, which needs no CLI runtime.

Choose an execution backend for each attempt. The script below passes sandbox.local, which runs with your user's permissions. Use local execution for trusted tasks. The sandbox guide covers container factories and host-side Python tools.

Run an agent, then evaluate the same task​

Start with a support ticket. Run Tool Loop once to inspect its answer, then define a success check and compare two model targets. Save this complete script as evaluate_agents.py:

evaluate_agents.py
from pathlib import Path
from tempfile import TemporaryDirectory

import litellm
from litellm import EvalTarget, EvalTask, Harness, PredicateCheck, sandbox


harness = Harness.TOOL_LOOP
model = "litellm_proxy/model-a"
prompt = (
"Classify this ticket as billing, technical, or sales. "
"Reply with only the category.\n"
"Ticket: I was charged twice for my subscription."
)

with TemporaryDirectory() as workspace:
result = litellm.agent(
harness,
prompt,
sandbox=sandbox.local(workspace),
model=model,
)
print(result.text)

task = EvalTask(
id="billing-ticket",
prompt=prompt,
check=PredicateCheck(
lambda result: result.text.strip().lower() == "billing"
),
)

targets = [
EvalTarget(harness, model, label="model-a"),
EvalTarget(harness, "litellm_proxy/model-b", label="model-b"),
]

report = litellm.agent_eval(
tasks=[task],
targets=targets,
sandbox=sandbox.local,
trials=3,
max_concurrency=2,
)

print(report.to_markdown())
Path("scorecard.json").write_text(report.to_json(), encoding="utf-8")

litellm.agent() runs the agent and returns a Result. TemporaryDirectory creates its local workspace. The task reuses the prompt, and the first evaluation target reuses the same harness and model. An EvalTask holds the prompt and the rule for success; an EvalTarget selects the harness and model.

agent_eval() runs each target again in a fresh workspace. It takes tasks and targets; you do not pass the earlier Result to it.

Run the script from the directory where you saved it:

python evaluate_agents.py

One task, two targets and three trials produce six evaluation runs. The initial agent call is separate from those six runs and their scorecard. Each target has three attempts. The script also writes scorecard.json for your own analysis. Add representative tickets and their expected outcomes before choosing a production target.

Read the failures before comparing cost​

A run passes when the agent stops with done and its success check passes. A failed success check or turn limit produces fail. Setup failures, runtime errors, timeouts and grading errors produce error.

Inspect unsuccessful runs after the evaluation call:

from litellm import EvalStatus

for run in report.runs:
if run.status is not EvalStatus.PASS:
print(run.task_id, run.target, run.trial, run.detail)

for summary in report.summaries:
print(summary.target, summary.pass_rate, summary.cost)
print("Complete metrics:", summary.metrics_complete)

Pass rate includes error rows in the denominator. An unavailable runtime or failed judge request therefore remains visible in the comparison.

Compare cost alongside pass rate, token usage, model calls and tool calls. Check metrics_complete before treating a total as complete. The report separates agent spend from judge spend. Mean duration includes workspace preparation, setup, execution, grading and cleanup; concurrent runs can overlap.

Compare harnesses with the same task​

To compare Claude Code and Codex in the evaluation, install their runtimes and replace targets in the script:

targets = [
EvalTarget(Harness.CLAUDE_CODE, "litellm_proxy/model-a", label="claude-code"),
EvalTarget(Harness.CODEX, "litellm_proxy/model-b", label="codex"),
]

The initial agent call still uses Tool Loop. The evaluation compares the two new targets. Use model groups supported by each harness. This changes both the harness and model; to measure one change, keep the other fixed where the harnesses support the same model.

The SDK runs each task against each target and repeats the pair trials times. Give each task a unique ID and each target a unique label.

Choose a check for the outcome you need​

Use the same task interface for answers, writing, research and file changes:

CheckUse it to verify
PredicateCheckA final answer, tool calls or another part of the typed Result
JudgeCheckAn answer against a rubric, using an explicit judge model
CommandCheckFiles or project behavior with a command whose zero exit code passes

For a writing task, replace task in the script and allow your key to call the judge model group:

from litellm import JudgeCheck

task = EvalTask(
id="support-reply",
prompt="Reply to a customer who was charged twice. Do not promise a refund.",
check=JudgeCheck(
model="litellm_proxy/judge",
rubric=(
"Acknowledge the duplicate charge, give a clear next step, "
"and do not guarantee a refund."
),
),
)

Choose a judge model that supports JSON output and budget for its calls. It receives the task prompt, answer and reported file changes with your rubric. A negative verdict produces fail; a failed request or invalid verdict produces error.

For tasks that need files, add workspace to the task. Each attempt gets a copy. The runner rejects symbolic links before creating the evaluation sandbox, including internal links, broken links and a linked workspace root. Local execution uses your user's permissions. Read the workspace and sandbox guide before running file tasks.

Key takeaways​

  • Define the task and its success check once, then choose your targets.
  • Use repeated trials and inspect the unsuccessful runs.
  • Compare pass rate with cost and time, checking for incomplete metrics.
  • Export the scorecard to JSON to keep a record of the comparison.

Frequently asked questions​

Do tasks need a repository?​

No. The support-routing and writing tasks above use prompts and checks. Add a workspace when your task needs starting files.

Do I need a database?​

The evaluation runner returns its scorecard in memory and can export it as JSON. It adds evaluation identifiers to model-request metadata so you can correlate calls in existing gateway logs. Persisted gateway spend logs use the gateway's existing database setup.

Can I run evaluations in an async application?​

Use await litellm.aagent(...) for the initial run and await litellm.aagent_eval(...) for the evaluation inside your async function or notebook. Keep the same tasks, targets and options.

Can I use this with LiteLLM OSS?​

The evaluation API is under development in the open-source SDK. The release version will appear in this guide when it ships. LiteLLM Enterprise provides gateway controls and support for production deployments.

Run your own comparison​

Pick one task you can grade, run it against two targets, and inspect the outcomes. Expand the task set as you find failures you want future changes to catch. The complete evaluation guide covers setup commands, sandboxes and execution limits.

Evaluation API reference, agent harness quickstart, and connect harnesses to your AI Gateway.

Monetize Your AI Agent with LiteLLM Customer Management

Misbah Syed
DevRel Engineer, LiteLLM

LiteLLM Customer Management

Monetize your AI agent

Keep every customer's model cost under what they pay you.

Agent requestsMeters cost · Enforces capsPrepaid creditsAcme$100 credit packCustomer pays$100Model cost$39cap $6061%marginRequests flowingPer seatGlobex12 seats × $100Customer pays$1,200Model cost$430cap $72064%marginRequests flowingUsage-basedInitechModel cost × 1.3Customer pays$390Model cost$300cap $50023%marginRequests flowingIllustrative customers and amounts · margin before other expenses

Illustrative agent traffic flows through LiteLLM. Three customers on prepaid credits, per-seat, and usage plans each show what they paid, their model cost, and their margin. The prepaid customer reaches a $60 cap and its requests pause.

If you sell an AI agent to other companies, each request from your customers costs you money at the model provider. To monetize the agent, each customer must pay you more than the model cost of that customer. This post gives five frequent ways to charge for an agent. For each way, it shows the LiteLLM Customer Management controls that keep model spend below what the customer pays.

Introducing LiteLLM Customer Management

Ishaan Jaffer
CTO, LiteLLM
Moe Khalil
AI Product Engineer, LiteLLM
Kerry Lu
Member of Technical Staff, LiteLLM
Tin Lo
AI Product Engineer, LiteLLM

LiteLLM AI Gateway

Customer Management

Budget and charge your agent's customers.

Agent trafficAI GatewayMeter cost · Apply budgets{ } Usage APICustomer ManagementCustomer ManagementUsage chargesBudgetCustomer organizationAcme$750.00$1,000Customer teamSupport$192.00$250Customer memberAlice$39.00$50Same request. Three levels of attribution.Illustrative customer traffic · $0.50 per highlighted request

Illustrative customer traffic flows through LiteLLM into Customer Management. Organization, team, and member budgets use three views of the same request cost.

Today we're launching LiteLLM Customer Management for AI-native companies using the gateway. If you're building an agent on LiteLLM and need to track spend and set budget limits for your customers by their organizations, teams, and individual members, this is for you.

Share your team's skills with LiteLLM

Tin Lo
AI Product Engineer, LiteLLM

Give your team one place to find and share useful agent skills. With LiteLLM Skills Gateway, teammates can reuse a skill's instructions and supporting files in their own AI agents.

To help you manage those skills, we've added:

  • ZIP uploads
  • Admin review
  • Version history
  • Download tracking

Available in the LiteLLM 1.110.x release. Upgrade to v1.110.0 or later to use these features.

Model Deprecation Management

Ishaan Jaffer
CTO, LiteLLM

MODEL LIFECYCLE MANAGEMENT / NEW IN THE LITELLM UI

ModelDeprecationManagement

The LiteLLM UI now has alerts
for model deprecations.

See how it works
Model deprecation management in LiteLLMProvider deprecation notice, an alert in the LiteLLM UI, migration, and resolution. Model providers: OpenAI, Anthropic, Google, Azure, and AWS.PROVIDER NOTICEA model is retiringUI ALERTFind affected keysMIGRATEMove use casesRESOLVEDConfirm zero trafficOpenAIAnthropicGoogleAzureAWSMODEL LIFECYCLEDeprecationalertReview affected keys

This blog post is about how you can do model lifecycle management with LiteLLM AI Gateway. We are giving you the following primitives to better manage model deprecations.

We heard from a lot of our production gateway users that model deprecation was a problem: for production use cases, it's hard to figure out how to prevent outages.

This helps prevent outages with:

  1. Alerting around model deprecations.
  2. Emails around model deprecations.
  3. Usage tracking for different models that are getting deprecated, so you can validate that traffic is going down once you migrate, and a recommended model to use.

Alerts around model deprecations​

Check the notification bell for model deprecations that need attention. Open Alerts under Observability, then select Model deprecations. You can see the provider, the model, its deprecation date, requests in the last 30 days, and when LiteLLM last saw traffic to it.

Open the alert to investigate the migration across teams. You can keep the alert open while you coordinate with the owners, then return to the same page to check progress.

LiteLLM Model deprecations inbox with provider logo, deadline, recent requests and recommended replacement

The Model deprecations inbox in the LiteLLM UI. Sample usage; the provider deadline and recommendation come from model metadata.

Emails around model deprecations​

From the alert, click Set up email reminders, then Open Email Settings. This opens Logging & Alerts → Email Alerts.

If your gateway already sends email, keep its delivery settings. Otherwise, enter the SMTP connection details from your email provider in Email Server Settings. Set Gateway URL to the public address owners use to reach your gateway and save. Enter TEST_EMAIL_ADDRESS, select Test Email Alerts, and confirm that the test email arrives.

In Email Notifications, turn on Model Deprecations and select Save Changes. This event is off by default.

LiteLLM schedules two reminders: 30 days before and 1 day before the provider's deprecation date, checked in UTC. It selects affected key owners and current team admins using requests from the last 30 days. Each recipient sees usage for their keys and teams they administer.

The email names the provider deprecating the model, shows the current and recommended models with provider logos, and gives the owner the migration steps. It includes the recommendation source, the provider notice, and a link to review usage in the gateway.

Model deprecation email naming Google as the provider, comparing the old and recommended models, and listing migration steps

The 30-day email template with sample usage. Owners can see what changes, when they need to act, and where to review the affected traffic.

Keep the gateway running on the reminder dates. The scheduler does not send a missed reminder later if the gateway is offline for the entire UTC date.

Track usage as you migrate​

Click Review impact to open the model. The Daily requests chart shows requests to that model over time, with a separate stack for each key. Compare the total bar height to check whether usage is falling, then inspect the colored stacks to find the keys that still send traffic.

Below the chart, inspect the affected keys, teams, owners, and tags used with that model. Filter by team when you need to coordinate one part of the migration. Use the key and owner list to follow up with the people responsible for each production use case.

Model deprecation detail with daily requests stacked by key falling to zero, and five use-case keys across two teams

Actual LiteLLM UI with a seeded migration example: five use cases across two teams. Daily requests decline as the example keys move off the model. The final complete days show zero requests; the current day is partial.

You do not need to wait for the 30-day total to reach zero. That total includes historical requests. Compare the recent daily bars instead, and check a period that covers your use case. A weekly batch job needs a longer verification window than a service that sends requests every minute.

In this example, customer support, research, document search, invoice extraction, and weekly reports each have a key. As an owner migrates a use case, its color drops out of the daily bars. The key stays in the 30-day history so you can verify what changed.

The alert shows the model to replace and the Recommended new model. LiteLLM reads the recommendation from its model cost map, or from the deployment's configured replacement. The UI identifies which source supplied it and links to the upstream provider's deprecation notice when one is recorded.

Use the provider notice to check the upstream deadline and migration guidance. The cost map records the recommendation; it does not establish that the new model meets your application's quality, latency, or cost requirements. Test the replacement with the workload behind each affected key before switching production traffic.

You can use Model routing to open the relevant model configuration. If your application chooses the model, update that application. Keep the alert open while owners make these changes.

Mark the alert resolved​

Return to Daily requests after the owners switch their applications or routing. Check that traffic to the old model reaches zero across the affected keys. Confirm the migration with owners of infrequent jobs before closing it.

Click Mark resolved when that work is complete. LiteLLM moves the alert to Resolved, so your open inbox shows the migrations that still need attention. New requests to the model reopen the alert.

LiteLLM model migration marked resolved after the daily requests chart reaches zero

The same example migration after marking it resolved. Closing the alert records the team's decision; it does not change model routing.

For help operating LiteLLM across production teams and use cases, talk to us. Read more about gateway alerting, email notifications, and model routing.

Use Hex to analyze LiteLLM costs, then set budgets and routing

Ishaan Jaffer
CTO, LiteLLM
← BlogCOST REPORTING / HEX TEMPLATE
LiteLLMHex

AI spend.Cost reporting, now in Hex.

Connect Hex to your LiteLLM Postgres data.
Set budgets and routing in LiteLLM.

View live Hex demo
TEAMS / MODELS / BUDGETSOPEN TEMPLATE · V2.0.0
THE HEX BOARDSYNTHETIC DEMO DATA

Explore the live Hex dashboard.

Open the working demo ↗ · Synthetic data · No sign-in

Hex board showing project spend, request volume, unit costs and daily trends with synthetic data

See a model change raise unit cost.

Zoomed Hex demo alternating between project model mix and the cost-change table, where document-review shows an 80 percent unit-cost increase
Model mix and seven-day cost changes in Hex. Synthetic demo data.

The public demo includes synthetic timing metrics. The daily team template reports recorded costs, requests and usage; daily rows do not include latency.

LiteLLM records AI costs in Postgres and maintains daily spend totals. Hex connects to that database and reads the totals. Use the dashboard to find a cost increase, then open LiteLLM to change routing or set a budget.

1. Connect Hex to your LiteLLM Postgres database​

In Hex, open Settings → Data sources → + Connection → Postgres. Add a read-only connection to the database your LiteLLM gateway uses.

READ-ONLY DATA FLOWEXISTING DAILY AGGREGATES
01 / YOUR POSTGRESDaily spendLiteLLM_DailyTeamSpend
date + team_id
Day and team
model + spend
Recorded costs
api_requests
Daily requests
prompt_tokens + completion_tokens
Token usage
02 / HEX SOURCE QUERYTeam namesLiteLLM_TeamTable

Join daily totals to team display names.
Keep ownership by stable team ID.

SQL included in the templateNo database objects created
03 / YOUR HEX APPCost reporting

Team costs. Model mix.
Daily trends. Cost changes.

Refresh with a scheduled run
LiteLLM writes the daily totals in Postgres. Hex reads reporting columns from these two existing tables. You change budgets and routing in LiteLLM; Hex does not write to the gateway.

The template reads two existing tables: LiteLLM_DailyTeamSpend for costs and usage, and LiteLLM_TeamTable for team display names. It groups the results by day, team and model. Give the reader access to those reporting fields. It does not need your key table, prompts or responses.

Connection details and SQL permissions.

2. Add the template and inspect your costs​

Open the live Hex demo first. You can filter it without signing in. Its records are synthetic.

To use your own data, download the Postgres template below. In Hex Home, click + → Upload project, then click Upload in the dialog and choose the .hex.yaml file. Select your Postgres connection on the Source SQL cell and Run all.

Hex dialog for uploading a YAML project

Filter by team. Compare recorded spend, request volume and cost per thousand requests, then open Models and cost changes. In the demo, the Legal team's document-review workload switches models and unit cost rises 80%.

3. Take an action in LiteLLM​

Use Hex to investigate the increase. Apply the change through your LiteLLM gateway.

Your decisionIn the LiteLLM UI
Try a cheaper modelOpen Teams → your team → Settings → Edit Settings → Model Aliases. Change the model behind an alias your application already calls, then save. Evaluate the candidate on your workload first.
Route by request complexityOpen Models + Endpoints → Auto Router → Add Auto Router. Choose the models for each tier, test routing and connections, then save. Configure your application or team alias to call that router.
Set a spending limitOpen Teams → your team → Settings → Edit Settings. Set Max Budget (USD) and Reset Budget, then save.

For example, select Legal in LiteLLM and set Max Budget (USD) to 1000 and Reset Budget to monthly (30 days). This limits new spend; it does not change recorded costs.

Set Legal’s budget in the LiteLLM team settings

Current LiteLLM UI with synthetic local gateway data.

After saving, check the team's budget status. Then refresh Hex after LiteLLM updates its daily totals to compare the next complete period. Set a Hex scheduled run if you want the report to refresh on a schedule.

Legal’s budget status after saving

Auto Router setup · Team budgets

Rust Migration: /v1/messages end-to-end in Rust by Dec 1

Nate Armstrong
Lead Rust Engineer, LiteLLM

The Rust migration is being slightly rerouted to get the largest performance improvements to the largest portion of requests as quickly as possible. We will be focusing on /v1/messages requests to providers which natively support Anthropic-style APIs, with the most common set of proxy features enabled. By December 1st, you can expect the proxy to handle these requests with common configurations with about 20x less latency.

Cache-Aware Routing for Self-Hosted Models

Yassin Kortam
Senior SWE @ LiteLLM

Claude Code and Codex requests pass through the LiteLLM gateway and return to the same vLLM pod for each session, allowing cached prompt prefixes to be reused.

LiteLLM now supports cache-aware routing for self-hosted inference on Kubernetes. Within a deployment, requests from the same session go to the same GPU pod. This lets vLLM reuse cached prompt prefixes across turns.

Agent requests often share system instructions, tool definitions, and conversation history. When requests reach different pods, each pod may process the same prefix again. Session affinity helps reuse that work.

Introducing Microsoft 365 Copilot in LiteLLM

Yassin Kortam
Senior SWE @ LiteLLM

LiteLLM × Microsoft 365 Copilot

Microsoft 365 Copilot is now a provider in LiteLLM. Use it from your applications to ask questions about your Microsoft 365 work data.

Users sign in with their Microsoft work account and ask questions like “Summarize my latest meeting.” Copilot answers using the data they already have permission to access.

How we cut LiteLLM's Redis round trips per request by 64%

Yassin Kortam
Senior SWE @ LiteLLM

LiteLLM's 22 Redis round trips merge into 8, a 64% reduction per request

This week, we cut the number of times a request to the LiteLLM proxy waits on Redis from 22 to 8.

A request from a key with a budget, in a team with a budget and TPM/RPM limits, against a model group with usage-based routing and a Redis response cache, made 22 Redis round trips: 12 before the model was called and 10 after. The same request, with the same checks and the same writes, now makes 5 before and 3 after.

Introducing LiteAdmin MCP

Tin Lo
AI Product Engineer, LiteLLM
Introducing LiteAdmin MCP: your AI toolkit for gateway management. LiteAdmin for Slack is built on LiteAdmin MCP.Introducing LiteAdmin MCP: your AI toolkit for gateway management. LiteAdmin for Slack is built on LiteAdmin MCP.

An engineer asks for an API key for a new project. You need to choose its models, set a budget, and assign it to a team. As usage grows, you need to check spending and adjust those limits.

LiteAdmin MCP lets your agent handle these tasks through your gateway's management API. Connect it to an MCP client or a custom agent. We built LiteAdmin, our Slack admin agent, on the same connector.

Claude Code server-side auto mode through LiteLLM

Mateo Wang
AI Engineer, LiteLLM

Last Updated: October 9, 2026

Anthropic is moving the Claude Code auto mode safety classifier from the client to the Claude API. From v2.1.278 onwards (released 19 September), sessions on Enterprise plans and Claude API accounts will have the safety classifier run on the server as part of the model request. There is no cost to Anthropic for the classifier to run on the server. The rollout began on 18 September and is gradual, starting with the Claude Code CLI and VS Code extension, followed by the desktop app and Claude Code in the browser over the following week. On 25 September auto mode will become the default permission mode in Claude Code. Currently the default is auto on Pro, Max and Team plans and manual on Enterprise plans and Claude API keys (the accounts that are typically behind a gateway) as per Anthropic's permission modes reference.

Server side auto mode is predicated on a contract between Claude Code and the API which some gateways (LiteLLM included) did not honour. The fix first appeared in the dev release cut on Tuesday 22nd September and is now in v1.104.0 stable as well as in the patch releases of the earlier lines. This post covers what Claude Code requires from an AI Gateway, what LiteLLM was doing wrong, what has changed, which release has it, and how you can verify your deployment is ready.

JEV Classifier: 5.43x as Fast as Haiku, 96% Lower Cost

Moe Khalil
AI Product Engineer, LiteLLM

Last Updated: September 18, 2026

An Auto Router pays for classification before the selected model can answer. In our benchmark, TypeSafe JEV classified requests 5.43x as fast as Haiku, comparing median classifier latency: 126.81 ms versus 688.40 ms. Registry-priced classifier cost was 96.12% lower, rounded to 96% in the title

JEV matched our benchmark's expected tiers on 95.00% of calls, versus 73.75% for Haiku. That result depends on the prompts, tier definitions, instructions and context used here. The expected tiers were authored with the synthetic prompts, without independent review. This comparison does not establish general classification accuracy or the quality of the final answers

Benchmarking the LiteLLM Rust AI Gateway: Overhead, Memory, and Cost

Ishaan Jaffer
CTO, LiteLLM

Last Updated: July 2026

We are launching an early beta of the LiteLLM AI Gateway in Rust, and we built AIGatewayBench to measure it against Portkey, Bifrost, and the current LiteLLM Python proxy. Across all four, the LiteLLM Rust gateway has the lowest p99 added latency and the smallest memory footprint by a wide margin: roughly 7x lower overhead and 9x less memory than the next-closest gateway (Bifrost), the lowest cost footprint, and the fastest whole-session times for coding agents. It holds its own on raw sustained throughput and pulls decisively ahead on overhead, memory, and cost.

Migrating LiteLLM to Rust - Building the Fastest and Litest AI Gateway

Ishaan Jaffer
CTO, LiteLLM

Last Updated: June 2026

Over the past year, we have heard the same thing from our users and our community: they want the fastest, most lightweight AI gateway they can run. We have heard you. We are addressing it by moving LiteLLM to Rust, and committing to sub-1ms overhead with a sub-100MB memory binary you can deploy. By the end of this migration, you will get a pure Rust server that can serve 100% of your AI traffic, with every hot path operation, including auth and rate limiting, running in Rust.

Want to help us build it?

We are opening an early beta and want to work directly with teams who care about a fast, lightweight gateway. If that is you, sign up here and we will get you testing the Rust gateway in your own stack, with a direct line to our team.

The reason it matters: under real load, CPU and memory climb with concurrency, and pods get OOM-killed at the worst time. Today the LiteLLM Python proxy peaks around 359MB of memory under load, and that cost multiplies across every pod, region, and retry you run.

We are already seeing the payoff in benchmarks. The Rust gateway serves about 15x the throughput (453 to 6,782 requests per second) on about 11x less memory (359MB to 32MB), and cuts per-request overhead from about 7.5ms on the Python path to about 0.05ms, well under the 1ms we commit to.

What you get​

You deploy a single Rust binary. It uses about 65MB of memory, gateway overhead stays under 1ms, and nothing in your setup changes: same config.yaml, same database, same client API, same providers. You keep LiteLLM's coverage of 100+ LLM providers behind one OpenAI-compatible API, with /chat/completions, /messages, /responses, and every other LLM endpoint LiteLLM supports today, now as the fastest and most lightweight LLM gateway you can self-host.

This is not a v2 and not a rewrite. There is no new major version to migrate to and nothing for you to change. The runtime under the hot path gets faster and lighter while your config stays exactly where it is.

We ship this the careful way. Each route moves to Rust only after it passes our full parity and end-to-end test suite, and it runs in production before the next route starts. Stability is the priority, and we target zero regressions on every release.

LiteLLM Labs: Announcing Lite-Harness SDK — Unified API for Claude Code, Codex, and Pi AI

Krrish Dholakia
CEO, LiteLLM
Ishaan Jaffer
CTO, LiteLLM

Harnesses are the next frontier of vendor lock-in. LiteLLM was built to swap across model providers easily. However, as the models get saturated, the next area for competition becomes the harnesses and managed agents. To make it easy to go across vendors at the harness layer, we're launching the Lite-Harness SDK. This is a simple TypeScript+Python SDK which allows developers to change harnesses, like they change models.

How we built a background agent to cover 30% of our backlog

Krrish Dholakia
CEO, LiteLLM
Ishaan Jaffer
CTO, LiteLLM
LiteLLM Agent Platform: agent.litellm.ai
info

The platform we built is open source. Check out litellm-agent-platform. The swappable harness layer is lite-harness.

Building the same thing inside your company?

Our goal was to 10x the productivity of our company with agents.

Three weeks ago we began building an agent that could own 30% of our engineering tickets. Here's what we've learnt so far.

Announcing Componentized Deployments

Yassin Kortam
Senior SWE @ LiteLLM

Last Updated: May 2026

The LiteLLM proxy container does 2 very different things. It's an LLM data plane, /chat/completions, /v1/messages, embeddings, passthroughs, where latency is measured in single-digit milliseconds of overhead and traffic is high-volume and bursty. It's also a management control plane for keys, teams, SSO, audit logs, and the spend/usage analytics that power the dashboard, where a single request can scan millions of rows.

Run both on the same event loop, and the slowest thing the control plane does sets the reliability floor for the fastest thing the data plane does. This post is about how we've improved LiteLLM's reliability at scale by offering a componentized deployment model.

Making the AI Gateway Resilient to Redis Failures

Ishaan Jaffer
CTO, LiteLLM

Last Updated: April 2026

Enterprise AI Gateway deployments put Redis in the hot path for nearly every request: rate limiting, cache lookups, spend tracking. When Redis is healthy, the latency contribution is single-digit milliseconds, invisible to end users. When it degrades, a production AI Gateway needs to stay up regardless.

Running LiteLLM at scale across 100+ pods means designing for failure modes before they appear. The easy case is Redis going fully down: fail fast, fall through to the database, continue serving requests. The hard case, the one that takes down gateways, is a slow Redis: still accepting connections, still responding, but timing out after 20-30 seconds per operation.