Skip to main content

Better Benchmarks

LITELLM PYTHONLITELLM RUST

We mentioned last week that we would be posting updated, more representative benchmarks with common features enabled. The headline numbers are in, and they're great!

0150 REQUESTS / SEC · FULL FEATURES

Same load, less overhead

Median overhead0123LiteLLM Rust0.9 msBifrost1.2 msPortkey2.1 msLiteLLM Python79.3 msoff scalep99 overhead0246LiteLLM Rust1.8 msBifrost4.2 msPortkey5.3 msLiteLLM Python1,587.8 msoff scaleCPU used0123LiteLLM Rust0.15 coresBifrost0.41 coresPortkey0.49 coresLiteLLM Python2.92 coresPeak memory05,00010,000LiteLLM Rust357 MiBBifrost2,809 MiBPortkey609 MiBLiteLLM Python9,553 MiB
Lower is better. Five minutes per gateway, 4 cores and 16 GiB each, single run, every request served. Latency subtracts a direct-to-mock run at the same rate. Portkey runs with no features; OpenRouter used a separate test.

Our previous benchmarks were with a much smaller subset of our functionality enabled, and they were not particularly representative of production workflows. We were very clear about this, as we just wanted to validate how much better Rust could do than Python as a language. In order to keep high performance as we build towards an end-to-end request handled in Rust in production, we have to be very mindful about choices made and the hot path of our requests.

With these benchmarks, we wanted to be sure that it represented realistic modern LLM scale and gateway features. These benchmarks are useful for us internally to ensure that our gateway is performing well at the scale that it actually operates. The benchmark covers:

  • Context can go up to 1M tokens, which means requests are ~4 MB
  • Requests can sometimes take 6+ minutes to fully complete
  • Gateways have many virtual keys with team, user, and deployment budget limits
  • Budget tracking, spend logging, prometheus metrics, dumping logs
  • Custom hooks for header logic, RPM/TPM limits on each deployment
  • Many deployments across multiple model groups
  • Retries and fallbacks enabled by the router

What "full features" turns on​

RustPythonBifrostPortkey
1,000 virtual keys✅✅✅❌
Budgets: key, user, team, member✅✅🟡¹❌
rpm/tpm: key, user, team, member✅✅🟡¹❌
max_parallel_requests per key✅✅❌❌
rpm/tpm per deployment✅✅✅❌
Dynamic rate limiter✅✅❌❌
Spend logs and spend tables✅✅🟡²❌
Prometheus metrics✅✅✅❌
S3 request logs✅✅✅❌
Forward and strip headers✅✅✅❌
Spend metadata from headers✅✅✅❌
Team as an upstream header✅✅❌❌
Shuffle over 4 deployments, 2 retries✅✅✅✅
Cooldown: 3 fails, 30 s✅✅❌❌

¹ Key, team and customer only; user and team member limits are enterprise-only. ² Its own logs store and budgets. Rust runs the header features as hooks, Python as custom callbacks, and Bifrost through its built-in equivalents. Portkey's open-source gateway has no auth, keys, limits, budgets, logs or metrics.

The rust implementation is a branch which we used in order to flesh out how we would handle some of our richer featureset, if we are able to rethink it with performance in mind. We expect comparable performance on the fast path which will be available on December 1st. Everything in this branch is compatible with a subset of LiteLLM's extensive featureset. It writes compatibly to the S3 log dumping and prometheus metrics, and writes to our spend logs in the same way as the Python implementation.

Detailed Benchmarks​

02BY REQUEST RATE · FULL FEATURES

LiteLLM Rust gets cheaper per request as load rises

Median overhead1 ms10 ms100 ms1 s10 s100 s10255075100 rpsLiteLLM PythonPortkeyBifrostLiteLLM Rust79% served29% servedp99 overhead1 ms10 ms100 ms1 s10 s100 s10255075100 rpsLiteLLM PythonPortkeyBifrostLiteLLM RustCPU used0 cores1 cores2 cores3 cores4 cores10255075100 rpsLiteLLM PythonBifrostPortkeyLiteLLM RustPeak memory0 GiB4 GiB8 GiB12 GiB16 GiB10255075100 rpsLiteLLM PythonBifrostPortkeyLiteLLM Rust
Latency axes are logarithmic and subtract a direct-to-mock run at the same rate (runs drift about ±0.7 ms). Open circles: Python was overloaded and served only 78.9% of requests at 75 rps and 29.3% at 100 rps. Portkey runs with no features.
031M-TOKEN REQUESTS · ~4 MB EACH · STREAMED

Large requests: flat first byte, flat memory

First-byte overhead, p500 ms25 ms50 ms75 ms100 ms0.51248 rpsPortkeyBifrostLiteLLM RustFirst-byte overhead, p990 ms50 ms100 ms150 ms200 ms0.51248 rpsPortkeyBifrostLiteLLM RustPeak memory0 GiB5 GiB10 GiB15 GiB0.51248 rpsBifrostPortkeyLiteLLM Rust
Basic features; Portkey has none. Three minutes per rate, 4 cores and 16 GiB per pod. First byte includes moving 4 MB over two hops, with no direct-to-mock run subtracted. Portkey served 99.85% at 4 rps; every other point served 100%.

What we did to achieve these benchmarks​

Everyone will say that the first three things to do if you want to improve performance is to measure, measure, and measure again. Following that logic, we have the following waterfalls of a Python Hot vs Cold path timing. These are generated by profiling a request, then scaling it to the average duration of such a request.

04ONE NON-STREAMED CHAT COMPLETION

Where a request spends its time

ALL FOUR · SAME SCALE01020304050 msLiteLLM Python, cached key33.7 msLiteLLM Python, uncached key51.5 msLiteLLM Rust, cached key1.7 msLiteLLM Rust, uncached key4.2 msLITELLM RUST ONLY · 12× ZOOM01234 msLiteLLM Rust, cached key1.7 msLiteLLM Rust, uncached key4.2 msauthlimits and hooksbuild request, call providereverything else
One pinned core per gateway, all callbacks on. Python phases are VizTracer times scaled to the untraced time of the same request; Rust phases come from in-gateway span timers.

We spend time doing virtual key lookup, auth checks, running pre-call hooks in parallel, and then a lot of time building the request for OpenAI. On the cold path, the same main areas are a pain point for performance.

  • Router deployment cooldowns are kept as an in-memory cache and updated in the background
  • Auth lookups are kept in an in-process cache, which can be shared in-memory more effectively in Rust than Python because the python workers are forked processes.
  • Admission hooks run in parallel against Redis instead of serially due to a different callback model that is more explicit about what data gets read and written when
  • on passthrough requests, we send the body upstream without reserializing when possible
05500K-TOKEN STREAM · 2.45 MB BODY

Heap held per open stream

1×2×3×4×copies of the bodyLiteLLM Pythonparsed request3.22 MBraw body2.85 MBjson.dumps for logging2.45 MBoutbound bytes2.43 MB10.95 MBBifrostparsed request3.59 MBre-encodedmessages1.9 MBtrace text1.89 MBtrace summary1.88 MBoutbound(SetBody)1.5 MB11.16 MBinbound read 0.4 MBLiteLLM Rust0.47 MBread buffers and per-call state 0.47 MBLiteLLM Rust + S3 logsbody kept forthe S3 log2.45 MB2.92 MBread buffers and per-call state 0.47 MB
Profiled heap allocations, not process RSS: memray for Python, pprof for Bifrost, heaptrack for Rust, one core per gateway. With S3 logs on, Rust keeps one copy of the body for the stream's life.

Rust LiteLLM is able to keep the absolute minimum of information in memory during a request due to efficient libraries and the semantics of dropping memory after it is no longer referenced. When S3 logging is disabled, it only keeps 0.47MB of state in memory for each 500k token stream.

Benchmark Configuration​

We ran a few different benchmarks, but most of them run on the exact setup, except for the one that includes OpenRouter.

The code to reproduce these runs (the load-test kit, the Kubernetes lane, the generated gateway configs and the data behind every chart) is in BerriAI/ai-gateway-bench: see lanes/k8s for how to run it and results/better-benchmarks for the data.

General Benchmarks​

  • Where: Kubernetes on AWS, every pod on m7i.4xlarge nodes (one instance type, so all gateways get the same CPU). Buildkite pipeline litellm-rust-perf.
  • Gateway pods: one replica per gateway, 4 dedicated cores and 16 GiB each (requests equal to limits). The mock upstream, an S3 stub, Postgres and the load generator run in their own pods in the same cluster.
  • Versions:
    • LiteLLM Rust: built with fat LTO from commit 5990e10f (the litellm_customer_proxy gateway plus the load-test kit).
    • LiteLLM Python: the proxy image from the same commit, 4 workers.
    • Bifrost: the published v2.3.0 image (maximhq/bifrost@sha256:d9caa7c5…).
    • Portkey: the published open-source gateway 1.15.2 (portkeyai/gateway@sha256:552b5399…), run as 4 Node.js processes with Node's cluster module, since one process uses about one core.
  • Mock upstream: OpenAI chat completions, Anthropic messages and Bedrock invoke, each with a scripted time to first token and total time.
  • Request mix, the same requests in the same order for every gateway:
    • half /v1/chat/completions and half /v1/messages, half of each streamed;
    • context 10k to 50k tokens (log-uniform);
    • time to first token 200 to 800 ms, total time 1 to 5 s.
  • Routing: 20 model groups (10 chat, 10 messages) of 4 deployments each, 80 in total. Half of each messages group goes through Bedrock invoke (SigV4 signed). Router: simple-shuffle, 2 retries, 3 allowed fails, 30 s cooldown.
  • Full features (LiteLLM Rust, LiteLLM Python, Bifrost):
    • each gateway has its own 1,000 virtual keys, 100 users and 10 teams;
    • every key, user, team and team member has rpm, tpm and a budget, set far above the test's load so nothing is refused;
    • every key has max_parallel_requests, and every deployment has rpm and tpm;
    • callbacks: dynamic_rate_limiter_v3, prometheus and s3_v2 (into the S3 stub);
    • four request hooks that forward, add and strip headers and attach spend metadata (Python runs equivalent custom callbacks).
  • Bifrost: each feature mapped onto Bifrost's own equivalent. It has no shared state across replicas. Its Bedrock calls reach the mock over TLS; the other gateways' do not.
  • Portkey: no features, because its open-source gateway has no auth, keys, limits, budgets, logs or metrics. Routing comes from an x-portkey-config header per model group (load balance over the same 4 deployments, 2 retries), which Portkey validates on every request. Its Bedrock calls go to the mock over plain HTTP.
  • Overhead: what the load generator observed minus the time the mock was told to take. First-byte overhead uses time to the first SSE event or first body byte.
  • Validity: a run counts only if the load generator shed no requests and sent them within 10 ms of schedule (p99). Every run in builds 28 and 29 passed this.
  • After each run, a checker compares each LiteLLM gateway's spend logs, key, team and user spend, daily tables, Prometheus counters, S3 logs and forwarded headers against what was sent.
  • Single run of each configuration.

Benchmarking OpenRouter​

Getting OpenRouter involved in the benchmarks involved a bit of a clunky process, because unless you have an enterprise account with the addon, you cannot use custom endpoints, which is what we use to take the randomness of normal model serving out of the benchmarks. The amount of noise in large requests to a model like Opus will drown out performance differences unless you run a massive sample size. For that reason, we wanted to find a way around the custom endpoints.

It is not guaranteed that this method will last now that we have made it public, but there is a way to run a performance test against a mock upstream with OpenRouter by leveraging their lax validation of Azure deployments (*.azure.com). You can configure a BYOK key for azure in the legacy fomat and give it a host that is <myprobe>.eastus.cloudapp.azure.com and points to a mock provider that you run in azure cloud. Then, you need to load a little bit of OpenRouter usage credits so that the low balance doesn't force it to higher coherency.

Once we have that, we can put all of our other providers in the eastus region on Azure, and run a benchmark from our runner, separately placed on eastus. Each of the providers reaches the mock upstream via the public internet. It compares four different test lanes:

  • Direct to the mock provider over public internet
  • LiteLLM Rust
  • LiteLLM Python
  • OpenRouter

Because this test is both more challenging to conduct and introduces potential interference from network topology, we used it to get a sense of OpenRouter's latency, but prefer the measurements from other tests.

For this test, we had the following setup:

  • Each self-hosted gateway ran on the same D8s_v4 VM with 2 cores each
  • 8 RPS with poisson arrivals over 20 minutes
  • 50% of requests were streaming, and 500 token prompts
  • LiteLLM and Bifrost were run with their full-featured configs