Skip to main content

Benchmarks

Benchmarks for LiteLLM Gateway (Proxy Server)

Locust Settings:

  • 2500 Users
  • 100 user Ramp Up

Basic Benchmarks​

Overhead when using a Deployed Proxy vs Direct to LLM

  • Latency overhead added by LiteLLM Proxy: 107ms
MetricDirect to Fake EndpointBasic Litellm Proxy
RPS11961133.2
Median Latency (ms)33140

Logging Callbacks​

GCS Bucket Logging​

Using GCS Bucket has no impact on latency, RPS compared to Basic Litellm Proxy

MetricBasic Litellm ProxyLiteLLM Proxy with GCS Bucket Logging
RPS1133.21137.3
Median Latency (ms)140138

LangSmith logging​

Using LangSmith has no impact on latency, RPS compared to Basic Litellm Proxy

MetricBasic Litellm ProxyLiteLLM Proxy with LangSmith
RPS1133.21135
Median Latency (ms)140132