Skip to main content

📈 Prometheus metrics

LiteLLM Exposes a /metrics endpoint for Prometheus to Poll

Quick Start​

If you're using the LiteLLM CLI with litellm --config proxy_config.yaml then you need to uv add prometheus_client==0.20.0. This is already pre-installed on the litellm Docker image

Add this to your proxy config.yaml

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
litellm_settings:
  callbacks:
    - prometheus

Start the proxy

litellm --config config.yaml --debug

Test Request

curl --location 'http://0.0.0.0:4000/chat/completions' \
    --header 'Content-Type: application/json' \
    --data '{
    "model": "gpt-4o",
    "messages": [
        {
        "role": "user",
        "content": "what llm are you"
        }
    ]
}'

View Metrics on /metrics:

curl http://localhost:4000/metrics \
  -H "Authorization: Bearer sk-..."

Multiple Workers​

When using LiteLLM with multiple workers, you need to set the PROMETHEUS_MULTIPROC_DIR environment variable to enable aggregated metric collection across worker processes.

export PROMETHEUS_MULTIPROC_DIR="/prometheus_multiproc"

This directory is used by the Prometheus client library to store metric files that can be shared across multiple worker processes. Make sure the directory exists and is writable by your LiteLLM process.

Virtual Keys, Teams, Internal Users​

Use this for tracking per user, key, team, etc.

Metric NameDescription
litellm_spend_metricTotal Spend, per "end_user", "hashed_api_key", "api_key_alias", "model", "team", "team_alias", "user", "user_email", "client_ip", "user_agent", "requested_model", "model_id", "api_provider", "service_tier"
litellm_total_tokens_metricinput + output tokens per "end_user", "hashed_api_key", "api_key_alias", "model", "team", "team_alias", "user", "user_email", "requested_model", "model_id", "api_provider"
litellm_input_tokens_metricinput tokens per "end_user", "hashed_api_key", "api_key_alias", "model", "team", "team_alias", "user", "user_email", "requested_model", "model_id", "api_provider"
litellm_output_tokens_metricoutput tokens per "end_user", "hashed_api_key", "api_key_alias", "model", "team", "team_alias", "user", "user_email", "requested_model", "model_id", "api_provider"

Token type detail metrics​

Per-token-type counters that break out the usage.prompt_tokens_details and usage.completion_tokens_details fields providers report (e.g. OpenAI prompt caching, Anthropic prompt caching, audio I/O, reasoning tokens). These are additive to the totals above — the existing litellm_input_tokens_metric / litellm_output_tokens_metric / litellm_total_tokens_metric counters are unchanged.

Each detail counter is sparse: it is only incremented when the provider reports a non-zero value for the corresponding field, so providers that don't expose a given detail will not produce a series for it. The label set is identical to the parent input / output token counter, so you can join cleanly in PromQL.

Metric NameSource field on usageTypical providers
litellm_input_cached_tokens_metricprompt_tokens_details.cached_tokensOpenAI prompt cache, Anthropic cache_read_input_tokens, DeepSeek prompt_cache_hit_tokens
litellm_input_cache_creation_tokens_metricprompt_tokens_details.cache_creation_tokensAnthropic cache_creation_input_tokens (prompt cache writes)
litellm_input_audio_tokens_metricprompt_tokens_details.audio_tokensOpenAI gpt-4o-audio-*, Gemini audio inputs
litellm_output_reasoning_tokens_metriccompletion_tokens_details.reasoning_tokensOpenAI o1-* / o3-*, Anthropic extended thinking
litellm_output_audio_tokens_metriccompletion_tokens_details.audio_tokensOpenAI gpt-4o-audio-* audio outputs

Example PromQL — cache-hit ratio for a model group:

sum by (requested_model) (rate(litellm_input_cached_tokens_metric_total[5m]))
/
sum by (requested_model) (rate(litellm_input_tokens_metric_total[5m]))

Example PromQL — reasoning-token share of output:

sum by (requested_model) (rate(litellm_output_reasoning_tokens_metric_total[5m]))
/
sum by (requested_model) (rate(litellm_output_tokens_metric_total[5m]))
info

litellm_input_cached_tokens_metric tracks provider-side prompt-cache reads (the provider reports a cached portion of the input). This is different from litellm_cached_tokens_metric, which tracks LiteLLM's own response-cache hits (the entire response was served from LiteLLM's cache and no provider request was made).

Cache Metrics​

Track LiteLLM's own response cache. All three metrics carry the labels "model", "hashed_api_key", "api_key_alias", "team", "team_alias", "end_user", "user", "model_id", "api_provider".

Metric NameDescription
litellm_cache_hits_metricRequests served entirely from LiteLLM's response cache (no provider request made)
litellm_cache_misses_metricRequests that missed LiteLLM's response cache
litellm_cached_tokens_metricTokens served from LiteLLM's response cache

Team - Budget​

Metric NameDescription
litellm_team_max_budget_metricMax Budget for Team Labels: "team", "team_alias"
litellm_remaining_team_budget_metricRemaining Budget for Team (A team created on LiteLLM) Labels: "team", "team_alias"
litellm_team_budget_remaining_hours_metricHours before the team budget is reset Labels: "team", "team_alias"

Virtual Key - Budget​

Metric NameDescription
litellm_api_key_max_budget_metricMax Budget for API Key Labels: "hashed_api_key", "api_key_alias"
litellm_remaining_api_key_budget_metricRemaining Budget for API Key (A key Created on LiteLLM) Labels: "hashed_api_key", "api_key_alias"
litellm_api_key_budget_remaining_hours_metricHours before the API Key budget is reset Labels: "hashed_api_key", "api_key_alias"

Internal User - Budget​

Only emitted for requests attached to an internal user; the max budget and reset hours gauges additionally require the user to have a budget set. Use Initialize Budget Metrics on Startup to emit them for all users on a schedule.

Metric NameDescription
litellm_remaining_user_budget_metricRemaining Budget for Internal User Labels: "user"
litellm_user_max_budget_metricMax Budget for Internal User Labels: "user"
litellm_user_budget_remaining_hours_metricHours before the Internal User budget is reset Labels: "user"

Organization - Budget​

Only emitted for requests attached to an organization, with the same conditions as the internal user metrics above.

Metric NameDescription
litellm_remaining_org_budget_metricRemaining Budget for Organization Labels: "org_id", "org_alias"
litellm_org_max_budget_metricMax Budget for Organization Labels: "org_id", "org_alias"
litellm_org_budget_remaining_hours_metricHours before the Organization budget is reset Labels: "org_id", "org_alias"

Virtual Key - Rate Limit​

Metric NameDescription
litellm_remaining_api_key_requests_for_modelRemaining Requests for a LiteLLM virtual API key, only if a model-specific rate limit (rpm) has been set for that virtual key. Labels: "hashed_api_key", "api_key_alias", "model", "model_id"
litellm_remaining_api_key_tokens_for_modelRemaining Tokens for a LiteLLM virtual API key, only if a model-specific token limit (tpm) has been set for that virtual key. Labels: "hashed_api_key", "api_key_alias", "model", "model_id"

Initialize Budget Metrics on Startup​

If you want litellm to emit the budget metrics for all keys, teams irrespective of whether they are getting requests or not, set prometheus_initialize_budget_metrics to true in the config.yaml

How this works:

  • If the prometheus_initialize_budget_metrics is set to true
    • Every 5 minutes litellm runs a cron job to read all keys, teams, internal users, and organizations from the database
    • It then emits the budget metrics for each of them
    • This is used to populate the budget metrics on the /metrics endpoint
litellm_settings:
  callbacks: ["prometheus"]
  prometheus_initialize_budget_metrics: true

Pod Health Metrics​

Use these to measure per-pod queue depth and diagnose latency that occurs before LiteLLM starts processing a request.

Metric NameTypeDescription
litellm_in_flight_requestsGaugeNumber of HTTP requests currently in-flight on this uvicorn worker. Tracks the pod's queue depth in real time. With multiple workers, values are summed across all live workers (livesum).

When to use this​

LiteLLM measures latency from when its handler starts. If a request waits in uvicorn's event loop before the handler runs, that wait is invisible to LiteLLM's own logs. litellm_in_flight_requests shows how loaded the pod was at any point in time.

high in_flight_requests + high ALB TargetResponseTime → pod overloaded, scale out
low  in_flight_requests + high ALB TargetResponseTime → delay is pre-ASGI (event loop blocking)

You can also check the current value directly without Prometheus:

curl http://localhost:4000/health/backlog \
  -H "Authorization: Bearer sk-..."
# {"in_flight_requests": 47}

Proxy Level Tracking Metrics​

Use this to track overall LiteLLM Proxy usage.

  • Track Actual traffic rate to proxy
  • Number of client side requests and failures for requests made to proxy
Metric NameDescription
litellm_proxy_failed_requests_metricTotal number of failed responses from proxy - the client did not get a success response from litellm proxy. Labels: "end_user", "hashed_api_key", "api_key_alias", "requested_model", "team", "team_alias", "user", "user_email", "exception_status", "exception_class", "route", "client_ip", "user_agent", "model_id", "api_provider"
litellm_proxy_total_requests_metricTotal number of requests made to the proxy server - track number of client side requests. Labels: "end_user", "hashed_api_key", "api_key_alias", "requested_model", "team", "team_alias", "user", "status_code", "user_email", "route", "client_ip", "user_agent", "model_id", "api_provider". Optionally includes "stream" — see Emit Stream Label.

Callback Logging Metrics​

Monitor failures while shipping logs to downstream callbacks like S3 cold storage

Metric NameDescription
litellm_callback_logging_failures_metricTotal number of failed attempts to emit logs to a configured callback. Labels: "callback_name". Use this to alert on callback delivery issues such as repeated failures when writing to S3, langfuse, or langfuse_otel and other otel providers

Supported Callbacks:

  • S3Logger - S3 cold storage failures
  • langfuse - Langfuse logging failures
  • otel - OpenTelemetry logging failures

Guardrail Metrics​

Only emitted when guardrails run.

Metric NameDescription
litellm_guardrail_requests_totalTotal number of guardrail invocations. Labels: "guardrail_name", "status", "hook_type"
litellm_guardrail_errors_totalTotal number of errors encountered during guardrail execution. Labels: "guardrail_name", "error_type", "hook_type"
litellm_guardrail_latency_secondsHistogram of guardrail execution latency (seconds). Labels: "guardrail_name", "status", "error_type", "hook_type"

MCP Gateway Metrics​

Only emitted for MCP tool calls made through the MCP gateway. Both metrics carry the labels "mcp_tool_name", "mcp_server_name", "hashed_api_key", "api_key_alias", "team", "team_alias", "user", "end_user".

Metric NameDescription
litellm_mcp_tool_calls_totalTotal MCP tool calls, segmented by tool and server name
litellm_mcp_tool_call_spend_metricTotal spend on MCP tool calls; only incremented when the call has a nonzero cost

Managed Batch Metrics​

Enterprise only. Emitted when using LiteLLM managed batches and the CheckBatchCost background job that cost-tracks them.

Metric NameDescription
litellm_managed_batch_created_totalTotal number of managed batches created. Labels: "model", "api_provider", "user", "user_email", "api_key_alias"
litellm_managed_file_created_totalTotal number of managed files created. Labels: "model", "api_provider", "user", "user_email", "api_key_alias"
litellm_managed_file_deleted_totalTotal number of managed file deletions, success or blocked. Labels: "result"
litellm_managed_file_size_bytesSize of the most recent managed batch file in bytes, last-seen value per label combination. Labels: "purpose", "file_type", "model", "api_provider", "user"
litellm_managed_batch_duration_secondsHistogram of completed batch duration in seconds (completed_at - created_at). Labels: "model", "api_provider"
litellm_check_batch_cost_jobs_polledNumber of unprocessed batches found by the last CheckBatchCost poll
litellm_check_batch_cost_jobs_processed_totalTotal number of batches successfully cost-tracked by CheckBatchCost. Labels: "model", "api_provider"
litellm_check_batch_cost_errors_totalTotal number of errors in CheckBatchCost by error type. Labels: "error_type"
litellm_check_batch_cost_last_run_timestampUnix timestamp of the last CheckBatchCost job run

LLM Provider Metrics​

Use this for LLM API Error monitoring and tracking remaining rate limits and token limits

Labels Tracked​

LabelDescription
litellm_model_nameThe name of the LLM model used by LiteLLM
requested_modelThe model sent in the request
model_idThe model_id of the deployment. Autogenerated by LiteLLM, each deployment has a unique model_id
api_baseThe API Base of the deployment
api_providerThe LLM API provider, used for the provider. Example (azure, openai, vertex_ai)
hashed_api_keyThe hashed api key of the request
api_key_aliasThe alias of the api key used
teamThe team of the request
team_aliasThe alias of the team used
exception_statusThe status of the exception, if any
exception_classThe class of the exception, if any

Success and Failure​

Metric NameDescription
litellm_deployment_success_responsesTotal number of successful LLM API calls for deployment. Labels: "requested_model", "litellm_model_name", "model_id", "api_base", "api_provider", "hashed_api_key", "api_key_alias", "team", "team_alias", "client_ip", "user_agent"
litellm_deployment_failure_responsesTotal number of failed LLM API calls for a specific LLM deployment. Labels: "requested_model", "litellm_model_name", "model_id", "api_base", "api_provider", "exception_status", "exception_class", "hashed_api_key", "api_key_alias", "team", "team_alias", "client_ip", "user_agent"
litellm_deployment_total_requestsTotal number of LLM API calls for deployment - success + failure. Labels: "requested_model", "litellm_model_name", "model_id", "api_base", "api_provider", "hashed_api_key", "api_key_alias", "team", "team_alias", "client_ip", "user_agent"

Remaining Requests and Tokens​

Metric NameDescription
litellm_remaining_requests_metricTrack x-ratelimit-remaining-requests returned from LLM API Deployment. Labels: "model_group", "api_provider", "api_base", "litellm_model_name", "hashed_api_key", "api_key_alias", "model_id"
litellm_remaining_tokens_metricTrack x-ratelimit-remaining-tokens return from LLM API Deployment. Labels: "model_group", "api_provider", "api_base", "litellm_model_name", "hashed_api_key", "api_key_alias", "model_id"

Provider Budget​

Metric NameDescription
litellm_provider_remaining_budget_metricRemaining budget for an LLM provider; only emitted when provider budget routing is configured. Labels: "api_provider"

Deployment State​

Metric NameDescription
litellm_deployment_stateThe state of the deployment: 0 = healthy, 1 = partial outage, 2 = complete outage. Labels: "litellm_model_name", "model_id", "api_base", "api_provider"
litellm_deployment_latency_per_output_tokenLatency per output token for deployment. Labels: "litellm_model_name", "model_id", "api_base", "api_provider", "hashed_api_key", "api_key_alias", "team", "team_alias"
litellm_deployment_tpm_limitConfigured TPM limit for the deployment; only set for deployments with a tpm limit in config. Labels: "litellm_model_name", "model_id", "api_base", "api_provider"
litellm_deployment_rpm_limitConfigured RPM limit for the deployment; only set for deployments with an rpm limit in config. Labels: "litellm_model_name", "model_id", "api_base", "api_provider"

Fallback (Failover) Metrics​

Metric NameDescription
litellm_deployment_cooled_downNumber of times a deployment has been cooled down by LiteLLM load balancing logic. Labels: "litellm_model_name", "model_id", "api_base", "api_provider", "exception_status"
litellm_deployment_successful_fallbacksNumber of successful fallback requests from primary model -> fallback model. Labels: "requested_model", "fallback_model", "hashed_api_key", "api_key_alias", "team", "team_alias", "exception_status", "exception_class", "model_id"
litellm_deployment_failed_fallbacksNumber of failed fallback requests from primary model -> fallback model. Labels: "requested_model", "fallback_model", "hashed_api_key", "api_key_alias", "team", "team_alias", "exception_status", "exception_class", "model_id"

Request Counting Metrics​

Metric NameDescription
litellm_requests_metricdeprecated use litellm_proxy_total_requests_metric. Total number of LLM calls to litellm, tracked per API key, team, user. Labels: "end_user", "hashed_api_key", "api_key_alias", "model", "team", "team_alias", "user", "user_email", "client_ip", "user_agent", "requested_model", "model_id", "api_provider"

Request Latency Metrics​

Metric NameDescription
litellm_request_total_latency_metricTotal latency (seconds) for a request to LiteLLM Proxy Server - tracked for labels "end_user", "hashed_api_key", "api_key_alias", "requested_model", "team", "team_alias", "user", "model", "model_id", "api_provider", "service_tier"
litellm_overhead_latency_metricLatency overhead (seconds) added by LiteLLM processing - tracked for labels "model_group", "api_provider", "api_base", "litellm_model_name", "hashed_api_key", "api_key_alias", "model_id"
litellm_overhead_with_guardrails_latency_metricLatency overhead (seconds) added by LiteLLM processing including pre_call and post_call guardrails - tracked for labels "model_group", "api_provider", "api_base", "litellm_model_name", "hashed_api_key", "api_key_alias", "model_id". During_call (moderation) guardrails run concurrently with the LLM API call, so they are excluded from this number
litellm_llm_api_latency_metricLatency (seconds) for just the LLM API call - tracked for labels "model", "hashed_api_key", "api_key_alias", "team", "team_alias", "requested_model", "end_user", "user", "model_id", "api_provider", "service_tier"
litellm_llm_api_time_to_first_token_metricTime to first token for LLM API call - tracked for labels model, hashed_api_key, api_key_alias, team, team_alias, requested_model, end_user, user, model_id, api_provider, service_tier [Note: only emitted for streaming requests]
litellm_request_queue_time_secondsTime (seconds) a request spent queued inside the proxy between arrival and the start of processing - tracked for labels "end_user", "hashed_api_key", "api_key_alias", "requested_model", "team", "team_alias", "user", "model", "model_id", "api_provider". Pairs well with litellm_in_flight_requests for diagnosing pod overload

Segmenting latency and spend by service tier​

The service_tier label on the request latency metrics and on litellm_spend_metric carries the tier a request actually ran on, so you can compare latency and cost before and after moving traffic to a cheaper or a faster tier:

histogram_quantile(0.95, sum by (service_tier, le) (rate(litellm_request_total_latency_metric_bucket[5m])))

The value is the tier the provider reports it served, which is what makes a request sent with "service_tier": "auto" show up under the concrete tier the provider picked (for example default) rather than under auto. When a provider reports no tier, the tier named in the request is used, and a request that neither asks for nor gets a tier is labelled with an empty value

Tracking end_user on Prometheus​

By default LiteLLM does not track end_user on Prometheus. This is done to reduce the cardinality of the metrics from LiteLLM Proxy.

If you want to track end_user on Prometheus, you can do the following:

config.yaml
litellm_settings:
  callbacks: ["prometheus"]
  enable_end_user_cost_tracking_prometheus_only: true

Emit Stream Label​

Add a stream label to litellm_proxy_total_requests_metric to split requests by streaming vs. non-streaming. Disabled by default.

config.yaml
litellm_settings:
  callbacks: ["prometheus"]
  prometheus_emit_stream_label: true

When enabled, litellm_proxy_total_requests_metric gains a stream label with values "True", "False", or "None".

litellm_proxy_total_requests_metric{..., stream="True"} 42
litellm_proxy_total_requests_metric{..., stream="False"} 100
note

This label is opt-in because adding a new label to an existing metric changes its cardinality and breaks existing Prometheus queries / Grafana dashboards that target this metric. Enable it only on fresh deployments or when you are ready to update your dashboards.

[BETA] Custom Metrics​

Track custom metrics on prometheus on all events mentioned above.

Custom Metadata Labels​

  1. Define the custom metadata labels in the config.yaml
model_list:
  - model_name: openai/gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  callbacks: ["prometheus"]
  custom_prometheus_metadata_labels: ["metadata.foo", "metadata.bar"]
  1. Make a request with the custom metadata labels
curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <LITELLM_API_KEY>' \
-d '{
    "model": "openai/gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What's in this image?"
          }
        ]
      }
    ],
    "max_tokens": 300,
    "metadata": {
        "foo": "hello world"
    }
}'
  1. Check your /metrics endpoint for the custom metrics
... "metadata_foo": "hello world" ...

Custom Tags​

Track specific tags as prometheus labels for better filtering and monitoring.

  1. Define the custom tags in the config.yaml
model_list:
  - model_name: openai/gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  callbacks: ["prometheus"]
  custom_prometheus_metadata_labels: ["metadata.foo", "metadata.bar"]
  custom_prometheus_tags: 
    - "prod"
    - "staging"
    - "batch-job"
    - "User-Agent: RooCode/*"
    - "User-Agent: claude-cli/*"
  1. Make a request with tags
curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <LITELLM_API_KEY>' \
-d '{
    "model": "openai/gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What's in this image?"
          }
        ]
      }
    ],
    "max_tokens": 300,
    "metadata": {
        "tags": ["prod", "user-facing"]
    }
}'
  1. Check your /metrics endpoint for the custom tag metrics
... "tag_prod": "true", "tag_staging": "false", "tag_batch_job": "false" ...

How Custom Tags Work:

  • Each configured tag becomes a boolean label in prometheus metrics
  • If a tag matches (exact or wildcard), the label value is "true", otherwise "false"
  • Tag names are sanitized for prometheus compatibility (e.g., "batch-job" becomes "tag_batch_job")
  • Wildcard patterns supported using * (e.g., "User-Agent: RooCode/*" matches "User-Agent: RooCode/1.0.0")

Example with wildcards:

litellm_settings:
  callbacks: ["prometheus"]
  custom_prometheus_tags:
    - "User-Agent: RooCode/*"
    - "User-Agent: claude-cli/*"

Use Cases:

  • Environment tracking (prod, staging, dev)
  • Request type classification (batch-job, user-facing, background)
  • Feature flags (new-feature, beta-users)
  • Team or service identification (team-a, service-xyz)
  • User-Agent Tracking - use this to track how much Roo Code, Claude Code, Gemini CLI are used (User-Agent: RooCode/*, User-Agent: claude-cli/*, User-Agent: gemini-cli/*)

Configuring Metrics and Labels​

You can selectively enable specific metrics and control which labels are included to optimize performance and reduce cardinality.

Enable Specific Metrics and Labels​

Configure which metrics to emit by specifying them in prometheus_metrics_config. Each configuration group needs a group name (for organization) and a list of metrics to enable. You can optionally include a list of include_labels to filter the labels for the metrics.

model_list:
 - model_name: gpt-4o
    litellm_params:
      model: gpt-4o

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_metrics_config:
    # High-cardinality metrics with minimal labels
    - group: "proxy_metrics"
      metrics:
        - "litellm_proxy_total_requests_metric"
        - "litellm_proxy_failed_requests_metric"
      include_labels:
        - "hashed_api_key"
        - "requested_model"
        - "model_group"

On starting up LiteLLM if your metrics were correctly configured, you should see the following on your container logs

Filter Labels Per Metric​

Control which labels are included for each metric to reduce cardinality:

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_metrics_config:
    - group: "token_consumption"
      metrics:
        - "litellm_input_tokens_metric"
        - "litellm_output_tokens_metric"
        - "litellm_total_tokens_metric"
      include_labels:
        - "model"
        - "team"
        - "hashed_api_key"
    - group: "request_tracking"
      metrics:
        - "litellm_proxy_total_requests_metric"
      include_labels:
        - "status_code"
        - "requested_model"

Advanced Configuration​

You can create multiple configuration groups with different label sets:

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_metrics_config:
    # High-cardinality metrics with minimal labels
    - group: "deployment_health"
      metrics:
        - "litellm_deployment_success_responses"
        - "litellm_deployment_failure_responses"
      include_labels:
        - "api_provider"
        - "requested_model"
    
    # Budget metrics with full label set
    - group: "budget_tracking"
      metrics:
        - "litellm_remaining_team_budget_metric"
      include_labels:
        - "team"
        - "team_alias"
        - "hashed_api_key"
        - "api_key_alias"
        - "model"
        - "end_user"
    
    # Latency metrics with performance-focused labels
    - group: "performance"
      metrics:
        - "litellm_request_total_latency_metric"
        - "litellm_llm_api_latency_metric"
      include_labels:
        - "model"
        - "api_provider"
        - "requested_model"

Configuration Structure:

  • group: A descriptive name for organizing related metrics
  • metrics: List of metric names to include in this group
  • include_labels: (Optional) List of labels to include for these metrics

Default Behavior: If no prometheus_metrics_config is specified, all metrics are enabled with their default labels (backward compatible).

Exclude Metrics and Labels Globally​

prometheus_metrics_config is include-based, so it requires you to enumerate every metric you want to keep. When you only want to drop a small set of metrics or labels, use the global prometheus_exclude_metrics and prometheus_exclude_labels options instead. Everything not listed stays enabled with its default labels, and metrics added in future LiteLLM releases are emitted automatically without further config changes.

litellm_settings:
  callbacks: ["prometheus"]
  # Drop these metrics entirely
  prometheus_exclude_metrics:
    - "litellm_input_tokens_metric"
    - "litellm_output_tokens_metric"
  # Drop these labels from every metric that would otherwise emit them
  prometheus_exclude_labels:
    - "hashed_api_key"
    - "api_key_alias"

Both lists are validated at startup; an unknown metric or label name raises a configuration error so typos surface immediately. Exclusion always wins, so a metric named in prometheus_exclude_metrics is dropped even if a prometheus_metrics_config group enables it, and a label in prometheus_exclude_labels is removed even when a group's include_labels lists it.

Monitor System Health​

To monitor the health of litellm adjacent services (redis / postgres), do:

model_list:
 - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
litellm_settings:
  service_callback: ["prometheus_system"]

Each monitored service emits three metrics, named litellm_<service>_<type>:

Metric NameTypeDescription
litellm_<service>_latencyHistogramLatency (seconds) for calls to the service
litellm_<service>_failed_requests_totalCounterNumber of failed calls to the service. Labels: "error_class", "function_name" for debugging what is failing
litellm_<service>_total_requests_totalCounterTotal number of calls to the service, use it with the failed counter to compute an error rate

The monitored services are:

ServiceDescription
redisRedis calls (caching, rate limiting state)
postgresPostgres DB calls (keys, teams, spend logs). Example: litellm_postgres_latency, litellm_postgres_failed_requests_total, litellm_postgres_total_requests_total
authRequest authentication checks. Useful for catching auth slowness caused by an overloaded DB
batch_write_to_dbBatched spend/usage writes to the DB
reset_budget_jobThe periodic budget reset job
routerLiteLLM router operations
proxy_pre_callPre-call hooks run by the proxy before hitting the LLM API
selfLiteLLM's own API call handling

Example alerts, DB failing or slow:

rate(litellm_postgres_failed_requests_total[5m]) > 0
histogram_quantile(0.95, sum by (le) (rate(litellm_postgres_latency_bucket[5m]))) > 1

DB Transaction Queue Health Metrics​

Use these metrics to monitor the health of the DB Transaction Queue. Eg. Monitoring the size of the in-memory and redis buffers.

Metric NameDescriptionStorage Type
litellm_pod_lock_manager_sizeIndicates which pod has the lock to write updates to the database.Redis
litellm_in_memory_daily_spend_update_queue_sizeNumber of items in the in-memory daily spend update queue. These are the aggregate spend logs for each user.In-Memory
litellm_redis_daily_spend_update_queue_sizeNumber of items in the Redis daily spend update queue. These are the aggregate spend logs for each user.Redis
litellm_redis_daily_end_user_spend_update_queue_sizeNumber of items in the Redis daily end user spend update queue.Redis
litellm_redis_daily_agent_spend_update_queue_sizeNumber of items in the Redis daily agent spend update queue.Redis
litellm_in_memory_spend_update_queue_sizeIn-memory aggregate spend values for keys, users, teams, team members, etc.In-Memory
litellm_redis_spend_update_queue_sizeRedis aggregate spend values for keys, users, teams, etc.Redis

🔥 LiteLLM Maintained Grafana Dashboards​

Link to Grafana Dashboards maintained by LiteLLM

https://github.com/BerriAI/litellm/tree/main/cookbook/litellm_proxy_server/grafana_dashboard

Here is a screenshot of the metrics you can monitor with the LiteLLM Grafana Dashboard

Deprecated Metrics​

Metric NameDescription
litellm_llm_api_failed_requests_metricdeprecated use litellm_proxy_failed_requests_metric

Authentication on /metrics endpoint​

By default, /metrics requires LiteLLM API key authentication (since v1.85.0).

For Prometheus, add authorization to your scrape config:

scrape_configs:
  - job_name: 'litellm'
    authorization:
      type: Bearer
      credentials: <LITELLM_API_KEY>
    static_configs:
      - targets: ['localhost:4000']

To allow unauthenticated access:

litellm_settings:
  require_auth_for_metrics_endpoint: false

FAQ​

What are _created vs. _total metrics?​

The Python Prometheus client emits a _created sample alongside every counter and histogram. It holds the unix timestamp of when that series was created (typically process start, or the first time that label combination was seen), and it never counts anything.

You should consume the _total metrics for your counting purposes

🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →