Skip to main content

Alerting / Webhooks

For findings in the Admin UI, see model deprecation alerts and performance alerts. This page covers notifications sent to external services.

Get alerts for:

CategoryAlert Type
LLM PerformanceHanging API calls, Slow API calls, Failed API calls, Model outage alerting
Budget & SpendBudget tracking per key/user, Soft budget alerts, Weekly & Monthly spend reports per Team/Tag
System HealthFailed database read/writes
Daily ReportsTop 5 slowest LLM deployments, Top 5 LLM deployments with most failed requests, Weekly & Monthly spend per Team/Tag

Works across:

Quick Start​

Set up a slack alert channel to receive alerts from proxy.

Step 1: Add a Slack Webhook URL to env​

Get a slack webhook url from https://api.slack.com/messaging/webhooks

You can also use Discord Webhooks, see here

Set SLACK_WEBHOOK_URL in your proxy env to enable Slack alerts.

export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/<>/<>/<>"

If SLACK_WEBHOOK_URL is unset, ALERTING_WEBHOOK_URL is read as a provider-neutral fallback. Use it to send the same Slack-format alerts to any Slack-compatible incoming webhook, such as Rocket.Chat or Mattermost

Step 2: Setup Proxy​

general_settings: 
alerting: ["slack"]
alerting_threshold: 300 # sends alerts if requests hang for 5min+ and responses take 5min+
spend_report_frequency: "1d" # [Optional] set as 1d, 2d, 30d .... Specify how often you want a Spend Report to be sent

# [OPTIONAL ALERTING ARGS]
alerting_args:
daily_report_frequency: 43200 # 12 hours in seconds
report_check_interval: 300 # 5 minutes in seconds
budget_alert_ttl: 86400 # 24 hours in seconds
outage_alert_ttl: 60 # 1 minute in seconds
region_outage_alert_ttl: 60 # 1 minute in seconds
minor_outage_alert_threshold: 5
major_outage_alert_threshold: 10
max_outage_alert_list_size: 10
log_to_console: false

Start proxy

$ litellm --config /path/to/config.yaml

Step 3: Test it!​

curl -X GET 'http://0.0.0.0:4000/health/services?service=slack' \
-H "Authorization: Bearer $LITELLM_API_KEY"

Advanced​

Route actionable alerts​

Give each alert an owner and a response. Route database failures and sustained provider outages to gateway operators. Route budgets and spend reports to the cost owner. Keep individual slow calls in a diagnostic channel; page from a monitoring rule tied to your latency or error target. Compare Flex latency with that workload's deadline before treating it as an incident.

Monitor gateway availability and missing metrics externally. A stopped or blocked gateway cannot reliably send its own outage alert.

This example selects five alert types and routes them to separate Slack incoming webhooks. Store the URLs in environment variables and merge the fragment with your existing configuration:

general_settings:
alerting: [slack]
alert_types: [db_exceptions, outage_alerts, llm_too_slow, budget_alerts, spend_reports]
alert_to_webhook_url:
db_exceptions: os.environ/SLACK_OPERATIONS_WEBHOOK_URL
outage_alerts: os.environ/SLACK_OPERATIONS_WEBHOOK_URL
llm_too_slow: os.environ/SLACK_DIAGNOSTICS_WEBHOOK_URL
budget_alerts: os.environ/SLACK_BUDGET_WEBHOOK_URL
spend_reports: os.environ/SLACK_BUDGET_WEBHOOK_URL
alert_type_config:
llm_too_slow:
digest: true
digest_interval: 300

litellm_settings:
redact_messages_in_exceptions: true

alert_types is an allowlist: other types are excluded. Review the full list before applying it. An enabled type without a mapping uses the default webhook, from SLACK_WEBHOOK_URL or its fallback ALERTING_WEBHOOK_URL. Each mapped environment variable must resolve to a webhook URL.

Keep urgent types such as db_exceptions out of digest mode. A digest delays delivery, retains the first message and an event count for each group, and lives in each proxy process's memory. Restarts can lose pending events, and several workers or pods can send separate summaries. Use your monitoring system for durable aggregation and concise incident pages. These routing settings do not customize alert text. The mapping and digest settings apply to Slack; Teams uses its own destination and sends individual alerts.

Add cooldown notifications to diagnostics when useful. fallback_reports is a scheduled report from Prometheus, not a notification for each fallback; it requires a configured PROMETHEUS_URL and collected fallback metrics.

In staging, trigger the selected events and check their destinations. Confirm that repeated slow calls produce a digest, while urgent events have no digest delay. Inspect payloads for sensitive metadata. See L7 proxy with LiteLLM and Security Best Practices for deployment and security checks.

Redacting Messages from Alerts​

By default alerts show the messages/input passed to the LLM. If you want to redact this from slack alerting set the following setting on your config

general_settings:
alerting: ["slack"]
alert_types: ["spend_reports"]

litellm_settings:
redact_messages_in_exceptions: True

Soft Budget Alerts for Virtual Keys​

Use this to send an alert when a key/team is close to it's budget running out

Step 1. Create a virtual key with a soft budget

Set the soft_budget to 0.001

curl -X 'POST' \
'http://localhost:4000/key/generate' \
-H 'accept: application/json' \
-H 'x-goog-api-key: sk-<your-litellm-api-key>' \
-H 'Content-Type: application/json' \
-d '{
"key_alias": "prod-app1",
"team_id": "113c1a22-e347-4506-bfb2-b320230ea414",
"soft_budget": 0.001
}'

Step 2. Send a request to the proxy with the virtual key

curl http://0.0.0.0:4000/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-Nb5eCf427iewOlbxXIH4Ow" \
-d '{
"model": "openai/gpt-6.1-sol",
"messages": [
{
"role": "user",
"content": "this is a test request, write a short poem"
}
]
}'

Step 3. Check slack for Expected Alert

Add Metadata to alerts​

Add alerting metadata to proxy calls for debugging.

import openai
client = openai.OpenAI(
api_key="anything",
base_url="http://0.0.0.0:4000"
)

# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(
model="gpt-6.1-sol",
messages = [],
extra_body={
"metadata": {
"alerting_metadata": {
"hello": "world"
}
}
}
)

Expected Response

Select specific alert types​

Set alert_types if you want to Opt into only specific alert types. When alert_types is not set, all Default Alert Types are enabled.

See all alert types here

general_settings:
alerting: ["slack"]
alert_types: [
"llm_exceptions",
"llm_too_slow",
"llm_requests_hanging",
"budget_alerts",
"spend_reports",
"db_exceptions",
"daily_reports",
"cooldown_deployment",
"new_model_added",
]

Map slack channels to alert type​

Use this if you want to set specific channels per alert type

This allows you to do the following

llm_exceptions -> go to slack channel #llm-exceptions
spend_reports -> go to slack channel #llm-spend-reports

Set alert_to_webhook_url on your config.yaml

model_list:
- model_name: gpt-6.1-sol
litellm_params:
model: openai/fake
api_key: fake-key
api_base: https://exampleopenaiendpoint-production.up.railway.app/

general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
alerting: ["slack"]
alerting_threshold: 0.0001 # (Seconds) set an artificially low threshold for testing alerting
alert_to_webhook_url: {
"llm_exceptions": "example-slack-webhook-url",
"llm_too_slow": "example-slack-webhook-url",
"llm_requests_hanging": "example-slack-webhook-url",
"budget_alerts": "example-slack-webhook-url",
"db_exceptions": "example-slack-webhook-url",
"daily_reports": "example-slack-webhook-url",
"spend_reports": "example-slack-webhook-url",
"cooldown_deployment": "example-slack-webhook-url",
"new_model_added": "example-slack-webhook-url",
"outage_alerts": "example-slack-webhook-url",
}

litellm_settings:
success_callback: ["langfuse"]

Test it - send a valid llm request - expect to see a llm_too_slow alert in it's own slack channel

curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-d '{
"model": "gpt-6.1-sol",
"messages": [
{"role": "user", "content": "Hello, Claude gm!"}
]
}'

MS Teams Webhooks​

Since v1.100.0

LiteLLM sends alerts to Microsoft Teams natively through the ms_teams alerting destination. Each alert is posted as an Adaptive Card, which both Teams Workflows webhooks and legacy incoming webhook connectors accept

Quick Start​
  1. Create an incoming webhook for your Microsoft Teams channel

  2. Add it to your .env

MS_TEAMS_WEBHOOK_URL="https://<your-tenant>.webhook.office.com/webhookb2/<...>/IncomingWebhook/<...>"
  1. Add ms_teams to alerting in your litellm config. It can sit alongside other destinations such as slack or email
model_list: 
model_name: "azure-model"
litellm_params:
model: "azure/gpt-6-luna"
api_key: "my-bad-key" # bad key

general_settings:
alerting: ["ms_teams"]
alerting_threshold: 300 # sends alerts if requests hang for 5min+ and responses take 5min+
  1. Run health check!

Call the proxy /health/services endpoint with service=ms_teams to send a test alert to your channel

curl --location 'http://0.0.0.0:4000/health/services?service=ms_teams' \
--header "Authorization: Bearer $LITELLM_API_KEY"

Expected Response

{
"status": "success",
"message": "Mock MS Teams Alert sent, verify MS Teams Alert Received in your channel"
}

You can also set the webhook URL from the Admin UI on the Logging & Alerts page under the MS Teams Alerts tab, which saves MS_TEAMS_WEBHOOK_URL and adds ms_teams to alerting for you

alert_types applies to Teams the same way it applies to Slack. alert_to_webhook_url and digest mode only route Slack alerts; Teams alerts always go to MS_TEAMS_WEBHOOK_URL and are sent individually rather than digested

Discord Webhooks​

Discord provides a slack compatible webhook url that you can use for alerting

Quick Start​
  1. Get a webhook url for your discord channel

  2. Append /slack to your discord webhook - it should look like

"https://discord.com/api/webhooks/1240030362193760286/cTLWt5ATn1gKmcy_982rl5xmYHsrM1IWJdmCL1AyOmU9JdQXazrp8L1_PYgUtgxj8x4f/slack"
  1. Add it to your litellm config
model_list: 
model_name: "azure-model"
litellm_params:
model: "azure/gpt-6-luna"
api_key: "my-bad-key" # bad key

general_settings:
alerting: ["slack"]
alerting_threshold: 300 # sends alerts if requests hang for 5min+ and responses take 5min+

environment_variables:
SLACK_WEBHOOK_URL: "https://discord.com/api/webhooks/1240030362193760286/cTLWt5ATn1gKmcy_982rl5xmYHsrM1IWJdmCL1AyOmU9JdQXazrp8L1_PYgUtgxj8x4f/slack"

[BETA] Webhooks for Budget Alerts​

Note: This is a beta feature, so the spec might change.

Set a webhook to get notified for budget alerts.

  1. Setup config.yaml

Add url to your environment, for testing you can use a link from here

export WEBHOOK_URL="https://webhook.site/6ab090e8-c55f-4a23-b075-3209f5c57906"

Add 'webhook' to config.yaml

general_settings: 
alerting: ["webhook"] # KEY CHANGE
  1. Start proxy
litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000
  1. Test it!
curl -X GET --location 'http://0.0.0.0:4000/health/services?service=webhook' \
--header "Authorization: Bearer $LITELLM_API_KEY"

Expected Response

{
"spend": 1, # the spend for the 'event_group'
"max_budget": 0, # the 'max_budget' set for the 'event_group'
"token": "example-api-key-123",
"user_id": "default_user_id",
"team_id": null,
"user_email": null,
"key_alias": null,
"projected_exceeded_date": null,
"projected_spend": null,
"event": "budget_crossed", # see the `event` spec below for all values
"event_group": "user",
"event_message": "User Budget: Budget Crossed"
}

API Spec for Webhook Event​

  • spend float: The current spend amount for the 'event_group'.

  • max_budget float or null: The maximum allowed budget for the 'event_group'. null if not set.

  • soft_budget float or null: The soft budget set for the 'event_group'. null if not set.

  • token str: A hashed value of the key, used for authentication or identification purposes.

  • customer_id str or null: The ID of the customer associated with the event (optional).

  • user_id str or null: The ID of the internal user associated with the event (optional).

  • team_id str or null: The ID of the team associated with the event (optional).

  • team_alias str or null: The alias of the team associated with the event (optional).

  • organization_id str or null: The ID of the organization associated with the event (optional).

  • user_email str or null: The email of the internal user associated with the event (optional).

  • key_alias str or null: An alias for the key associated with the event (optional).

  • projected_exceeded_date str or null: The date when the budget is projected to be exceeded, returned when 'soft_budget' is set for key (optional).

  • projected_spend float or null: The projected spend amount, returned when 'soft_budget' is set for key (optional).

  • event Literal["budget_crossed", "max_budget_alert", "soft_budget_crossed", "threshold_crossed", "projected_limit_exceeded", "key_created", "key_rotated", "internal_user_created", "spend_tracked"]: The type of event that triggered the webhook. key_created, key_rotated, internal_user_created and max_budget_alert are shared with email alerts. Budget webhook values are:

    • "spend_tracked": Emitted whenever spend is tracked for a customer id.
    • "budget_crossed": Indicates that the spend has exceeded the max budget.
    • "soft_budget_crossed": Indicates that the spend has exceeded the soft budget.
    • "threshold_crossed": Indicates that spend has crossed a threshold (currently sent when 85% and 95% of budget is reached).
    • "projected_limit_exceeded": For "key" only - Indicates that the projected spend is expected to exceed the soft budget threshold.
  • event_group Literal["customer", "internal_user", "key", "team", "proxy"]: The group associated with the event. Possible values are:

    • "customer": The event is related to a specific customer
    • "internal_user": The event is related to a specific internal user.
    • "key": The event is related to a specific key.
    • "team": The event is related to a team.
    • "proxy": The event is related to a proxy.
  • event_message str: A human-readable description of the event.

Digest Mode (Reducing Alert Noise)​

By default, LiteLLM sends a separate Slack message for every alert event. For high-frequency alert types like llm_requests_hanging or llm_too_slow, this can produce hundreds of duplicate messages per day.

Digest mode aggregates duplicate alerts within a configurable time window and emits a single summary message with the total count and time range.

Configuration​

Use alert_type_config in general_settings to enable digest mode per alert type:

general_settings:
alerting: ["slack"]
alert_type_config:
llm_requests_hanging:
digest: true
digest_interval: 86400 # 24 hours (default)
llm_too_slow:
digest: true
digest_interval: 3600 # 1 hour
llm_exceptions:
digest: true
# uses default interval (86400 seconds / 24 hours)
ParameterTypeDefaultDescription
digestboolfalseEnable digest mode for this alert type
digest_intervalint86400 (24h)Time window in seconds. Alerts are aggregated within this interval.

How It Works​

  1. When an alert fires for a digest-enabled type, it is grouped by (alert_type, request_model, api_base) instead of being sent immediately
  2. A counter tracks how many times the alert fires within the interval
  3. When the interval expires, a single summary message is sent:
Alert type: `llm_requests_hanging` (Digest)
Level: `Medium`
Start: `2026-02-19 03:27:39`
End: `2026-02-20 03:27:39`
Count: `847`

Message: `Requests are hanging - 600s+ request time`
Request Model: `gemini-3.8-flash`
API Base: `None`

Limitations​

  • Per-instance: Digest state is held in memory per proxy instance. If you run multiple instances (e.g., Cloud Run with autoscaling), each instance maintains its own digest and emits its own summary.
  • Not durable: If an instance is terminated before the digest interval expires, the aggregated alerts for that instance are lost.

Region-outage alerting (Enterprise feature)​

Setup alerts if a provider region is having an outage.

general_settings:
alerting: ["slack"]
alert_types: ["region_outage_alerts"]

By default this will trigger if multiple models in a region fail 5+ requests in 1 minute. '400' status code errors are not counted (i.e. BadRequestErrors).

Control thresholds with:

general_settings:
alerting: ["slack"]
alert_types: ["region_outage_alerts"]
alerting_args:
region_outage_alert_ttl: 60 # time-window in seconds
minor_outage_alert_threshold: 5 # number of errors to trigger a minor alert
major_outage_alert_threshold: 10 # number of errors to trigger a major alert

All Possible Alert Types​

Here is how you can set specific alert types

LLM-related Alerts

Alert TypeDescriptionDefault On
llm_exceptionsAlerts for LLM API exceptions✅
llm_too_slowNotifications for LLM responses slower than the set threshold✅
llm_requests_hangingAlerts for LLM requests that are not completing✅
cooldown_deploymentAlerts when a deployment is put into cooldown✅
new_model_addedNotifications when a new model is added to litellm proxy through /model/new✅
outage_alertsAlerts when a specific LLM deployment is facing an outage✅
region_outage_alertsAlerts when a specific LLM region is facing an outage. Example us-east-1✅

Budget and Spend Alerts

Alert TypeDescriptionDefault On
budget_alertsNotifications related to budget limits or thresholds✅
spend_reportsPeriodic reports on spending across teams or tags✅
failed_tracking_spendAlerts when spend tracking fails✅
daily_reportsDaily Spend reports✅
fallback_reportsWeekly Reports on LLM fallback occurrences✅

Database Alerts

Alert TypeDescriptionDefault On
db_exceptionsNotifications for database-related exceptions✅

MCP Gateway Alerts

Alert TypeDescriptionDefault On
mcp_tool_description_blockedA pre_mcp_call guardrail blocked an upstream MCP tool's description, so the tool was hidden from tools/list. See MCP Guardrails✅
mcp_pinned_tools_changedA pinned MCP server's upstream tool list, descriptions, or input schemas drifted from the pin. See Pin a Server's Tool List✅

Management Endpoint Alerts - Virtual Key, Team, Internal User

Alert TypeDescriptionDefault On
new_virtual_key_createdNotifications when a new virtual key is created❌
virtual_key_updatedAlerts when a virtual key is modified❌
virtual_key_deletedNotifications when a virtual key is removed❌
new_team_createdAlerts for the creation of a new team❌
team_updatedNotifications when team details are modified❌
team_deletedAlerts when a team is deleted❌
new_internal_user_createdNotifications for new internal user accounts❌
internal_user_updatedAlerts when an internal user's details are changed❌
internal_user_deletedNotifications when an internal user account is removed❌

alerting_args Specification​

ParameterDefaultDescription
daily_report_frequency43200 (12 hours)Frequency of receiving deployment latency/failure reports in seconds
report_check_interval300 (5 minutes)How often to check if a report should be sent (background process) in seconds
budget_alert_ttl86400 (24 hours)Cache TTL for budget alerts to prevent spam when budget is crossed
outage_alert_ttl60 (1 minute)Time window for collecting model outage errors in seconds
region_outage_alert_ttl60 (1 minute)Time window for collecting region-based outage errors in seconds
minor_outage_alert_threshold5Number of errors that trigger a minor outage alert (400 errors not counted)
major_outage_alert_threshold10Number of errors that trigger a major outage alert (400 errors not counted)
max_outage_alert_list_size10Maximum number of errors to store in cache per model/region
log_to_consolefalseIf true, prints alerting payload to console as a .warning log.