Auto-Inject Prompt Caching Checkpoints

Reduce costs by up to 90% by using LiteLLM to auto-inject prompt caching checkpoints.

Supported Providers (cache_control marker):

Anthropic API (anthropic/)
AWS Bedrock - Claude (bedrock/)
Vertex AI - Claude and Gemini (vertex_ai/)
Google AI Studio - Gemini (gemini/)
Azure AI - Claude (azure_ai/)
OpenRouter - Claude, Gemini, MiniMax, GLM, z-ai routes (openrouter/)
Databricks - Claude (databricks/)
DashScope / Qwen (dashscope/)
MiniMax (minimax/)
Z.ai / GLM (zai/)

Provider Managed (automatic, no marker needed):

OpenAI (openai/)
DeepSeek (deepseek/)
xAI (xai/)

How it works

LiteLLM can automatically inject prompt caching checkpoints into your requests to LLM providers. This allows:

Cost Reduction: Long, static parts of your prompts can be cached to avoid repeated processing
No need to modify your application code: You can configure the auto-caching behavior in the LiteLLM UI or in the litellm config.yaml file.

Who the cache is shared with

This is provider-side prompt caching, a different feature from LiteLLM response caching. It needs no Redis.

The provider caches a prefix against the upstream credentials that sent it, not against the LiteLLM key, team or end user.
Anyone whose request repeats that prefix exactly reuses it. That sharing is the point: a long system prompt cached by one user is reused by everyone else on the same credentials, and an agent benefits from a prefix a user already cached.
The flip side: responses report cache_read_input_tokens, and a cache hit is faster, so a caller repeating a prefix exactly can tell someone else on those credentials sent it recently.
That is bounded. The prefix must match exactly, and providers will not cache a prefix below a minimum size (for Anthropic that is model-dependent, currently 1k to 4k tokens), so it reveals whether a prompt the caller already holds was sent, not its contents.
For isolation, give tenants separate credentials. The boundary is the provider account, not anything LiteLLM can enforce.

Automatic checkpoints for Claude models

info

Requires LiteLLM v1.94.0

Everything below asks you to decide where the checkpoints go. If all you want is the common case for Claude, one flag does it for you.

config.yaml
litellm_settings:
  enable_anthropic_prompt_caching: true

Or, without a config file:

export LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING=true

It is also a switch on the Admin UI, on the General tab under Router Settings.

What it actually does

It adds the same cache_control_injection_points you would have written by hand, one checkpoint on the system prompt and one on the trailing turn, so the stable prefix stays cached while the checkpoint advances with the conversation. That is what a client like Claude Code needs, since it never sets cache_control itself.

Off by default. Upgrading changes nothing until you enable it.
Claude only. anthropic/ and bedrock/ models the cost map marks as supporting prompt caching. Claude on vertex_ai/ and azure_ai/ is not covered; use cache_control_injection_points for those.
Never double-injects. If the request already carries its own cache_control, LiteLLM stands down and the client's checkpoints win.
Explicit config wins. The flag only fills in when no cache_control_injection_points are set.
Respects the 4 block provider limit, counting client-supplied blocks toward it.
Works on /v1/messages and /chat/completions.

Cache lifetime

The default is Anthropic's 5 minute ephemeral cache, which is their API default and what Claude Code uses. For long agentic sessions you can ask for the 1 hour cache instead:

config.yaml
litellm_settings:
  enable_anthropic_prompt_caching: true
  anthropic_prompt_caching_ttl: "1h"

The equivalent environment variable is LITELLM_ANTHROPIC_PROMPT_CACHING_TTL, and the Admin UI exposes it as a dropdown. Note that a 1 hour cache write costs more than a 5 minute one, so it pays off only when the prefix is reused over a longer session.

litellm_settings wins over the environment variable when both are set, since the config is applied after startup.

When to use injection points instead

Reach for cache_control_injection_points, described below, when you need a provider the flag does not cover, a checkpoint somewhere other than the system prompt and the trailing turn, per-model rather than gateway-wide behavior, or location: tool_config to cache tool definitions.

Configuration

You need to specify cache_control_injection_points in your model configuration. This tells LiteLLM:

Where to add the caching directive (location)
Which message to target (role)

LiteLLM will then automatically add a cache_control directive to the specified messages in your requests:

cache_control_directive.json
"cache_control": {
    "type": "ephemeral"
}

LiteLLM Python SDK Usage

Use the cache_control_injection_points parameter in your completion calls to automatically inject caching directives.

Basic Example - Cache System Messages

cache_system_messages.py
from litellm import completion
import os

os.environ["ANTHROPIC_API_KEY"] = ""

response = completion(
    model="anthropic/claude-3-5-sonnet-20240620",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are an AI assistant tasked with analyzing legal documents.",
                },
                {
                    "type": "text",
                    "text": "Here is the full text of a complex legal agreement" * 400,
                },
            ],
        },
        {
            "role": "user",
            "content": "what are the key terms and conditions in this agreement?",
        },
    ],
    # Auto-inject cache control to system messages
    cache_control_injection_points=[
        {
            "location": "message",
            "role": "system",
        }
    ],
)

print(response.usage)

Key Points:

Use cache_control_injection_points parameter to specify where to inject caching
location: "message" targets messages in the conversation
role: "system" targets all system messages
LiteLLM automatically adds cache_control to the last content block of matching messages (per Anthropic's API specification)

LiteLLM's Modified Request:

LiteLLM automatically transforms your request by adding cache_control to the last content block of the system message:

modified_request_system.json
{
    "messages": [
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are an AI assistant tasked with analyzing legal documents."
                },
                {
                    "type": "text",
                    "text": "Here is the full text of a complex legal agreement...",
                    "cache_control": {"type": "ephemeral"}  // Added by LiteLLM
                }
            ]
        },
        {
            "role": "user",
            "content": "what are the key terms and conditions in this agreement?"
        }
    ]
}

Target Specific Messages by Index

You can target specific messages by their index in the messages array. Use negative indices to target from the end.

cache_by_index.py
from litellm import completion
import os

os.environ["ANTHROPIC_API_KEY"] = ""

response = completion(
    model="anthropic/claude-3-5-sonnet-20240620",
    messages=[
        {
            "role": "user",
            "content": "First message",
        },
        {
            "role": "assistant",
            "content": "Response to first",
        },
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Here is a long document to analyze:"},
                {"type": "text", "text": "Document content..." * 500},
            ],
        },
    ],
    # Target the last message (index -1)
    cache_control_injection_points=[
        {
            "location": "message",
            "index": -1,  # -1 targets the last message, -2 would target second-to-last, etc.
        }
    ],
)

print(response.usage)

Important Notes:

When a message has multiple content blocks (like images or multiple text blocks), cache_control is only added to the last content block
This follows Anthropic's API specification which requires: "When using multiple content blocks, only the last content block can have cache_control"
Anthropic has a maximum of 4 blocks with cache_control per request

LiteLLM's Modified Request:

LiteLLM adds cache_control to the last content block of the targeted message (index -1 = last message):

modified_request_index.json
{
    "messages": [
        {
            "role": "user",
            "content": "First message"
        },
        {
            "role": "assistant",
            "content": "Response to first"
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Here is a long document to analyze:"
                },
                {
                    "type": "text",
                    "text": "Document content...",
                    "cache_control": {"type": "ephemeral"}  // Added by LiteLLM to last content block only
                }
            ]
        }
    ]
}

LiteLLM Proxy Usage

You can configure cache control injection in the proxy configuration file.

litellm config.yaml
LiteLLM UI

litellm config.yaml
model_list:
  - model_name: anthropic-auto-inject-cache-system-message
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20240620
      api_key: os.environ/ANTHROPIC_API_KEY
      cache_control_injection_points:
        - location: message
          role: system

On the LiteLLM UI, you can specify the cache_control_injection_points in the Advanced Settings tab when adding a model.

Detailed Example

1. Original Request to LiteLLM

In this example, we have a very long, static system message and a varying user message. It's efficient to cache the system message since it rarely changes.

original_request.json
{
    "messages": [
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a helpful assistant. This is a set of very long instructions that you will follow. Here is a legal document that you will use to answer the user's question."
                }
            ]
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is the main topic of this legal document?"
                }
            ]
        }
    ]
}

2. LiteLLM's Modified Request

LiteLLM auto-injects the caching directive into the system message based on our configuration:

modified_request.json
{
    "messages": [
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a helpful assistant. This is a set of very long instructions that you will follow. Here is a legal document that you will use to answer the user's question.",
                    "cache_control": {"type": "ephemeral"}
                }
            ]
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is the main topic of this legal document?"
                }
            ]
        }
    ]
}

When the model provider processes this request, it will recognize the caching directive and only process the system message once, caching it for subsequent requests.

Manual Prompt Caching - Learn how to manually add cache_control directives to your messages

How it works​

Who the cache is shared with​

Automatic checkpoints for Claude models​

What it actually does​

Cache lifetime​

When to use injection points instead​

Configuration​

LiteLLM Python SDK Usage​

Basic Example - Cache System Messages​

Target Specific Messages by Index​

LiteLLM Proxy Usage​

Detailed Example​

1. Original Request to LiteLLM​

2. LiteLLM's Modified Request​

Related Documentation​