Skip to main content

Prompt Caching

Supported Providers:

  • OpenAI (openai/)
  • Anthropic API (anthropic/)
  • Google AI Studio (gemini/)
  • Vertex AI (vertex_ai/, vertex_ai_beta/)
  • Bedrock (bedrock/, bedrock/invoke/, bedrock/converse) (All models bedrock supports prompt caching on)
  • Deepseek API (deepseek/)
  • xAI (xai/)
Minimum token requirements

Prompt caching is silently skipped when the input is below the provider's minimum, and no error is returned. Always verify caching occurred by checking cache_creation_input_tokens in the response.

ProviderMinimum input tokens
OpenAI1,024
Anthropic (Claude Opus 5, Fable 5, Mythos 5)512
Anthropic (Claude Sonnet 5, Opus 4.8, Sonnet 4.x, Opus 4, 4.1, Claude 3.x)1,024
Anthropic (Claude Haiku 4.5, Opus 4.5, 4.6)4,096
Bedrock (Claude Opus 5)512
Bedrock (Claude Sonnet 5, Opus 4.8, Sonnet 4.x, Claude 3.5, 3.7)1,024
Bedrock (Claude Haiku 4.5, Opus 4.5, 4.6, 4.7)4,096
Google Gemini1,024

For the supported providers, LiteLLM follows the OpenAI prompt caching usage object format:

"usage": {
  "prompt_tokens": 2006,
  "completion_tokens": 300,
  "total_tokens": 2306,
  "prompt_tokens_details": {
    "cached_tokens": 1920
  },
  "completion_tokens_details": {
    "reasoning_tokens": 0
  }
  # ANTHROPIC_ONLY #
  "cache_creation_input_tokens": 0
}
  • prompt_tokens: These are all prompt tokens including cache-miss and cache-hit input tokens.
  • completion_tokens: These are the output tokens generated by the model.
  • total_tokens: Sum of prompt_tokens + completion_tokens.
  • prompt_tokens_details: Object containing cached_tokens.
    • cached_tokens: Tokens that were a cache-hit for that call.
  • completion_tokens_details: Object containing reasoning_tokens.
  • ANTHROPIC_ONLY: cache_creation_input_tokens are the number of tokens that were written to cache. (Anthropic charges for this).

Quick Start​

Note: OpenAI caching is only available for prompts containing 1024 tokens or more

from litellm import completion 
import os

os.environ["OPENAI_API_KEY"] = ""

for _ in range(2):
    response = completion(
        model="gpt-5.6-terra",
        messages=[
            # System Message
            {
                "role": "system",
                "content": [
                    {
                        "type": "text",
                        "text": "Here is the full text of a complex legal agreement"
                        * 400,
                    }
                ],
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What are the key terms and conditions in this agreement?",
                    }
                ],
            },
            {
                "role": "assistant",
                "content": "Certainly! the key terms and conditions are the following: the contract is 1 year long for $10/mo",
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What are the key terms and conditions in this agreement?",
                    }
                ],
            },
        ],
        temperature=0.2,
        max_tokens=10,
    )

print("response=", response)
print("response.usage=", response.usage)

assert "prompt_tokens_details" in response.usage
assert response.usage.prompt_tokens_details.cached_tokens > 0

OpenAI prompt_cache_key and prompt_cache_retention​

OpenAI prompt caching is automatic; no cache_control message annotations are needed. Any request with 1024+ prompt tokens is eligible for caching.

OpenAI also supports two optional parameters for more control over caching behavior:

  • prompt_cache_key (string): a routing hint that improves cache hit rates for requests sharing long common prefixes. Requests with the same cache key are routed to the same backend, increasing the likelihood of a cache hit.
  • prompt_cache_retention ("in_memory" or "24h"): controls cache TTL. Default is "in_memory" (5–10 min). Set to "24h" for extended caching that offloads KV tensors to GPU-local storage.
from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = ""

response = completion(
    model="gpt-5.6-terra",
    messages=[
        {
            "role": "system",
            "content": "You are an AI assistant tasked with analyzing legal documents. "
            + "Here is the full text of a complex legal agreement " * 400,
        },
        {
            "role": "user",
            "content": "What are the key terms and conditions?",
        },
    ],
    prompt_cache_key="legal-doc-analysis",
    prompt_cache_retention="24h",
)
print(response.usage)

OpenAI explicit breakpoints (GPT-5.6 and newer)​

GPT-5.6 and newer also accept explicit cache breakpoints: a prompt_cache_breakpoint marker on a content block plus a request-level prompt_cache_options that picks the mode (implicit keeps OpenAI's automatic breakpoint on the latest message alongside yours, explicit uses only yours) and the cache ttl (30m). LiteLLM passes both through on /chat/completions, /responses and, for Anthropic-shaped clients, /v1/messages. Models that accept the marker carry supports_prompt_cache_breakpoint: true in the cost map, and a GPT-5.6 or newer OpenAI model name the map has not flagged yet is treated the same way. To have LiteLLM place the marker for you from the deployment config, see the auto-inject tutorial

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = ""

response = completion(
    model="openai/gpt-5.6",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are an AI assistant tasked with analyzing legal documents. "
                    + "Here is the full text of a complex legal agreement " * 400,
                    "prompt_cache_breakpoint": {"mode": "explicit"},
                }
            ],
        },
        {
            "role": "user",
            "content": "What are the key terms and conditions?",
        },
    ],
    prompt_cache_options={"mode": "explicit", "ttl": "30m"},
)
print(response.usage.prompt_tokens_details)

Anthropic Example​

Anthropic charges for cache writes.

Specify the content to cache with "cache_control": {"type": "ephemeral"}.

This same format also works for Gemini / Vertex AI. For other providers, it will be ignored.

from litellm import completion 
import litellm 
import os 

litellm.set_verbose = True # 👈 SEE RAW REQUEST
os.environ["ANTHROPIC_API_KEY"] = "" 

response = completion(
    model="anthropic/claude-sonnet-5",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are an AI assistant tasked with analyzing legal documents.",
                },
                {
                    "type": "text",
                    "text": "Here is the full text of a complex legal agreement" * 400,
                    "cache_control": {"type": "ephemeral"},
                },
            ],
        },
        {
            "role": "user",
            "content": "what are the key terms and conditions in this agreement?",
        },
    ]
)

print(response.usage)
Minimum tokens (Anthropic)

Prompts below the minimum are processed without caching, and no error is returned. Check cache_creation_input_tokens in the response.

ModelMin tokens
Claude Opus 5, Fable 5, Mythos 5512
Claude Sonnet 5, Opus 4.81,024
Claude Opus 4.72,048
Claude Haiku 4.5, Opus 4.5, Opus 4.64,096
Claude Sonnet 4, Sonnet 4.5, Sonnet 4.6, Opus 4, Opus 4.11,024
Claude 3.5 Haiku2,048
Claude 3.x Haiku, Sonnet, Opus, 3.5 Sonnet, 3.7 Sonnet1,024

See Anthropic's prompt caching docs for the full list; these minimums apply on every platform where each model is available.

Bedrock Example​

LiteLLM automatically translates OpenAI-format cache_control markers to Bedrock's native cachePoint format, so no changes are needed to your existing code if you're already using cache_control.

Minimum tokens (Bedrock)

Prompts below the minimum are processed without caching, and no error is returned. Check cache_creation_input_tokens in the response.

Model familyMin tokens per request
Claude Opus 5512
Claude Sonnet 5, Opus 4.81,024
Claude Haiku 4.5, Opus 4.5, Opus 4.6, Opus 4.74,096
Claude Sonnet 4.5, Sonnet 4.61,024
Claude 3.5 Sonnet v2, Claude 3.7 Sonnet1,024

See the Bedrock prompt caching docs for the full per-model table.

import litellm

response = litellm.completion(
    model="bedrock/us.anthropic.claude-sonnet-5",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "<your large system prompt here, at least 1,024 tokens on Claude Sonnet 4.x, 4,096 on Haiku 4.5 and Opus 4.5+>",
                    "cache_control": {"type": "ephemeral"}
                }
            ]
        },
        {"role": "user", "content": "What is prompt caching?"}
    ]
)

print(response.usage)
# cache_creation_input_tokens > 0 on first call (cache written)
# cache_read_input_tokens > 0 on subsequent calls (cache hit)

Supported Bedrock models:

ModelBedrock Model IDMin TokensTTL Options
Claude Opus 5anthropic.claude-opus-55125 min, 1 hour
Claude Sonnet 5anthropic.claude-sonnet-51,0245 min, 1 hour
Claude Opus 4.8anthropic.claude-opus-4-81,0245 min, 1 hour
Claude 3.5 Sonnet v2anthropic.claude-3-5-sonnet-20241022-v2:01,0245 min, 1 hour
Claude 3.7 Sonnetanthropic.claude-3-7-sonnet-20250219-v1:01,0245 min, 1 hour
Claude Opus 4anthropic.claude-opus-4-20250514-v1:01,0245 min, 1 hour
Claude Sonnet 4.5, 4.6us.anthropic.claude-sonnet-4-5-*, us.anthropic.claude-sonnet-4-6-*1,0245 min, 1 hour
Claude Haiku 4.5us.anthropic.claude-haiku-4-5-*4,0965 min, 1 hour
Claude Opus 4.5, 4.6, 4.7us.anthropic.claude-opus-4-5-*, us.anthropic.claude-opus-4-6-*, us.anthropic.claude-opus-4-7-*4,0965 min, 1 hour

Cross-region inference profiles are also supported for the models above.

See the AWS Bedrock prompt caching docs for the full list of supported models and regions.

Google AI Studio / Vertex AI (Gemini) Example​

Use the same Anthropic-style cache_control format; LiteLLM automatically translates it to Google's context caching API.

How it works under the hood:

  1. Messages with cache_control are separated and sent to Google's cachedContents API
  2. The cached content ID is then passed as cachedContent in the Gemini request body
  3. Works across all three providers: gemini/ (Google AI Studio), vertex_ai/, and vertex_ai_beta/
  4. Requires a minimum of 1024 tokens in the cached content. Below that, caching is silently skipped
from litellm import completion
import os

os.environ["GEMINI_API_KEY"] = ""

response = completion(
    model="gemini/gemini-3.8-flash",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are an AI assistant tasked with analyzing legal documents.",
                },
                {
                    "type": "text",
                    "text": "Here is the full text of a complex legal agreement" * 400,
                    "cache_control": {"type": "ephemeral"},
                },
            ],
        },
        {
            "role": "user",
            "content": "what are the key terms and conditions in this agreement?",
        },
    ],
)

print(response.usage)

Vertex AI​

For Vertex AI, use vertex_ai/ prefix:

from litellm import completion

response = completion(
    model="vertex_ai/gemini-3.8-flash",
    vertex_project="my-gcp-project",
    vertex_location="us-central1",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are an AI assistant tasked with analyzing legal documents.",
                },
                {
                    "type": "text",
                    "text": "Here is the full text of a complex legal agreement" * 400,
                    "cache_control": {"type": "ephemeral"},
                },
            ],
        },
        {
            "role": "user",
            "content": "what are the key terms and conditions in this agreement?",
        },
    ],
)

print(response.usage)

Deepeek Example​

Works the same as OpenAI.

from litellm import completion 
import litellm
import os 

os.environ["DEEPSEEK_API_KEY"] = "" 

litellm.set_verbose = True # 👈 SEE RAW REQUEST

model_name = "deepseek/deepseek-chat"
messages_1 = [
    {
        "role": "system",
        "content": "You are a history expert. The user will provide a series of questions, and your answers should be concise and start with `Answer:`",
    },
    {
        "role": "user",
        "content": "In what year did Qin Shi Huang unify the six states?",
    },
    {"role": "assistant", "content": "Answer: 221 BC"},
    {"role": "user", "content": "Who was the founder of the Han Dynasty?"},
    {"role": "assistant", "content": "Answer: Liu Bang"},
    {"role": "user", "content": "Who was the last emperor of the Tang Dynasty?"},
    {"role": "assistant", "content": "Answer: Li Zhu"},
    {
        "role": "user",
        "content": "Who was the founding emperor of the Ming Dynasty?",
    },
    {"role": "assistant", "content": "Answer: Zhu Yuanzhang"},
    {
        "role": "user",
        "content": "Who was the founding emperor of the Qing Dynasty?",
    },
]

message_2 = [
    {
        "role": "system",
        "content": "You are a history expert. The user will provide a series of questions, and your answers should be concise and start with `Answer:`",
    },
    {
        "role": "user",
        "content": "In what year did Qin Shi Huang unify the six states?",
    },
    {"role": "assistant", "content": "Answer: 221 BC"},
    {"role": "user", "content": "Who was the founder of the Han Dynasty?"},
    {"role": "assistant", "content": "Answer: Liu Bang"},
    {"role": "user", "content": "Who was the last emperor of the Tang Dynasty?"},
    {"role": "assistant", "content": "Answer: Li Zhu"},
    {
        "role": "user",
        "content": "Who was the founding emperor of the Ming Dynasty?",
    },
    {"role": "assistant", "content": "Answer: Zhu Yuanzhang"},
    {"role": "user", "content": "When did the Shang Dynasty fall?"},
]

response_1 = litellm.completion(model=model_name, messages=messages_1)
response_2 = litellm.completion(model=model_name, messages=message_2)

# Add any assertions here to check the response
print(response_2.usage)

Calculate Cost​

Cost cache-hit prompt tokens can differ from cache-miss prompt tokens.

Use the completion_cost() function for calculating cost (handles prompt caching cost calculation as well). See more helper functions

cost = completion_cost(completion_response=response, model=model)

Usage​

from litellm import completion, completion_cost
import litellm 
import os 

litellm.set_verbose = True # 👈 SEE RAW REQUEST
os.environ["ANTHROPIC_API_KEY"] = "" 
model = "anthropic/claude-sonnet-5"
response = completion(
    model=model,
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are an AI assistant tasked with analyzing legal documents.",
                },
                {
                    "type": "text",
                    "text": "Here is the full text of a complex legal agreement" * 400,
                    "cache_control": {"type": "ephemeral"},
                },
            ],
        },
        {
            "role": "user",
            "content": "what are the key terms and conditions in this agreement?",
        },
    ]
)

print(response.usage)

cost = completion_cost(completion_response=response, model=model) 

formatted_string = f"${float(cost):.10f}"
print(formatted_string)

Check Model Support​

Check if a model supports prompt caching with supports_prompt_caching()

from litellm.utils import supports_prompt_caching

supports_pc: bool = supports_prompt_caching(model="anthropic/claude-sonnet-5")

assert supports_pc

This checks our maintained model info/cost map

Read More​

Auto-Inject Prompt Caching

Want LiteLLM to automatically add cache_control directives without modifying your code?

See Auto-Inject Prompt Caching Tutorial to learn how to use cache_control_injection_points to automatically cache system messages, specific messages by index, or custom injection patterns.

🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →