Prompt Formatting

LiteLLM automatically translates the OpenAI ChatCompletions prompt format, to other models. You can control this by setting a custom prompt template for a model as well.

Stored Templates

Prompt templates only apply to providers that take a single raw text prompt, such as ollama/, petals/, replicate/, sagemaker/, and predibase/. huggingface/ and together_ai/ call the providers' OpenAI-compatible chat completions APIs, so LiteLLM sends your messages as-is and the provider applies the model's own chat template. Neither the stored templates below nor templates registered with register_prompt_template are used for them

For raw-prompt providers, LiteLLM supports Huggingface Chat Templates and falls back to the model's registered chat template on the Hugging Face Hub (e.g. Mistral-7b). On sagemaker/, pass hf_model_name to pick the template for your endpoint's base model. For popular models, the templates are saved as part of the package

Model Name	Works for Models
mistralai/Mistral-7B-Instruct-v0.1	mistralai/Mistral-7B-Instruct-v0.1
meta-llama/Llama-2-7b-chat	All meta-llama llama2 chat models
tiiuae/falcon-7b-instruct	All falcon instruct models
mosaicml/mpt-7b-chat	All mpt chat models
codellama/CodeLlama-34b-Instruct-hf	All codellama instruct models
WizardLM/WizardCoder-Python-34B-V1.0	All wizardcoder models
Phind/Phind-CodeLlama-34B-v2	All phind-codellama models

Jump to code

Format Prompt Yourself

You can also format the prompt yourself. Register the template under the model name without the provider prefix. Here's how:

import litellm
from litellm import completion

# Create your own custom prompt template 
litellm.register_prompt_template(
	    model="llama2",
        initial_prompt_value="You are a good assistant", # [OPTIONAL]
	    roles={
            "system": {
                "pre_message": "[INST] <<SYS>>\n", # [OPTIONAL]
                "post_message": "\n<</SYS>>\n [/INST]\n" # [OPTIONAL]
            },
            "user": { 
                "pre_message": "[INST] ", # [OPTIONAL]
                "post_message": " [/INST]" # [OPTIONAL]
            }, 
            "assistant": {
                "pre_message": "\n", # [OPTIONAL]
                "post_message": "\n" # [OPTIONAL]
            }
        },
        final_prompt_value="Now answer as best you can:" # [OPTIONAL]
)

messages = [{"role": "user", "content": "Hey, how's it going?"}]
response = completion(model="ollama/llama2", messages=messages, api_base="http://localhost:11434")
print(response['choices'][0]['message']['content'])

This is supported for raw-prompt providers such as Ollama (ollama/, not ollama_chat/), Petals, Replicate, SageMaker, and Predibase. It has no effect on huggingface/ or together_ai/ chat models

Other providers either have fixed prompt templates (e.g. Anthropic), or accept chat messages and format them server-side (e.g. Hugging Face, Together AI). If there's a provider we're missing coverage for, let us know!

All Providers

Here's the code for how we format all providers. Let us know how we can improve this further

Provider	Model Name	Code
Anthropic	claude-instant-1, claude-instant-1.2, claude-2	Code
OpenAI Text Completion	text-davinci-003, text-curie-001, text-babbage-001, text-ada-001, babbage-002, davinci-002,	Code
Replicate	all model names starting with replicate/	Code
Cohere	command-nightly, command, command-light, command-medium-beta, command-xlarge-beta, command-r-plus	Code
Huggingface	all model names starting with huggingface/	Code
OpenRouter	all model names starting with openrouter/	Code
AI21	j2-mid, j2-light, j2-ultra	Code
VertexAI	text-bison, text-bison@001, chat-bison, chat-bison@001, chat-bison-32k, code-bison, code-bison@001, code-gecko@001, code-gecko@latest, codechat-bison, codechat-bison@001, codechat-bison-32k	Code
Bedrock	all model names starting with bedrock/	Code
Sagemaker	sagemaker/jumpstart-dft-meta-textgeneration-llama-2-7b	Code
TogetherAI	all model names starting with together_ai/	Code
AlephAlpha	all model names starting with aleph_alpha/	Code
Palm	all model names starting with palm/	Code
NLP Cloud	all model names starting with palm/	Code
Petals	all model names starting with petals/	Code