--- title: "CLI - Quick Start" url: "/docs/proxy/quick_start" canonical_url: "https://docs.litellm.ai/docs/proxy/quick_start" type: "docs" last_updated: "2026-10-14" summary: "Setup LiteLLM Proxy quickly via CLI." related: - "/docs/proxy/docker_quick_start" - "/docs/benchmarks" --- # CLI - Quick Start > Index of all LiteLLM docs: https://docs.litellm.ai/llms.txt Setup LiteLLM Proxy quickly via CLI. LiteLLM Server (LLM Gateway) manages: * **Unified Interface**: Calling 100+ LLMs [Huggingface/Bedrock/TogetherAI/etc.](/docs/proxy/quick_start#supported-llms) in the OpenAI `ChatCompletions` & `Completions` format * **Cost tracking**: Authentication, Spend Tracking & Budgets [Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys) * **Load Balancing**: between Multiple Models + Deployments of the same model - LiteLLM proxy can handle 1.5k+ requests/second during load tests. ```shell $ uv tool install 'litellm[proxy]' ``` :::warning[Minimum Python version] LiteLLM 1.84.0 and newer require Python 3.10 or higher (`requires-python >=3.10`). `uv tool install` handles this for you by provisioning a compatible Python automatically. A bare `pip install 'litellm[proxy]'` does not; on an older interpreter pip silently resolves down to the last release whose `requires-python` still allowed it (1.83.9 for the previous floor), with no error. If you pinned to an old version unexpectedly, check `python --version` and upgrade to 3.10+ (or use uv), then reinstall ::: ## Quick Start - LiteLLM Proxy CLI Run the following command to start the litellm proxy ```shell $ litellm --model huggingface/bigcode/starcoder #INFO: Proxy running on http://0.0.0.0:4000 ``` :::info Run with `--detailed_debug` if you need detailed debug logs ```shell $ litellm --model huggingface/bigcode/starcoder --detailed_debug ``` ::: ### Test In a new shell, run, this will make an `openai.chat.completions` request. Ensure you're using openai v1.0.0+ ```shell litellm --test ``` This will now automatically route any requests for gpt-3.5-turbo to bigcode starcoder, hosted on huggingface inference endpoints. ### Supported LLMs All LiteLLM supported LLMs are supported on the Proxy. Seel all [supported llms](https://docs.litellm.ai/docs/providers) **AWS Bedrock** ```shell $ export AWS_ACCESS_KEY_ID= $ export AWS_REGION_NAME= $ export AWS_SECRET_ACCESS_KEY= ``` ```shell $ litellm --model bedrock/us.anthropic.claude-sonnet-5 ``` **Azure OpenAI** ```shell $ export AZURE_API_KEY=my-api-key $ export AZURE_API_BASE=my-api-base ``` ``` $ litellm --model azure/my-deployment-name ``` **OpenAI** ```shell $ export OPENAI_API_KEY=my-api-key ``` ```shell $ litellm --model gpt-5.6-luna ``` **Ollama** ``` $ litellm --model ollama/ ``` **OpenAI Compatible Endpoint** ```shell $ export OPENAI_API_KEY=my-api-key ``` ```shell $ litellm --model openai/ --api_base # e.g. http://0.0.0.0:3000 ``` **Vertex AI [Gemini]** ```shell $ export VERTEX_PROJECT="hardy-project" $ export VERTEX_LOCATION="us-west" ``` ```shell $ litellm --model vertex_ai/gemini-3.8-flash ``` **Huggingface (TGI) Deployed** ```shell $ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL] ``` ```shell $ litellm --model huggingface/ --api_base # e.g. http://0.0.0.0:3000 ``` **Huggingface (TGI) Local** ```shell $ litellm --model huggingface/ --api_base http://0.0.0.0:8001 ``` **AWS Sagemaker** ```shell export AWS_ACCESS_KEY_ID= export AWS_REGION_NAME= export AWS_SECRET_ACCESS_KEY= ``` ```shell $ litellm --model sagemaker/jumpstart-dft-meta-textgeneration-llama-2-7b ``` **Anthropic** ```shell $ export ANTHROPIC_API_KEY=my-api-key ``` ```shell $ litellm --model claude-sonnet-5 ``` **VLLM** Assuming you're running vllm locally ```shell $ litellm --model vllm/facebook/opt-125m ``` **TogetherAI** ```shell $ export TOGETHERAI_API_KEY=my-api-key ``` ```shell $ litellm --model together_ai/lmsys/vicuna-13b-v1.5-16k ``` **Replicate** ```shell $ export REPLICATE_API_KEY=my-api-key ``` ```shell $ litellm \ --model replicate/meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3 ``` **Petals** ```shell $ litellm --model petals/meta-llama/Llama-2-70b-chat-hf ``` **Gemini (Google AI Studio)** ```shell $ export GEMINI_API_KEY=my-gemini-key ``` ```shell $ litellm --model gemini/gemini-3.8-flash ``` **AI21** ```shell $ export AI21_API_KEY=my-api-key ``` ```shell $ litellm --model j2-light ``` **Cohere** ```shell $ export COHERE_API_KEY=my-api-key ``` ```shell $ litellm --model command-nightly ``` ## Quick Start - LiteLLM Proxy + Config.yaml The config allows you to create a model list and set `api_base`, `max_tokens` (all litellm params). See more details about the config [here](https://docs.litellm.ai/docs/proxy/configs) ### Create a Config for LiteLLM Proxy Example config ```yaml model_list: - model_name: gpt-5.6-luna # user-facing model alias litellm_params: # all params accepted by litellm.completion() - https://docs.litellm.ai/docs/completion/input model: azure/ api_base: api_key: - model_name: gpt-5.6-luna litellm_params: model: azure/gpt-turbo-small-ca api_base: https://my-endpoint-canada-berri992.openai.azure.com/ api_key: - model_name: vllm-model litellm_params: model: openai/ api_base: # e.g. http://0.0.0.0:3000/v1 api_key: ``` ### Run proxy with config ```shell litellm --config your_config.yaml ``` ## Using LiteLLM Proxy - Curl Request, OpenAI Package, Langchain :::info LiteLLM is compatible with several SDKs - including OpenAI SDK, Anthropic SDK, Mistral SDK, LLamaIndex, Langchain (Js, Python) [More examples here](user_keys) ::: **Curl Request** ```shell curl --location 'http://0.0.0.0:4000/chat/completions' \ --header 'Content-Type: application/json' \ --data ' { "model": "gpt-5.6-luna", "messages": [ { "role": "user", "content": "what llm are you" } ] } ' ``` **OpenAI v1.0.0+** ```python import openai client = openai.OpenAI( api_key="anything", base_url="http://0.0.0.0:4000" ) # request sent to model set on litellm proxy, `litellm --model` response = client.chat.completions.create(model="gpt-5.6-luna", messages = [ { "role": "user", "content": "this is a test request, write a short poem" } ]) print(response) ``` **Langchain** ```python from langchain.chat_models import ChatOpenAI from langchain.prompts.chat import ( ChatPromptTemplate, HumanMessagePromptTemplate, SystemMessagePromptTemplate, ) from langchain.schema import HumanMessage, SystemMessage chat = ChatOpenAI( openai_api_base="http://0.0.0.0:4000", # set openai_api_base to the LiteLLM Proxy model = "gpt-5.6-luna", temperature=0.1 ) messages = [ SystemMessage( content="You are a helpful assistant that im using to make a test request to." ), HumanMessage( content="test from litellm. tell me why it's amazing in 1 sentence" ), ] response = chat(messages) print(response) ``` **Langchain Embeddings** ```python from langchain.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings(model="sagemaker-embeddings", openai_api_base="http://0.0.0.0:4000", openai_api_key="temp-key") text = "This is a test document." query_result = embeddings.embed_query(text) print(f"SAGEMAKER EMBEDDINGS") print(query_result[:5]) embeddings = OpenAIEmbeddings(model="bedrock-embeddings", openai_api_base="http://0.0.0.0:4000", openai_api_key="temp-key") text = "This is a test document." query_result = embeddings.embed_query(text) print(f"BEDROCK EMBEDDINGS") print(query_result[:5]) embeddings = OpenAIEmbeddings(model="bedrock-titan-embeddings", openai_api_base="http://0.0.0.0:4000", openai_api_key="temp-key") text = "This is a test document." query_result = embeddings.embed_query(text) print(f"TITAN EMBEDDINGS") print(query_result[:5]) ``` **LiteLLM SDK** This is **not recommended**. There is duplicate logic as the proxy also uses the sdk, which might lead to unexpected errors. ```python from litellm import completion response = completion( model="openai/gpt-5.6-luna", messages = [ { "role": "user", "content": "this is a test request, write a short poem" } ], api_key="anything", base_url="http://0.0.0.0:4000" ) print(response) ``` **Anthropic Python SDK** ```python import os from anthropic import Anthropic client = Anthropic( base_url="http://localhost:4000", # proxy endpoint api_key="sk-test-proxy-key-123", # litellm proxy virtual key (example) ) message = client.messages.create( max_tokens=1024, messages=[ { "role": "user", "content": "Hello, Claude", } ], model="claude-sonnet-5", ) print(message.content) ``` [**More Info**](./configs.md) ## 📖 Proxy Endpoints - [Swagger Docs](https://docs.litellm.ai/api-reference/) - POST `/chat/completions` - chat completions endpoint to call 100+ LLMs - POST `/completions` - completions endpoint - POST `/embeddings` - embedding endpoint for Azure, OpenAI, Huggingface endpoints - GET `/models` - available models on server - POST `/key/generate` - generate a key to access the proxy ## Debugging Proxy Events that occur during normal operation ```shell litellm --model gpt-5.6-luna --debug ``` Detailed information ```shell litellm --model gpt-5.6-luna --detailed_debug ``` ### Set Debug Level using env variables Events that occur during normal operation ```shell export LITELLM_LOG=INFO ``` Detailed information ```shell export LITELLM_LOG=DEBUG ``` Errors only ```shell export LITELLM_LOG=ERROR ``` `LITELLM_LOG` must be a valid Python logging level (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`). Setting it to `None` makes `import litellm` fail. ## Related pages - [Quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start.md) - [Benchmarks](https://docs.litellm.ai/docs/benchmarks.md)