Skip to main content

MongoDB - Vector Store (BETA)

BETA

The MongoDB vector store integration is a BETA feature. It supports searching existing MongoDB Vector Search indexes through the optional LiteLLM MongoDB sidecar. Prepare your collections, indexes, and embedded documents before connecting them to LiteLLM.

Use documents in MongoDB Atlas or a self-managed MongoDB deployment as context for chat completions. LiteLLM embeds the user's query, searches your index, and passes the retrieved text to your chat model. You can also search directly to retrieve documents and similarity scores without generating an answer.

Before you begin​

You need:

  • A MongoDB deployment with Vector Search enabled. For self-managed deployments, follow MongoDB's deployment guide and version compatibility requirements. A MongoDB server without Vector Search cannot serve these queries.
  • A populated collection and a queryable Vector Search index. Documents must contain both readable text and stored embeddings. If you need to create an index, see MongoDB's Vector Search index guide.
  • A connection string reachable from the sidecar. The database user needs permission to search the collection and list its search indexes. Allow the sidecar host through your database's network rules.
  • The embedding model used for your documents. Query embeddings must use the same model and output dimensions as the stored vectors. A different model with the same dimensions can return irrelevant results without an error.

Install LiteLLM normally. The MongoDB driver runs only in the separate sidecar:

pip install 'litellm[proxy]'

For direct Python SDK use, install litellm, deploy the sidecar, then follow Search with the Python SDK. Neither the SDK nor the standard LiteLLM images need PyMongo. LiteLLM does not start or install the sidecar automatically.

RC configuration change

If you tried MongoDB in v1.101.0-rc.1, move mongodb_connection_string to the sidecar's MONGODB_CONNECTION_STRING environment variable. Replace it in your LiteLLM registration with api_base and api_key. Existing MongoDB data and indexes stay in place; application search and chat requests stay the same. This integration remains BETA.

Choose your models​

For proxy requests, use a configured LiteLLM proxy. Registration through the UI or management API also requires a proxy database. Add your models under Models in the Admin UI or in your existing model_list configuration:

ModelUsed forRequirement
Embedding modelConverting each search query into a vectorMust match the model and dimensions used to embed your collection.
Chat modelGenerating an answer from retrieved textA LiteLLM-supported chat model. Only needed for chat completions.

Configure credentials and any provider-specific settings on each model deployment. MongoDB does not require OpenAI: choose the embedding provider that matches your stored vectors and the chat provider you want to use. The sample-document example shows one setup using OpenAI.

Deploy the sidecar​

Run one sidecar per MongoDB connection string. Multiple LiteLLM registrations can use that sidecar for databases, collections, and indexes accessible to its MongoDB user. The sidecar runs the official MongoDB driver; LiteLLM continues to generate query embeddings through your configured model and route chat requests normally.

Set MONGODB_CONNECTION_STRING and a strong MONGODB_SIDECAR_API_KEY in the sidecar's deployment secrets. Give LiteLLM the same sidecar key. Keep MongoDB credentials and TLS files in the sidecar environment.

For a LiteLLM proxy or SDK running on the same host:

docker run --rm --name mongodb-sidecar \
  -p 127.0.0.1:8080:8080 \
  -e MONGODB_CONNECTION_STRING \
  -e MONGODB_SIDECAR_API_KEY \
  ghcr.io/berriai/litellm-mongodb:v0.1.0-beta.1

Use http://127.0.0.1:8080 as the Sidecar URL. Pin the release tag or digest. The image supports Linux amd64 and arm64 and runs as user 10001:10001.

Each secret can alternatively be mounted read-only and referenced with MONGODB_CONNECTION_STRING_FILE or MONGODB_SIDECAR_API_KEY_FILE. Set either the value or its file variable, never both. Files must be readable by container user 10001. For MongoDB TLS, include options such as tlsCAFile or tlsCertificateKeyFile in the URI and mount the files at those paths inside the sidecar. Certificate verification remains enabled by default.

LiteLLM requires HTTPS for remote sidecars. HTTP is accepted only for a literal loopback IP, such as 127.0.0.1 or [::1], when both processes share a host or network namespace. This keeps the sidecar bearer key and query data off unencrypted network hops. For a separate sidecar host or Deployment, use an HTTPS reverse proxy with a certificate trusted by LiteLLM; api_base can include the proxy's path prefix, without /v1. Keep the service's HTTP port private behind that proxy.

Check /health/liveness for the HTTP process and /health/readiness for a bounded MongoDB ping. See the sidecar operations guide for timeouts, connection pooling, and releases.

Connect your index​

Register the existing index once, then reference it by ID in requests. Replace all values in angle brackets with your deployment's values.

ValueWhere to find it
<index-name>The exact name of your MongoDB Vector Search index. This becomes the LiteLLM vector_store_id.
<database-name> / <collection-name>The database and collection containing your documents.
<vector-field>The vector path in the index definition.
<text-field>The document field containing readable text, such as text or metadata.body.
<embedding-model-name>The name of the embedding model registered on your LiteLLM proxy.
<chat-model-name>The name of the chat model registered on your LiteLLM proxy.

The index must be READY and queryable before searching. A registration's display name is independent of its ID. Saving a registration does not create an index, ingest documents, or verify that the connection works.

Choose one registration method:

  1. Open Tools > Vector Stores, select Manage Vector Stores, and click + Add Vector Store.
  2. Select MongoDB (BETA) as the provider. This connector also supports self-managed MongoDB with Vector Search.
  3. Enter the exact index name as Vector Store ID, then fill in Sidecar URL, Sidecar API Key, Database, and Collection.
  4. Select the registered Embedding Model. Set Vector Field Name and Text Field to the fields in your collection. Leave Candidates Considered blank to use the default.
  5. Click Create, then use Test Vector Store to search for something you know is in your documents. Inspect the returned text to verify the connection and field mapping.

Use Manage Vector Stores for registration. The separate Create Vector Store flow creates a new store on the provider and does not support MongoDB.

First registration through the UI or management API

If the proxy started without any registered vector stores, direct search can work before chat retrieval is ready. Wait for the proxy's database sync or restart it after saving the first store before using file_search in chat completions. Loading a store through config.yaml at startup avoids this initial delay.

Use MongoDB in chat completions​

Call /v1/chat/completions with a file_search tool referencing your registered index ID. LiteLLM retrieves the context and calls your chat model in the same request.

Set LITELLM_API_KEY to a LiteLLM key with access to the chat model and registered vector store. Replace the proxy URL, model name, index name, and question below with your own values.

Chat with your MongoDB documents
curl -X POST 'http://localhost:4000/v1/chat/completions' \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<chat-model-name>",
    "messages": [
      {
        "role": "system",
        "content": "Answer using the provided context. If the context does not contain the answer, say you do not know."
      },
      {
        "role": "user",
        "content": "<question-about-your-documents>"
      }
    ],
    "tools": [
      {
        "type": "file_search",
        "vector_store_ids": ["<index-name>"]
      }
    ]
  }'

LiteLLM embeds the final user message with the store's configured embedding model, runs MongoDB's $vectorSearch aggregation, and adds the retrieved text to the conversation. The chat model then generates the answer in choices[0].message.content.

file_search on this endpoint is handled by LiteLLM before calling the chat model. Your application does not need to execute a tool call or send a separate search request. Use the exact index name in vector_store_ids, not the registration's display name.

Verify retrieval​

Successful retrieval returns its documents at choices[0].message.provider_specific_fields.search_results. Inspect their text and scores to confirm that the answer has relevant source material. See Accessing Search Results for more examples.

A successful chat response alone does not prove that MongoDB retrieval worked: chat can complete even if retrieval fails. If sources are missing, run a direct search to diagnose the connection independently.

Search the index directly​

Use direct search when you want to retrieve documents without generating a chat response. Set LITELLM_API_KEY to a LiteLLM key with access to the registered store.

Search through the proxy
curl -X POST 'http://localhost:4000/v1/vector_stores/<index-name>/search' \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "<question-about-your-documents>",
    "max_num_results": 3
  }'

Results appear in the response's data array:

Response fieldMeaning
contentText read from your configured mongodb_text_field.
file_id / filenameThe MongoDB document's _id converted to a string.
scoreMongoDB's vectorSearchScore. Higher values indicate more similar results; the score is not an answer-confidence percentage.

max_num_results defaults to 10 and accepts values from 1 to 50. Use a query relevant to known documents to check search quality. See the sample-document tutorial for a concrete query and expected result.

Search with the Python SDK​

Pass the sidecar settings directly to litellm.vector_stores.search; proxy registration is not required. Set MONGODB_SIDECAR_API_KEY and your embedding provider's credentials in the SDK process's environment. The MongoDB URI belongs in the separately deployed sidecar.

Search without a proxy
import os

import litellm

response = litellm.vector_stores.search(
    vector_store_id="<index-name>",
    query="<question-about-your-documents>",
    custom_llm_provider="mongodb",
    api_base="http://127.0.0.1:8080",
    api_key=os.environ["MONGODB_SIDECAR_API_KEY"],
    mongodb_database="<database-name>",
    mongodb_collection="<collection-name>",
    mongodb_text_field="<text-field>",
    mongodb_embedding_field="<vector-field>",
    litellm_embedding_model="<provider>/<embedding-model>",
    max_num_results=3,
)

print(response)

For async usage, call await litellm.vector_stores.asearch(...) with the same arguments. In direct SDK calls, use the provider's embedding model name. On the proxy, use the registered embedding model name.

Settings reference​

Pass these in the registered store's litellm_params, or as keyword arguments in a direct SDK search.

SettingRequiredDescription
vector_store_idYesExact MongoDB Vector Search index name.
custom_llm_providerYesSet to mongodb.
api_baseYesSidecar HTTPS origin or reverse-proxy path prefix. HTTP is supported only for literal loopback IPs. Do not append /v1.
api_keyYesSidecar bearer key. Can also be supplied through MONGODB_SIDECAR_API_KEY in the LiteLLM environment.
mongodb_databaseYesDatabase containing the collection. Required even if the URI contains a database name.
mongodb_collectionYesCollection containing the documents and vectors.
litellm_embedding_modelYesModel used to embed queries. Must match the model used for stored vectors.
mongodb_embedding_fieldNoVector field covered by the index. Defaults to embedding.
mongodb_text_fieldNoField containing readable text. Defaults to text; supports dotted paths such as metadata.body.
mongodb_num_candidatesNoNumber of candidates considered before returning the top results. Defaults to max(100, 10 * max_num_results). An explicit value must be at least max_num_results and at most 10000. Higher values can improve recall at the cost of latency.
litellm_embedding_configNoAdditional embedding-call arguments, such as dimensions, api_key, or api_base. On the proxy, configure these on the embedding model deployment when possible.

The search query must be non-empty and no longer than 32,000 characters. A list of strings is joined with spaces and embedded as a single query.

Troubleshooting​

SymptomWhat to check
Old configuration asks for pymongo or mongodb_connection_stringUse the LiteLLM release containing the BETA sidecar adapter, deploy the sidecar, and configure api_base and api_key.
HTTP 401: sidecar authentication failedMatch LiteLLM's api_key to the sidecar's MONGODB_SIDECAR_API_KEY. This is separate from your MongoDB password and LiteLLM client key.
HTTP 400: missing or non-queryable indexCheck the exact database, collection, and index name. Wait for the index to become READY and queryable.
HTTP 400: dimension mismatch or vector field not indexedMatch the query embedding model and dimensions to your documents, and mongodb_embedding_field to the index's path.
HTTP 400: none of the matched documents has the text fieldSet mongodb_text_field to the field containing readable text, including its dotted path if nested.
HTTP 400: credentials rejectedCheck the sidecar's URI credentials, authentication database, and database user's permissions.
Atlas reports bad auth : authentication failed (code 8000)Verify the database user's password and cluster in the saved connection string. This is an authentication failure before index search. The database user is separate from your Atlas website login; see Atlas connection troubleshooting.
Sidecar fails to start: invalid URI or missing secretCheck the sidecar environment and secret-file mounts. Percent-encode special characters in URI credentials; for example, p@ss/word becomes p%40ss%2Fword.
TLS file cannot be readEnsure tlsCAFile and tlsCertificateKeyFile point to files readable by the sidecar process. In a container, use paths inside the sidecar container.
HTTP 408: deployment or query timed outCheck hostname resolution and connectivity. On Atlas, check the IP access list and whether the cluster is paused. On self-managed deployments, check the host, port, and firewall.
HTTP 503: connection dropped or refusedCheck that the sidecar is running and its URL is reachable, then check MongoDB connectivity and TLS settings. Retry after a node restart or replica set failover.
Chat returns Invalid value: 'file_search' after registering the first storeWait for database sync or restart the proxy to load the registration before retrying. Confirm vector_store_ids contains the registered index ID.

Timeouts and dropped connections return retryable errors (408 and 503). After connectivity is restored, searches can resume without restarting LiteLLM.

BETA limitations​

  • Search only: create collections and indexes, generate document embeddings, and ingest documents outside LiteLLM. MongoDB does not support LiteLLM's vector store creation, file management, or /rag/ingest APIs.
  • No search filtering or query rewriting: filters, ranking_options, and rewrite_query are rejected, including rewrite_query: false. Provider-specific mongodb_filter is also unsupported and rejected.
  • No automatic embedding through MongoDB: configure litellm_embedding_model; MongoDB's automated embedding integration is not used.
  • First registration can take time to reach chat requests: when a running proxy has no vector store registry yet, its first UI or API registration needs database synchronization or a restart before file_search works in chat completions.
🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →