Skip to main content

Azure Document Intelligence OCR

Overview​

PropertyDetails
DescriptionAzure Document Intelligence (formerly Form Recognizer) provides advanced document analysis capabilities including text extraction, layout analysis, and structure recognition
Provider Route on LiteLLMazure_ai/doc-intelligence/
Supported Operations/ocr
Link to Provider DocAzure Document Intelligence ↗

Extract text and analyze document structure using Azure Document Intelligence's powerful prebuilt models.

Quick Start​

LiteLLM SDK​

SDK Usage
import litellm
import os

# Set environment variables
os.environ["AZURE_DOCUMENT_INTELLIGENCE_API_KEY"] = "your-api-key"
os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] = "https://your-resource.cognitiveservices.azure.com"

# OCR with PDF URL
response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

# Access extracted text
for page in response.pages:
    print(f"Page {page.index}:")
    print(page.markdown)

LiteLLM PROXY​

proxy_config.yaml
model_list:
  - model_name: azure-doc-intel
    litellm_params:
      model: azure_ai/doc-intelligence/prebuilt-layout
      api_key: os.environ/AZURE_DOCUMENT_INTELLIGENCE_API_KEY
      api_base: os.environ/AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT
    model_info:
      mode: ocr

Start Proxy

litellm --config proxy_config.yaml

Call OCR via Proxy

cURL Request
curl -X POST http://localhost:4000/ocr \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "azure-doc-intel",
    "document": {
      "type": "document_url",
      "document_url": "https://arxiv.org/pdf/2201.04234"
    }
  }'

How It Works​

Azure Document Intelligence uses an asynchronous API pattern. LiteLLM AI Gateway handles the request/response transformation and polling automatically.

Complete Flow Diagram​

What LiteLLM Does For You​

When you call litellm.ocr() via SDK or /ocr via Proxy:

  1. Request Transformation: Converts Mistral OCR format → Azure Document Intelligence format
  2. Submits Document: Sends transformed request to Azure DI API
  3. Handles 202 Response: Captures the Operation-Location URL from response headers
  4. Automatic Polling:
    • Polls the operation URL at intervals specified by retry-after header (default: 2 seconds)
    • Continues until status is succeeded or failed
    • Respects Azure's rate limiting via retry-after headers
  5. Response Transformation: Converts Azure DI format → Mistral OCR format
  6. Returns Result: Sends unified Mistral format response to client

Polling Configuration:

  • Default timeout: 120 seconds
  • Configurable via AZURE_OPERATION_POLLING_TIMEOUT environment variable
  • Uses sync (time.sleep()) or async (await asyncio.sleep()) based on call type
info

Typical processing time: 2-10 seconds depending on document size and complexity

Supported Models​

Azure Document Intelligence offers several prebuilt models optimized for different use cases:

Best for general document OCR with structure preservation.

Layout Model - SDK
import litellm
import os

os.environ["AZURE_DOCUMENT_INTELLIGENCE_API_KEY"] = "your-api-key"
os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] = "https://your-resource.cognitiveservices.azure.com"

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

Features:

  • Text extraction, with Markdown output when you ask for it (see Markdown Output)
  • Table detection and extraction
  • Document structure analysis
  • Paragraph and section recognition

Pricing: $10 per 1,000 pages

prebuilt-read​

Optimized for reading text from documents - fastest and most cost-effective.

Read Model - SDK
import litellm
import os

os.environ["AZURE_DOCUMENT_INTELLIGENCE_API_KEY"] = "your-api-key"
os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] = "https://your-resource.cognitiveservices.azure.com"

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-read",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

Features:

  • Fast text extraction
  • Optimized for reading-heavy documents
  • Basic structure recognition

Pricing: $1.50 per 1,000 pages

prebuilt-document​

General-purpose document analysis with key-value pairs.

Document Model - SDK
import litellm
import os

os.environ["AZURE_DOCUMENT_INTELLIGENCE_API_KEY"] = "your-api-key"
os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] = "https://your-resource.cognitiveservices.azure.com"

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-document",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

Pricing: $10 per 1,000 pages

Document Types​

Azure Document Intelligence supports various document formats.

PDF Documents​

PDF OCR
response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

Image Documents​

Image OCR
response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "image_url",
        "image_url": "https://example.com/image.png"
    }
)

Supported image formats: JPEG, PNG, BMP, TIFF

Base64 Encoded Documents​

Base64 PDF
import base64

# Read and encode PDF
with open("document.pdf", "rb") as f:
    pdf_base64 = base64.b64encode(f.read()).decode()

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "document_url",
        "document_url": f"data:application/pdf;base64,{pdf_base64}"
    }
)

Response Format​

Response Structure
# Response has the following structure
response.pages          # List of pages with extracted text
response.model          # Model used
response.object         # "ocr"
response.usage_info     # Token usage information

# Access page content
for page in response.pages:
    print(f"Page {page.index}:")
    print(page.markdown)
    
    # Page dimensions (in pixels)
    if page.dimensions:
        print(f"Width: {page.dimensions.width}px")
        print(f"Height: {page.dimensions.height}px")

Markdown Output​

Since v1.109.0

Set outputContentFormat to get Azure's Markdown output instead of plain text. It takes text or markdown, and the value is case-insensitive. The snake_case spelling output_content_format works the same way. If you leave it out, the response is the same text output as before.

Markdown comes from Azure's prebuilt-layout model. Azure ignores the setting on other models and returns text.

In the default LiteLLM response format, content is Azure's Markdown, and each pages[].markdown is that page's slice of content. Joining the pages in order gives you content back.

When you ask for Markdown in the default format, the response also carries "stringIndexType": "unicodeCodePoint". This means the span offsets in tables and keyValuePairs count Unicode code points, which is what Python string indexing uses, instead of Azure's default text elements.

With "req_format": "native", the proxy returns Azure's raw analyze body. Its analyzeResult.contentFormat reads markdown, and its offsets keep Azure's default unit.

Any other value returns a 400 error with invalid OCR outputContentFormat, expected text or markdown. With drop_params set in the request, on the deployment's litellm_params, or as litellm.drop_params = True, the value is dropped instead and you get text output.

Markdown Output - SDK
import litellm
import os

os.environ["AZURE_DOCUMENT_INTELLIGENCE_API_KEY"] = "your-api-key"
os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] = "https://your-resource.cognitiveservices.azure.com"

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    },
    outputContentFormat="markdown",
)

print(response.content)

for page in response.pages:
    print(f"Page {page.index}:")
    print(page.markdown)

Async Support​

Async Usage
import litellm
import asyncio

async def process_document():
    response = await litellm.aocr(
        model="azure_ai/doc-intelligence/prebuilt-layout",
        document={
            "type": "document_url",
            "document_url": "https://example.com/document.pdf"
        }
    )
    return response

# Run async function
response = asyncio.run(process_document())

Cost Tracking​

LiteLLM automatically tracks costs for Azure Document Intelligence OCR:

ModelCost per 1,000 Pages
prebuilt-read$1.50
prebuilt-layout$10.00
prebuilt-document$10.00
View Cost
response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={"type": "document_url", "document_url": "https://..."}
)

# Access cost information
print(f"Cost: ${response._hidden_params.get('response_cost', 0)}")

Additional Resources​

LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails, built for production.
Learn more →