Skip to main content

Computer Use

Computer use allows models to interact with computer interfaces by taking screenshots and performing actions like clicking, typing, and scrolling. This enables AI models to autonomously operate desktop environments.

Supported Providers:

  • Anthropic API (anthropic/)
  • Bedrock (Anthropic) (bedrock/)
  • Vertex AI (Anthropic) (vertex_ai/)

Supported Tool Types:

  • computer - Computer interaction tool with display parameters
  • bash - Bash shell tool
  • text_editor - Text editor tool
  • web_search - Web search tool

LiteLLM will standardize the computer use tools across all supported providers.

Quick Start​

import os 
from litellm import completion

os.environ["ANTHROPIC_API_KEY"] = "your-api-key"

# Computer use tool
tools = [
    {
        "type": "computer_20241022",
        "name": "computer",
        "display_height_px": 768,
        "display_width_px": 1024,
        "display_number": 0,
    }
]

messages = [
    {
        "role": "user", 
        "content": [
            {
                "type": "text",
                "text": "Take a screenshot and tell me what you see"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg=="
                }
            }
        ]
    }
]

response = completion(
    model="anthropic/claude-3-5-sonnet-latest",
    messages=messages,
    tools=tools,
)

print(response)

Checking if a model supports computer use​

Use supports_computer_use(model="") from litellm.utils -> returns True if model supports computer use and False if not. It is not exported on the top-level litellm package, so litellm.supports_computer_use raises AttributeError

from litellm.utils import supports_computer_use

assert supports_computer_use(model="anthropic/claude-sonnet-5") == True
assert supports_computer_use(model="anthropic/claude-opus-5") == True
assert supports_computer_use(model="bedrock/us.anthropic.claude-sonnet-5") == True
assert supports_computer_use(model="vertex_ai/claude-sonnet-5") == True
assert supports_computer_use(model="openai/gpt-5.6-terra") == False

Different Tool Types​

Computer use supports several different tool types for various interaction modes:

The computer_20241022 tool provides direct screen interaction capabilities.

import os 
from litellm import completion

os.environ["ANTHROPIC_API_KEY"] = "your-api-key"

tools = [
    {
        "type": "computer_20241022", 
        "name": "computer",
        "display_height_px": 768,
        "display_width_px": 1024,
        "display_number": 0,
    }
]

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text", 
                "text": "Click on the search button in the screenshot"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg=="
                }
            }
        ]
    }
]

response = completion(
    model="anthropic/claude-3-5-sonnet-latest",
    messages=messages,
    tools=tools,
)

print(response)

Advanced Usage with Multiple Tools​

You can combine different computer use tools in a single request:

import os 
from litellm import completion

os.environ["ANTHROPIC_API_KEY"] = "your-api-key"

tools = [
    {
        "type": "computer_20241022",
        "name": "computer", 
        "display_height_px": 768,
        "display_width_px": 1024,
        "display_number": 0,
    },
    {
        "type": "bash_20241022",
        "name": "bash"
    },
    {
        "type": "text_editor_20250124", 
        "name": "str_replace_editor"
    }
]

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Take a screenshot, then create a file describing what you see, and finally use bash to show the file contents"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg=="
                    }
                }
            ]
        }
    ]
    
response = completion(
    model="anthropic/claude-3-5-sonnet-latest",
            messages=messages,
            tools=tools,
)

print(response)

Spec​

Computer Tool (computer_20241022)​

{
  "type": "computer_20241022",
  "name": "computer",
  "display_height_px": 768,    // Required: Screen height in pixels  
  "display_width_px": 1024,    // Required: Screen width in pixels
  "display_number": 0          // Optional: Display number (default: 0)
}

Bash Tool (bash_20241022)​

{
  "type": "bash_20241022", 
  "name": "bash"              // Required: Tool name
}

Text Editor Tool (text_editor_20250124)​

{
  "type": "text_editor_20250124",
  "name": "str_replace_editor"  // Required: Tool name
}

Web Search Tool (web_search_20250305)​

{
  "type": "web_search_20250305",
  "name": "web_search"         // Required: Tool name
}
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails, built for production.
Learn more →