The OpenAI Responses and Chat Completions APIs let you declare a code_interpreter tool and the model runs Python inside an OpenAI-hosted container. That container is opaque, billed by OpenAI, and the code (often customer data) leaves your perimeter. LiteLLM now let's you intercept that tool call and runs it in a sandbox you control. The client request is unchanged.
Available starting LiteLLM v1.91.0.dev1. Check here for releases.
Last updated: June 2026
Agent infrastructure is already separating into three layers: models, harnesses, and runtimes. We believe a fourth layer will emerge: the unified agent control plane. This will allow calling agents living in different agent runtimes, all from 1 place.
The reason is that companies will not run every agent on one runtime. Coding agents may run on Bedrock AgentCore or Claude Managed Agents. Data agents may run inside Elastic, Databricks, or Snowflake. Internal workflow agents may run on custom infrastructure. The control plane emerges because companies want one place where all of these agents can be used, regardless of where they were built or run.
But a registry alone is not enough. Anyone can build a list of agents.
The harder problem is invocation. Agent runtimes expose similar primitives — agents, sessions, events, tools — but they do not expose them through the same APIs. So if you want one place to actually use these agents, not just list them, the control plane has to manage agent runtimes, schedules, memory, and sessions.
This is the same pattern LiteLLM saw with models. Companies did not just need a catalog of models. They needed one interface to call them. The only change, is that the primitive is now the agent session, not the model call.
The important shift is that the gateway is no longer just routing model calls. It is routing agent work.
With LLMs, the stack became:
With agents, we think the stack becomes:
At LiteLLM, we are already seeing our team work across multiple agent runtimes. Some people are building on Claude Managed Agents, others are on N8N or Cursor.
This fragmentation makes it hard for agents built on these platforms to be shareable, and everyone to benefit from the work done so far.
By having the agents live in 1 place, everyone can leverage these agents - even if the PR Babysitter Agent was written in Claude Managed Agents, which not everyone has direct access to.
That is the control plane problem.
This is also why we think the AI Gateway moves up the stack. The gateway starts by managing model calls. But as agents become the dominant use-case for AI, the gateway has to manage agent sessions too.
LiteLLM Agent Platform is our experiment in this direction.
LiteLLM Agent Platform is a Rust-based AI Gateway and Agent Control Plane. The goal is to let teams register, invoke, observe, and govern agents across multiple runtimes.
We are starting with coding agents because the need is obvious. They are long-running, stateful, tool-heavy, and expensive enough to require real infrastructure.
We are already seeing early users resonate with this pattern. Some companies want LAP to act as a central control plane for agents built by different teams on different runtimes. For example, one team might build an agent on Elastic’s runtime to analyze Kibana logs, but the company may want to expose that agent internally through a common gateway.
This is the architecture we believe is coming: models become interchangeable, harnesses become specialized, runtimes become managed, and the gateway becomes the control plane for agent work.
If this matches what you are seeing, we would love feedback on LiteLLM Agent Platform:
https://github.com/LiteLLM-Labs/litellm-agent-platform
No. LAP is an experimental project. The goal is to learn quickly and bring the right pieces into LiteLLM over time.
No. LAP is pre-v0. APIs may change as we work with early users and contributors.
If you want to contribute, file an issue or join our Discord:
Harnesses are the next frontier of vendor lock-in. LiteLLM was built to swap across model providers easily. However, as the models get saturated, the next area for competition becomes the harnesses and managed agents. To make it easy to go across vendors at the harness layer, we're launching the Lite-Harness SDK. This is a simple TypeScript+Python SDK which allows developers to change harnesses, like they change models.
It exposes harnesses in a unified Claude Agents SDK spec. This means that if you wrote your app with the Claude Agents SDK, and want to try another harness (Pi AI, Hermes, Codex, OpenCode), you can do so without rewriting your code.
Today, it supports 3 harnesses - Claude Code, Codex, and Pi AI. Please file an issue here, if you want us to add another harness.
Here's how it works:
TypeScript Example
import { query } from "@lite-harness/sdk";
const prompt = "Fix the failing test";
// Claude Code harness
for await (const message of query({
prompt,
options: { harness: "claude-code", model: "claude-opus-4-8" },
})) {
console.log(message);
}
// Codex harness
for await (const message of query({
prompt,
options: { harness: "codex", model: "gpt-5.5" },
})) {
console.log(message);
}
Python Example
from lite_harness import query, AgentOptions
prompt = "Fix the failing test"
# Claude Code harness
async for message in query(
prompt=prompt,
options=AgentOptions(harness="claude-code", model="claude-opus-4-8"),
):
print(message)
# Codex harness
async for message in query(
prompt=prompt,
options=AgentOptions(harness="codex", model="gpt-5.5"),
):
print(message)
Lite-Harness supports proxy'ing harnesses via LiteLLM AI Gateway. This enables easy model swapping, cost controls and logging.
Point Lite-Harness at your gateway by setting two environment variables:
export LITELLM_API_BASE=https://litellm.your-company.com/v1
export LITELLM_API_KEY=sk-litellm-...
Then call as usual — every underlying model request routes through the gateway:
from lite_harness import query, AgentOptions
prompt = "Fix the failing test"
# Claude Code harness
async for message in query(
prompt=prompt,
options=AgentOptions(harness="claude-code", model="claude-opus-4-8"),
):
print(message)
# Codex harness
async for message in query(
prompt=prompt,
options=AgentOptions(harness="codex", model="gpt-5.5"),
):
print(message)
No. lite-harness works standalone — point it at provider APIs with native keys. AI Gateway integration is opt-in for teams that want central key management, budgets, fallbacks, and a single audit log across every model call.
Yes — that's the point. Each harness keeps its native loop, tool-calling semantics, and prompt format. lite-harness unifies how you invoke them, not how they run internally. Run the same prompt across all three to see which combo lands the task best.
lite-harness is an early, experimental project. This is in public beta. Please join our discord, to help design it to your preference.
Yes. lite-harness is MIT-licensed at github.com/LiteLLM-Labs/lite-harness. LiteLLM Enterprise adds SSO/SCIM, air-gapped deployment, 24/7 SLA, and advanced guardrails on top of the AI Gateway it pairs with.
The platform we built is open source. Check out litellm-agent-platform. The swappable harness layer is lite-harness.
Building the same thing inside your company?
Our goal was to 10x the productivity of our company with agents.
Three weeks ago we began building an agent that could own 30% of our engineering tickets. Here's what we've learnt so far.
Thank you to everyone who joined our May town hall.
We covered security hardening, release versioning changes, new product launches (MCP toolsets, on-behalf-of OAuth), performance wins, and our biggest bet yet — the LiteLLM Agent Platform.
We're introducing the LiteLLM Managed Agents Platform - a simple, self-hosted infrastructure platform for running multiple agents in production.