Skip to main content

16 posts tagged with "agents"

View All Tags

Evaluate agents on your own tasks

Ishaan Jaffer
CTO, LiteLLM
LiteLLMPYTHON SDK / AGENT EVALUATIONS

Compare agents on your tasks

Typed tasks, harness and model targets, and a report you can inspect.

evaluate.py
from pathlib import Path
from tempfile import TemporaryDirectory
import litellm
from litellm import EvalTask, EvalTarget, Harness, PredicateCheck, sandbox

harness = Harness.TOOL_LOOP
model = "litellm_proxy/model-a"
prompt = "Classify: I was charged twice. Reply only with billing, technical, or sales."

# Run the agent.
with TemporaryDirectory() as workspace:
result = litellm.agent(
harness, prompt, model=model, sandbox=sandbox.local(workspace),
)
print(result.text)

# Evaluate the same setup and a second model.
task = EvalTask(
id="billing-ticket",
prompt=prompt,
check=PredicateCheck(lambda result: result.text.strip().lower() == "billing"),
)
targets = [
EvalTarget(harness, model, label="model-a"),
EvalTarget(harness, "litellm_proxy/model-b", label="model-b"),
]

report = litellm.agent_eval(
tasks=[task], targets=targets, trials=3, max_concurrency=2,
sandbox=sandbox.local,
)
print(report.to_markdown())
Path("scorecard.json").write_text(report.to_json(), encoding="utf-8")
Terminalselected columns

$ python evaluate.py

Target    Passed   Rate     Errors
model-a   3/3      100.0%   0
model-b   0/3      0.0%     0

Selected run
Task     billing-ticket
Target   model-b
Trial    1
Result   fail
Detail   predicate failed
JSON exportscorecard.json

Recorded SDK example with mocked answers. No live model calls.

Run your agent, then evaluate the same setup and compare a second model.Set up the example

Run an agent on a support ticket, then evaluate the same setup across repeated attempts. With litellm.agent_eval(), you define typed tasks and success checks, compare harness and model targets, and read pass rate, cost and duration in one scorecard.

Preview

LiteLLM's main branch includes the evaluation implementation. Published PyPI releases do not include this API yet. The examples below show the upcoming SDK interface.

Connect to your AI Gateway​

Use a LiteLLM AI Gateway virtual key with access to two model groups. The examples call them model-a and model-b. Replace those names with your groups.

In Bash or Zsh, set the gateway root URL without /v1. Run read, then enter your virtual key. The terminal hides the input.

export LITELLM_PROXY_API_BASE=https://litellm.example.com
export LITELLM_PROXY_API_KEY
read -rs LITELLM_PROXY_API_KEY

Skip read if your secret manager has set the key. The first comparison uses Tool Loop, which needs no CLI runtime.

Choose an execution backend for each attempt. The script below passes sandbox.local, which runs with your user's permissions. Use local execution for trusted tasks. The sandbox guide covers container factories and host-side Python tools.

Run an agent, then evaluate the same task​

Start with a support ticket. Run Tool Loop once to inspect its answer, then define a success check and compare two model targets. Save this complete script as evaluate_agents.py:

evaluate_agents.py
from pathlib import Path
from tempfile import TemporaryDirectory

import litellm
from litellm import EvalTarget, EvalTask, Harness, PredicateCheck, sandbox


harness = Harness.TOOL_LOOP
model = "litellm_proxy/model-a"
prompt = (
"Classify this ticket as billing, technical, or sales. "
"Reply with only the category.\n"
"Ticket: I was charged twice for my subscription."
)

with TemporaryDirectory() as workspace:
result = litellm.agent(
harness,
prompt,
sandbox=sandbox.local(workspace),
model=model,
)
print(result.text)

task = EvalTask(
id="billing-ticket",
prompt=prompt,
check=PredicateCheck(
lambda result: result.text.strip().lower() == "billing"
),
)

targets = [
EvalTarget(harness, model, label="model-a"),
EvalTarget(harness, "litellm_proxy/model-b", label="model-b"),
]

report = litellm.agent_eval(
tasks=[task],
targets=targets,
sandbox=sandbox.local,
trials=3,
max_concurrency=2,
)

print(report.to_markdown())
Path("scorecard.json").write_text(report.to_json(), encoding="utf-8")

litellm.agent() runs the agent and returns a Result. TemporaryDirectory creates its local workspace. The task reuses the prompt, and the first evaluation target reuses the same harness and model. An EvalTask holds the prompt and the rule for success; an EvalTarget selects the harness and model.

agent_eval() runs each target again in a fresh workspace. It takes tasks and targets; you do not pass the earlier Result to it.

Run the script from the directory where you saved it:

python evaluate_agents.py

One task, two targets and three trials produce six evaluation runs. The initial agent call is separate from those six runs and their scorecard. Each target has three attempts. The script also writes scorecard.json for your own analysis. Add representative tickets and their expected outcomes before choosing a production target.

Read the failures before comparing cost​

A run passes when the agent stops with done and its success check passes. A failed success check or turn limit produces fail. Setup failures, runtime errors, timeouts and grading errors produce error.

Inspect unsuccessful runs after the evaluation call:

from litellm import EvalStatus

for run in report.runs:
if run.status is not EvalStatus.PASS:
print(run.task_id, run.target, run.trial, run.detail)

for summary in report.summaries:
print(summary.target, summary.pass_rate, summary.cost)
print("Complete metrics:", summary.metrics_complete)

Pass rate includes error rows in the denominator. An unavailable runtime or failed judge request therefore remains visible in the comparison.

Compare cost alongside pass rate, token usage, model calls and tool calls. Check metrics_complete before treating a total as complete. The report separates agent spend from judge spend. Mean duration includes workspace preparation, setup, execution, grading and cleanup; concurrent runs can overlap.

Compare harnesses with the same task​

To compare Claude Code and Codex in the evaluation, install their runtimes and replace targets in the script:

targets = [
EvalTarget(Harness.CLAUDE_CODE, "litellm_proxy/model-a", label="claude-code"),
EvalTarget(Harness.CODEX, "litellm_proxy/model-b", label="codex"),
]

The initial agent call still uses Tool Loop. The evaluation compares the two new targets. Use model groups supported by each harness. This changes both the harness and model; to measure one change, keep the other fixed where the harnesses support the same model.

The SDK runs each task against each target and repeats the pair trials times. Give each task a unique ID and each target a unique label.

Choose a check for the outcome you need​

Use the same task interface for answers, writing, research and file changes:

CheckUse it to verify
PredicateCheckA final answer, tool calls or another part of the typed Result
JudgeCheckAn answer against a rubric, using an explicit judge model
CommandCheckFiles or project behavior with a command whose zero exit code passes

For a writing task, replace task in the script and allow your key to call the judge model group:

from litellm import JudgeCheck

task = EvalTask(
id="support-reply",
prompt="Reply to a customer who was charged twice. Do not promise a refund.",
check=JudgeCheck(
model="litellm_proxy/judge",
rubric=(
"Acknowledge the duplicate charge, give a clear next step, "
"and do not guarantee a refund."
),
),
)

Choose a judge model that supports JSON output and budget for its calls. It receives the task prompt, answer and reported file changes with your rubric. A negative verdict produces fail; a failed request or invalid verdict produces error.

For tasks that need files, add workspace to the task. Each attempt gets a copy. The runner rejects symbolic links before creating the evaluation sandbox, including internal links, broken links and a linked workspace root. Local execution uses your user's permissions. Read the workspace and sandbox guide before running file tasks.

Key takeaways​

  • Define the task and its success check once, then choose your targets.
  • Use repeated trials and inspect the unsuccessful runs.
  • Compare pass rate with cost and time, checking for incomplete metrics.
  • Export the scorecard to JSON to keep a record of the comparison.

Frequently asked questions​

Do tasks need a repository?​

No. The support-routing and writing tasks above use prompts and checks. Add a workspace when your task needs starting files.

Do I need a database?​

The evaluation runner returns its scorecard in memory and can export it as JSON. It adds evaluation identifiers to model-request metadata so you can correlate calls in existing gateway logs. Persisted gateway spend logs use the gateway's existing database setup.

Can I run evaluations in an async application?​

Use await litellm.aagent(...) for the initial run and await litellm.aagent_eval(...) for the evaluation inside your async function or notebook. Keep the same tasks, targets and options.

Can I use this with LiteLLM OSS?​

The evaluation API is under development in the open-source SDK. The release version will appear in this guide when it ships. LiteLLM Enterprise provides gateway controls and support for production deployments.

Run your own comparison​

Pick one task you can grade, run it against two targets, and inspect the outcomes. Expand the task set as you find failures you want future changes to catch. The complete evaluation guide covers setup commands, sandboxes and execution limits.

Evaluation API reference, agent harness quickstart, and connect harnesses to your AI Gateway.

Share your team's skills with LiteLLM

Tin Lo
AI Product Engineer, LiteLLM

Give your team one place to find and share useful agent skills. With LiteLLM Skills Gateway, teammates can reuse a skill's instructions and supporting files in their own AI agents.

To help you manage those skills, we've added:

  • ZIP uploads
  • Admin review
  • Version history
  • Download tracking

Available in the LiteLLM 1.110.x release. Upgrade to v1.110.0 or later to use these features.

How to build an agent with grounding on LiteLLM

Ishaan Jaffer
CTO, LiteLLM
A DEVELOPER GUIDE BY LITELLMPYTHON SDK / OCTOBER 2026

BUILD AN AGENT WITHGROUNDING

Grounding checks whether an answer is supported
by the documents you give it.

Start building
01 / YOUR EVIDENCE$500Home office equipment policy

Employees receive a one-time home office equipment allowance of $500. Submit receipts within 30 days of purchase.

home-office
native structured findings
WORKPLACE ASSISTANTRecorded fixture
How much can I spend on my home office?
YOUR AGENT

You receive a one-time home office allowance of $500. Submit your receipts within 30 days of purchase.

Grounded in your documents

The policy supports the allowance and receipt deadline.

source_ids: ["home-office"]
true
RECORDED FIXTURE / NO MODEL CALLS

Build an agent that answers questions from your documents, then checks each answer against the evidence before showing it in chat.

We will connect to a LiteLLM gateway, load a policy, generate and check an answer, then display the grounding result. The Python snippets form one async request handler. Start with one document; connect your retriever once you can inspect the complete flow.

Source preview

This guide uses the unreleased SDK helper #49469, native endpoint response #49460, and Bedrock assessment setting #49470, for issue #49217. Use that implementation for both your SDK and gateway until release. The template at the end pins the complete revision.

Step 1. Connect to your gateway​

Configure your SDK once. Use your gateway URL and a virtual key with access to the agent model and grounding guardrail:

agent.py / connection
import osimport litellm
litellm.api_base = os.environ["LITELLM_API_BASE"]litellm.api_key = os.environ["LITELLM_API_KEY"]

We use a model named agent on the gateway. The openai/ prefix below selects the OpenAI-compatible transport; the gateway can route agent to a model from another provider.

For the grounding check, use Amazon Bedrock Guardrails through LiteLLM. AWS maintains the assessment service, and you can use it with answers from other model providers. Add this to your gateway configuration:

config.yaml / grounding
guardrails:  - guardrail_name: answer-grounding    litellm_params:      guardrail: bedrock      mode: post_call      default_on: false      guardrailIdentifier: os.environ/BEDROCK_GUARDRAIL_ID      guardrailVersion: os.environ/BEDROCK_GUARDRAIL_VERSION      aws_region_name: os.environ/AWS_REGION_NAME      contextual_grounding_from_messages: true      outputScope: FULL

Create an AWS guardrail with contextual grounding enabled. For conversational answers, choose Standard tier and its required cross-Region inference profile. Set the AWS filter action to NONE so your application can receive an unsupported finding and decide what to display. Set outputScope: FULL in LiteLLM to receive supported assessments too. See AWS setup, filter actions, and full assessments.

default_on: false leaves this guardrail available for the explicit SDK call in step 3. You control when the check runs.

Step 2. Retrieve the evidence​

Start with a home office policy. Save this text as home-office.md:

home-office.md
Employees receive a one-time home office equipment allowance of $500.Submit receipts within 30 days of purchase.

Load the policy and keep it with the question:

agent.py / evidence
from pathlib import Path
question = "How much can I spend on my home office?"evidence = Path("home-office.md").read_text()
source_messages = [    {"role": "system", "content": evidence},    {"role": "user", "content": question},]

Your agent and the grounding service must receive the same evidence. With contextual_grounding_from_messages: true, LiteLLM sends the system text to Bedrock as the grounding source and the user question as the query.

For your own documents, replace the file read with your retriever. Pass only documents the current user can access. Keep previous assistant answers out of the reference evidence.

Step 3. Generate and check the answer​

Ask your model to answer from that evidence:

agent.py / answer
completion = await litellm.acompletion(    model="openai/agent",    messages=[        {"role": "system", "content": "Answer using only this policy:\n" + evidence},        {"role": "user", "content": question},    ],)answer = completion.choices[0].message.content

Then call the grounding guardrail with the answer and source messages:

agent.py / grounding
native_result = await litellm.guardrails.apply(    "answer-grounding",    response=answer,    messages=source_messages,)

"answer-grounding" selects the configured guardrail. response=answer tells the SDK to check generated output; the SDK infers the direction. You do not need to provide an input type or a response-format flag.

The call returns the guardrail's native structured findings, after LiteLLM's existing reporting sanitization. For Bedrock, the grounding assessment looks like this illustrative excerpt:

Bedrock assessment
{  "contextualGroundingPolicy": {    "filters": [{      "type": "GROUNDING",      "score": 0.98,      "threshold": 0.7,      "detected": false    }]  }}

Bedrock compares the answer with the supplied evidence. It does not verify whether the source document itself is correct. Evaluate the check on supported, contradicted, and incomplete answers from your use case before using it to enforce policy.

Step 4. Show the result in chat​

Read the GROUNDING assessments. Require an assessment before showing a positive badge, and check the score against its configured threshold:

agent.py / verdict
results = native_result if isinstance(native_result, list) else [native_result]checks_by_result = [    [check     for assessment in result.get("assessments", [])     for check in assessment.get("contextualGroundingPolicy", {}).get("filters", [])     if check["type"] == "GROUNDING"]    for result in results]
is_grounded = Noneif checks_by_result and all(checks_by_result):    is_grounded = all(        check["score"] >= check["threshold"] and not check.get("detected", False)        for checks in checks_by_result        for check in checks    )

Check all returned chunks. Do not infer success from Bedrock's overall action: a report-only guardrail can return NONE with a negative grounding finding.

Return the answer and verdict from your chat backend. In your chat component, show the result beside the answer:

GroundingBadge.jsx
export function GroundingBadge({grounded}) {  if (grounded == null) {    return <span>Grounding check unavailable</span>;  }
  return (    <span data-status={grounded ? "supported" : "unsupported"}>      {grounded ? "Supported by your documents" : "Not supported"}    </span>  );}
01 / SUPPORTEDGrounded

Show the source and the check's explanation.

is_grounded: true
02 / UNSUPPORTEDNot supported

Explain the gap or ask the agent to retry.

is_grounded: false
03 / CHECK FAILEDUnavailable

Keep this separate from a completed check.

no valid verdict

A timeout, missing assessment, or invalid result should produce Check unavailable, with no green badge. The template below includes typed response validation and those failure states.

Bedrock evaluates the supplied documents together. Label the references Documents checked; its response does not attribute an answer to an individual document. For an unsupported answer, you can show the gap, retrieve more evidence, or withhold the answer. Wait for the check before rendering if your product gates delivery on the result.

Deploy the complete chat template​

You have the core flow: evidence, answer, grounding call, and a verdict beside the response. Use the template to get the complete chat UI, a tool-calling document agent, and validated result handling in a separate repository.

Run the complete app
git clone https://github.com/BerriAI/litellm-grounded-agent-starter.gitcd litellm-grounded-agent-starteruv syncuv run grounded-agent-demo

Open localhost:8502. Choose Home office allowance for a supported answer, or Check an unsupported answer to see a negative verdict. This local demo uses fixed model and judge responses, labelled in the UI, so you can exercise the SDK, gateway, and chat without provider credentials.

For a live deployment, follow the template's Bedrock setup, replace the included documents with your retriever, and connect your models. The launcher supports local development; deploy the gateway and chat app with your platform's normal service configuration. Add sign-in and document authorization before sharing the app with coworkers. Live AWS execution remains unverified in this example; the template tests its adapter with representative responses.

Key takeaways​

Give your agent and grounding guardrail the same evidence. Call litellm.guardrails.apply after the answer exists, read the provider's assessment, and keep supported, unsupported, and failed checks distinct in your UI.

Frequently asked questions​

Can I use an existing agent and guardrail?​

Yes. Keep your framework and evaluation logic. Replace the answer-generation snippet with your agent's final answer, call litellm.guardrails.apply, and read your guardrail's result schema.

Does LiteLLM generate a second answer for the check?​

The SDK invokes your configured guardrail. Bedrock evaluates the supplied answer. A custom LLM judge makes an evaluation request; it does not regenerate the agent's answer to obtain the verdict.

Can I use this with LiteLLM OSS?​

The SDK changes and template are open source. Use the linked implementation revision for both the SDK and AI Gateway until release. LiteLLM Enterprise adds controls and support for production deployments.

Call a guardrail directly, configure Bedrock grounding, and write a custom guardrail.

Workshop: Build a self-hosted cloud agent.

Ishaan Jaffer
CTO, LiteLLM

Workshop: Build a
self-hosted cloud agent.

Build and deploy with the LiteLLM team, on infrastructure your team controls.

LITELLM WORKSHOP · OCTOBER 19–30

Build a self-hosted cloud agent.

Tell us about your team and what you want to achieve. We’ll follow up to arrange a 30-minute intake. Remote, over two weeks.

Workshop signup details

* Required. We’ll use your details to arrange your workshop intake. Submitting this form does not reserve a time slot.

Remote 2 weeks October 19–30, 2026

Work with your preferred agent

Claude Code
Codex
Hermes
OpenClaw
A Slack request reaches an agent harness and sandbox in your cloud, then produces a pull request.

Open Sourcing Moyai: Self-Hosted Cloud Coding Agent

Ishaan Jaffer
CTO, LiteLLM
Tin Lo
AI Product Engineer, LiteLLM
Moe Khalil
AI Product Engineer, LiteLLM

MoyaiOpen source cloud agent

October 7, 2026
HermesHermes
Claude CodeClaude Code
CodexCodex
OpenCodeOpenCode
Deep AgentsDeep Agents
OpenAIOpenAI
AnthropicAnthropic
FireworksFireworks
GoogleGoogle
xAIxAI
MistralMistral
DeepSeekDeepSeek
BedrockBedrock

Works with Claude Code and CodexSelf-hosted · 100+ providers through LiteLLM

  1. [1]The problem
  2. [2]Our cost estimate
  3. [3]Why we're open sourcing it
  4. [4]A cloud agent that keeps working
  5. [5]Choose your agent
  6. [6]Choose your models
  7. [7]Know what you spend
  8. [8]Set up your first task
Published:
Ishaan JafferCTO, LiteLLMTin LoAI Product Engineer, LiteLLMMoe KhalilAI Product Engineer, LiteLLM

Today we're open sourcing Moyai, the cloud coding agent our team runs at LiteLLM. Give it a task in Slack or the browser, then come back to a pull request with code and test results to review. You choose the agent and compatible model, run it on your infrastructure, and track the cost through LiteLLM.

Set up your first cloud task, or see the cost comparison and a real bug fix below.

How we built our own internal Devin in 2 days

Tin Lo
AI Product Engineer, LiteLLM

How we built our own internal Devin in 2 days

Our team's cloud coding agent, built with Render and Temporal.

Our internal Devin, running in the cloud.
Published:
Tin LoAI Product Engineer, LiteLLM

At BerriAI, we built Moyai Devin, an internal engineering agent that runs in the cloud. Teammates can give it a task in Slack, follow its progress in a web app, and ask it to prepare a pull request.

Introducing LiteAgents

Moe Khalil
AI Product Engineer, LiteLLM
Agents in the LiteLLM SDK

For new projects, use litellm.agent(), included in the LiteLLM Python SDK. This launch post describes the separate LiteAgents package. Its source and reference remain in the LiteAgents repository.

LiteAgents: switch harnesses, keep your agent. A ProfileOptions example changes deepagents to claude-sdk while keeping the model, tools, and MCP connections.

You've built an agent with your own tools, prompts, and model configuration. Now you want to try a different harness on the same task.

LiteAgents lets you switch agent harnesses without rewriting your application. Its interface is modeled after the Claude Agent SDK, including the query() pattern and typed messages. Choose Deep Agents, Pydantic AI, Claude Agent SDK, Codex, or OpenCode while keeping your tools, MCP connections, model configuration, and client code. Each selected harness runs its own agent loop.

LiteLLM gives you a common interface to models. LiteAgents brings that approach to agent harnesses, with native controls and optional durability through Temporal.

Introducing LiteAdmin MCP

Tin Lo
AI Product Engineer, LiteLLM
Introducing LiteAdmin MCP: your AI toolkit for gateway management. LiteAdmin for Slack is built on LiteAdmin MCP.Introducing LiteAdmin MCP: your AI toolkit for gateway management. LiteAdmin for Slack is built on LiteAdmin MCP.

An engineer asks for an API key for a new project. You need to choose its models, set a budget, and assign it to a team. As usage grows, you need to check spending and adjust those limits.

LiteAdmin MCP lets your agent handle these tasks through your gateway's management API. Connect it to an MCP client or a custom agent. We built LiteAdmin, our Slack admin agent, on the same connector.

Reduce agent context with TypeSafe Jev and LiteLLM

Yassin Kortam
Senior SWE @ LiteLLM

A bot looks up the weather, then checks a shop's opening hours. The user asks, "What time does the shop close?" The bot still sends the old weather report to the model, even though it no longer helps answer the question.

TypeSafe Jev helps LiteLLM spot tool results that are no longer needed. LiteLLM replaces those results with a short notice before calling the model. This is called compaction, and it can reduce the input tokens used by long conversations.

Secure shared AI agents with identity-aware access and spend controls

Yassin Kortam
Senior SWE @ LiteLLM

Shared agents can preserve individual identity, access, and spend controls.

When a finance agent serves multiple business units, platform teams need a consistent way to identify who initiated each request, apply the right model and tool permissions, and attribute spend. LiteLLM keeps this context available across shared-agent workflows so each business unit can operate under its own access and budget policies.

LiteLLM provides one control plane for this workflow across the Agent Gateway, Model Gateway, and MCP Gateway. Teams can share the same agent infrastructure while keeping access, credentials, spend, and audit data tied to the right caller.

Swap OpenAI Code Interpreter for E2B/OpenSandbox

Krrish Dholakia
CEO, LiteLLM

Swap OpenAI Code Interpreter with E2B

The OpenAI Responses and Chat Completions APIs let you declare a code_interpreter tool and the model runs Python inside an OpenAI-hosted container. That container is opaque, billed by OpenAI, and the code (often customer data) leaves your perimeter. LiteLLM now let's you intercept that tool call and runs it in a sandbox you control. The client request is unchanged.

Available starting LiteLLM v1.91.0.dev1. Check here for releases.

LiteLLM Labs: Announcing Lite-Harness SDK — Unified API for Claude Code, Codex, and Pi AI

Krrish Dholakia
CEO, LiteLLM
Ishaan Jaffer
CTO, LiteLLM

Harnesses are the next frontier of vendor lock-in. LiteLLM was built to swap across model providers easily. However, as the models get saturated, the next area for competition becomes the harnesses and managed agents. To make it easy to go across vendors at the harness layer, we're launching the Lite-Harness SDK. This is a simple TypeScript+Python SDK which allows developers to change harnesses, like they change models.

How we built a background agent to cover 30% of our backlog

Krrish Dholakia
CEO, LiteLLM
Ishaan Jaffer
CTO, LiteLLM
LiteLLM Agent Platform: agent.litellm.ai
info

The platform we built is open source. Check out litellm-agent-platform. The swappable harness layer is lite-harness.

Building the same thing inside your company?

Our goal was to 10x the productivity of our company with agents.

Three weeks ago we began building an agent that could own 30% of our engineering tickets. Here's what we've learnt so far.