Evaluate agents on your own tasks
Compare agents on your tasks
Typed tasks, harness and model targets, and a report you can inspect.
from pathlib import Path
from tempfile import TemporaryDirectory
import litellm
from litellm import EvalTask, EvalTarget, Harness, PredicateCheck, sandbox
harness = Harness.TOOL_LOOP
model = "litellm_proxy/model-a"
prompt = "Classify: I was charged twice. Reply only with billing, technical, or sales."
# Run the agent.
with TemporaryDirectory() as workspace:
result = litellm.agent(
harness, prompt, model=model, sandbox=sandbox.local(workspace),
)
print(result.text)
# Evaluate the same setup and a second model.
task = EvalTask(
id="billing-ticket",
prompt=prompt,
check=PredicateCheck(lambda result: result.text.strip().lower() == "billing"),
)
targets = [
EvalTarget(harness, model, label="model-a"),
EvalTarget(harness, "litellm_proxy/model-b", label="model-b"),
]
report = litellm.agent_eval(
tasks=[task], targets=targets, trials=3, max_concurrency=2,
sandbox=sandbox.local,
)
print(report.to_markdown())
Path("scorecard.json").write_text(report.to_json(), encoding="utf-8")
$ python evaluate.py
Target Passed Rate Errors model-a 3/3 100.0% 0 model-b 0/3 0.0% 0 Selected run Task billing-ticket Target model-b Trial 1 Result fail Detail predicate failed
scorecard.jsonRecorded SDK example with mocked answers. No live model calls.
Run an agent on a support ticket, then evaluate the same setup across repeated attempts. With litellm.agent_eval(), you define typed tasks and success checks, compare harness and model targets, and read pass rate, cost and duration in one scorecard.
LiteLLM's main branch includes the evaluation implementation. Published PyPI releases do not include this API yet. The examples below show the upcoming SDK interface.
Connect to your AI Gateway
Use a LiteLLM AI Gateway virtual key with access to two model groups. The examples call them model-a and model-b. Replace those names with your groups.
In Bash or Zsh, set the gateway root URL without /v1. Run read, then enter your virtual key. The terminal hides the input.
export LITELLM_PROXY_API_BASE=https://litellm.example.com
export LITELLM_PROXY_API_KEY
read -rs LITELLM_PROXY_API_KEY
Skip read if your secret manager has set the key. The first comparison uses Tool Loop, which needs no CLI runtime.
Choose an execution backend for each attempt. The script below passes sandbox.local, which runs with your user's permissions. Use local execution for trusted tasks. The sandbox guide covers container factories and host-side Python tools.

