Skip to main content

Lens Profiles: Turn Production Traffic Into Task-Level Model Evals

Understand the work. Find the cost drivers. Evaluate the models that serve it.

Your coding agent fixes tests, implements features, and compares designs. Each task puts different demands on the model. Knowing which tasks dominate your traffic and spend helps you decide where a model change would have the greatest impact.

The revised Lens Profiles discovers those tasks automatically from production traffic. It connects each category to gateway-recorded costs, shows how model deployments compare, and takes you directly to the runs behind the numbers. You get a workload-specific starting point for optimization and model evals.

Discover what your agents actually do​

Lens samples an agent's runs, extracts the intent of each request, and groups similar work into named task categories. Requests about a failing login test and a flaky retry test can become Debug failing tests, even when they refer to different parts of your codebase.

TASK DISCOVERY

From agent traces to reusable task categories

Discover the intent shared by requests with different wording.

AGENT RUNSGROUP BY INTENTDISCOVERED CATEGORIESLogin integration testAdd CSV exportCompare caching designsGenerate a JSON schemaDebug failing testsDiagnose and verify a fixImplement a featureAdd requested behaviorCompare optionsWeigh libraries and designsGenerate structured configProduce configuration or schemasSample requestsDistill intent → embed → clusterReview, then activate classification
Lens discovers categories from request intent, then classifies new runs against the definitions you activate.

You don't need to label every run or design a taxonomy from scratch. Review the discovered categories and activate classification for new runs, with optional historical backfill. As your agent takes on new work, rediscover categories to keep the view relevant.

See which tasks dominate your traffic​

The similarity map groups sampled runs by task, so you can explore related requests and inspect the work within each category. The legend puts those clusters in context: it shows each category's total run count alongside the number plotted on the map.

The map plots 300 sampled runs; the legend shows total runs for each task. Illustrative demo data. Select to view the full dashboard.

The screenshot shows 300 sampled runs from a workload of 24,000 runs. The debugging cluster contains 96 sampled runs; its legend entry reports 7,680 total debugging runs, or 32% of the workload. Feature implementation accounts for another 5,760 runs. Those totals help you prioritize improvements and choose which tasks deserve substantial representation in your evals.

Find the tasks that drive spend​

Lens attributes costs from gateway spend records to the tasks behind your agent's traces. You can rank profiles by spend, compare that with their traffic share, and open the supporting runs to understand what makes a task expensive.

Spend selected: debugging accounts for $4,285.44 across 7,680 runs. Illustrative demo data. Select to view the full dashboard.

Prioritize the work with the largest opportunity. With Spend selected, debugging leads the breakdown at $4,285.44 across 7,680 runs. That is approximately 54% of the $7,970.40 total shown above, despite representing 32% of runs. Inspect those model calls and test whether another deployment can handle representative cases at a lower cost.

You can move from a budget question to the specific work behind it, with task-level costs drawn from the same gateway records used to track model spend.

Compare models on the tasks that matter​

The Avg cost view below pairs each task's average request cost with the number of observed requests for each deployment. Lens also provides Requests and p95 latency views, so you can compare traffic volume, cost, and responsiveness at the task level.

Avg cost selected: compare request costs and observed request counts by task. Illustrative demo data. Select to view the full dashboard.

In the debugging row, the dashboard displays an average request cost of $0.01 on haiku-production and $0.05 on sonnet-production, alongside 34,560 and 80,640 observed requests. Use these task-specific differences to choose which model comparisons are worth running, then evaluate representative cases before changing deployments.

Open a profile to inspect its model breakdown and recent runs. Investigate these runs prepares an investigation scoped to the selected profile, so you can examine the behavior behind a costly or slow workload.

Build evals around real customer work​

Discovered categories give your evals a structure; their underlying traces give you realistic cases. Select examples from frequent or expensive tasks, define what success means for each, and compare candidate models on work your users actually ask for.

FROM TRACE EVIDENCE TO EVAL CASES

Give every comparison a production context

Choose representative runs and define what a successful result means.

SOURCE RUNSDebug failing testsInspect the work behind the category.Read the failureInspect the test outputRevise the codeFollow the agent’s changesRun the checksInspect the reported resulttrace 01trace 07trace 13EVALUATION BRIEFDefine success for this taskSelectRepresentative failing-test casesAssessDoes the fix pass the relevant tests?CompareQuality alongside cost and latencyFocus comparisons on high-impact tasks.
Use representative Lens traces as cases in your evaluation workflow, with success criteria specific to the task.

For debugging, that might mean checking whether a proposed fix passes the relevant tests. For structured configuration, it might mean validation against a schema. Combine those quality checks in your evaluation workflow with the task-level cost and latency Lens exposes to build a capability view specific to your workloads.

Use production traffic to prioritize your next eval. Focus comparisons on the tasks where better model selection could have the greatest effect on cost, responsiveness, or the user experience.

Get started​

Open Profiles → Task discovery, choose your discovery, embedding, and decision models, and select a period with at least 20 eligible runs. Discover the categories, review their descriptions, and start classifying new runs.

Start with your highest-spend profile. Compare its deployments, inspect representative runs, and choose the next model evaluation based on the opportunity you find.

Put your agent traffic to work

Follow the Lens setup guide and the Profiles reference to get started.