Build AI Agent Evals from Production Traces

​One prompt or model change can quietly break an agent that worked yesterday. Most teams test by trying a few questions by hand, so their users find the regressions first. Live, I turn real agent runs from Langfuse into a test set, score them with code checks and an LLM judge, and fail the build when a change makes the agent worse. Part 6 of 7 in the Agentic AI Systems series.

​This session is hosted on Maven. Register at https://maven.com/p/34dbeb to get the Zoom link.

​What you will leave with

• ​Turn real agent runs into a test set: Pick traces from real use and label what a good answer looks like.

• ​Score answers with checks and an LLM judge: Code checks for facts and format, a judge for the rest, tuned to your labels.

• ​Fail the build when the agent gets worse: Run the evals in CI on every prompt or model change.

​Join this free live lesson to build evals for an AI agent from real runs, then put them in your deploy path in the Deploy AI Agents to Production workshop.