Build an Eval Loop for More Reliable Agents (Evals 101)
Every team building agents runs into this sooner or later. The demo works, the first users are happy, and then someone changes a prompt or upgrades the model. Something quietly breaks three tool calls into a workflow, and nobody catches it until a customer does.
Gut checks are fine while you’re prototyping. They fall apart in production. So does reading traces one by one once you have real traffic. At that point, you need an eval loop: a consistent way to score what your agent does, catch regressions before you deploy, and learn from the traces you’re already collecting.
We’ll start with a short, practical introduction to evals. You’ll learn what scorers do, when to use an LLM judge, when a deterministic check is enough, and how scoring fits into the day-to-day process of building and maintaining an agent.
Then we’ll get into the code. Most of the workshop is live demos, including building an eval loop in Mastra and reusing the same scorers for offline tests and production monitoring. We’ll show how to catch regressions in CI, run experiments over datasets, and find useful patterns in production traces without inspecting them one at a time.
We’ll cover:
Mastra’s built-in scorers for accuracy, hallucination, and tool-call correctness
Fast, deterministic checks that do not require another LLM call
Custom scorers for behavior specific to your agent
Regression tests with runEvals and your existing test framework
Dataset experiments for comparing prompts, models, and code changes
Mocking tools with side effects so experiments stay repeatable
Sampling, tags, and metadata for filtering large volumes of production traces
Scoring live traffic and turning useful traces into new dataset items
We’ll work through real code throughout the session. Bring the eval problems you’re dealing with, too. The engineers who built Mastra’s evals, datasets, and tracing will be there to dig into them with you.
Hosted by
Alex Booker, Developer Experience, Mastra
Yujohn Nattrass, Engineering, Mastra
Recording and code examples will be available to everyone who registers.