Evals 101: From agent failures to your first eval suite
You know your AI agent needs evals. But where do you even start?
What should you test? How many examples are enough? When do you need an LLM judge? And how do you know whether your evals are actually catching the right failures?
Here's an idea: Start with what your agent already gets wrong.
In this free, 60-minute live session, we'll take a real AI agent, dig into its traces, find failures, and turn them into a working first eval suite.
No hypothetical test cases. We'll start with real agent behavior and work backwards to tests.
What we'll cover:
How to spot failures worth testing in your agent's traces
When to use code checks, LLM-as-judge, or human review
How to turn real runs into a small, useful eval dataset
How to write an LLM judge and check whether you can trust it
How to keep your evals relevant as your agent changes
Who it's for: Engineers and PMs building with LLMs or AI agents. Whether you're starting from zero or already have evals you're not confident in, you'll leave knowing what to test next.
🗓 November 3, 2026 · 7 PM IST · 60 mins · Free
🎥 Can't make it live? Register anyway. We'll send you the recording.
Hosted by Antaripa Saha, applied AI consultant.