AI Evals 101 for PMs and Engineers: From Rubric to CI Gate
Overview
Most AI teams accidentally maintain two definitions of quality. Product defines what success looks like, engineering turns it into evaluation code, and somewhere along the way those definitions drift apart. Even worse, many AI evaluators are never validated against human judgment, making their scores difficult to trust.
In this hands-on session, we'll take a real AI agent from product requirements to production-ready evaluations. You'll learn how PMs and engineers can build a shared evaluation workflow - from writing rubrics and validating AI judges to enforcing quality in CI and keeping evaluations reliable as your product evolves.
What you'll learn
This session is designed for AI engineers and technical product leaders building and shipping LLM applications together. We'll cover:
How to identify real user-facing failures before writing a single evaluation metric.
Turning product requirements into evaluation rubrics that engineering can implement consistently.
Defining clear ownership between PMs and engineers across the evaluation lifecycle.
Validating AI judges against human reviewers so you know when their scores can be trusted.
Running evaluations automatically in CI to prevent quality regressions before deployment.
Evolving your evaluation suite as prompts, models, and user behavior change in production.
What you'll leave with
A clear split of who owns what, from the rubric to the production monitor. A way to check whether your LLM judge actually agrees with a human before you trust its scores. And a working eval-in-CI example you can rebuild on your own agent that week.
Who should attend
AI Engineers, Engineering Managers building AI products, Technical Product Managers, Heads of Product, etc
Whether you're building with LangChain, LlamaIndex, CrewAI, OpenAI, Anthropic, or your own stack, the concepts apply regardless of framework.
About Future AGI
Future AGI is an open-source AI engineering and optimization platform for shipping self-improving AI Agents. Teams use it to simulate agents before they ship, score them against real failure modes, and keep watching them in production so quality doesn't quietly degrade over time. It's self-hostable and OpenTelemetry-native, with tracing that plugs into 35+ frameworks.