Closing the Loop: A Holistic Framework for Evals in AI-Powered Products

​Everyone by now, knows evals are important. But in this fast paced age of AI, when things are continually improving and breaking things, there isn't a set framework on how to build evals. I will share my take on it. I’ll walk through the full lifecycle of evaluations—from where evals live in the stack, the data-flywheel, getting feedback from domain experts, when to build internal tools vs off-the-shelf, doing error analysis, auto-eval(human aligned LLMs-as-judge) and tracking over time, to actually improve your product.

​Speaker: Dhaval Sing
x: https: //x.com/Dhavalsingh7
LinkedIn: https: //www.linkedin.com/in/dhaval-singh-502b79161/

​Current: Founding Engineer @ Seezo.io
Prev: SDE-III @Anarock,
M.S.C in Physics + BE Electronics & Instrumentation from Bits Pilani Goa Campus