Vibes Don't Scale: Building Production Systems with AI Evals

​Demos are easy, but production is where LLMs break.

​Evaluations have emerged as the single most critical component of the AI stack as the industry goes beyond just “vibes”. Join us for an evening of deep-dives into the eval systems powering the world’s most advanced AI products.

​Panelists:

• ​Neel Kapse, Engineering Manager (Evaluations), OpenAI

• ​Niko Grupen, Head of Applied Research, Harvey

• ​Sam Crowder, Head of Core Platform, LangChain

• ​Iz Shalom, Head of Product, Cartesia

​Together, we’ll unpack what good actually looks like in practice and how to define safety before you launch. We’ll cover how to bridge the gap between offline testing and online monitoring, and how to identify failure modes that only surface under real-world usage.

​We’ll leave plenty of time for audience Q&A and networking afterward.