Build AI Evals with a Microsoft PM Who Shipped to Millions
Learn AI Evals From Microsoft PM who is evaluating for an AI Product with 500M Daily Users
Shipping a feature is easy with AI now, but it's much harder to evaluate what is breaking that nobody has noticed yet
Bring the AI feature you're working on and the part of it you currently have no way to measure. You'll spend the session working out what to measure, how to build a first eval dataset, and what to do with a score once you have one.
You'll work on
Getting a definition of "eval" that holds up Deciding what to measure when there's no correct answer, only better and worse ones
Building a first eval dataset without a labelling team or a budget
Catching the failures users won't report, because they won't notice them
Turning a score into a decision you can actually ship on
By the end of the session, you should leave with one or more of these:
A clear first metric for your AI feature
A dataset plan you can build this week
A way to test whether a prompt change helped
An answer to the question you brought in
Ashish is keeping this question-led, not lecture-style. The last 30 minutes are open, so come with the specific thing you're stuck on rather than a general interest in evals.
This is for engineers, PMs, founders, and early-stage teams building AI products. If you've shipped anything with an LLM in it, you'll be able to follow. No prior evals experience needed.
You must bring a live AI feature, prototype, or product you're currently working on.
If you work in growth or GTM, this one goes fairly deep into the build layer.
Who is your host?
Ashish Muralidharan is an AI PM at Microsoft working on agentic AI for Excel, a product more than 500 million people use, and he builds the evals behind it. When AI is wrong in a chat app you can see it. When it's wrong in a spreadsheet, it just looks like a number, and nobody catches it for a quarter. He's an IIM Calcutta alum and guest lectures on AI strategy and the future of work.
Details
Wednesday, 20 August 7:30 PM to 9:00 PM IST Online Free
About GrowthX
GrowthX is an exclusive community where top founders, growth leaders & operators come to accelerate their careers & companies.
To view full details, click here.