Simulated-User Evaluation Infrastructure: Evaluating AI Products with 8.3 Billion Persona Agents
Caltech World Model Club × Caltech C Seminar Series: Simulated-User Evaluation Infrastructure: Evaluating AI Products with 8.3 Billion Persona Agents
We are excited to welcome Haixin Wang, Founding Engineer of MatrAIx, for the next talk in our joint seminar series! MatrAIx is building simulated-user evaluation infrastructure powered by 8.3 billion persona agents. Its research brings together leading AI researchers from Harvard, MIT, Stanford, and Berkeley, with an advisory committee featuring Nobel Laureate Vernon Smith, Microsoft Chief Scientific Officer Eric Horvitz, and MIT CSAIL Director Daniela Rus.
Learn more about MatrAIx: https://matraix.ai/
RSVP if you'd like to learn how world models can simulate human behavior and transform AI product evaluation, or if you're just curious!
Open to everyone in the greater LA area (🍕 free food provided)!
Event Details
Speaker: Haixin Wang, Founding Engineer of MatrAIx
Date: Friday, October 30, 2026
Time: 5:30–6:30 PM
Location: Annenberg 105, Caltech
Abstract
AI products are usually judged on offline benchmarks, but a model that ranks first on a leaderboard can still disappoint the people who actually use it. In this talk, I will introduce MatrAIx, a simulated-user evaluation infrastructure built on a population of 8.3 billion persona agents. Each persona is described with a shared schema of 1,290 attributes covering background, psychology, capability, behavior and lifestyle, and the population combines personas extracted from real-world sources with synthetic generation. I will show how we sample persona cohorts to evaluate four kinds of products (surveys, chatbots, web products and apps) across more than 1,000 tasks in 25+ domains, and how the results show which users a product works for and which it fails. I will also cover how we check that persona agents behave as specified, and where we see simulated users going next, from personalized AI agents to embodied persona AI.