AI Builders Meetup: Evaluating Sessions & Scaling with Multi-Agent Systems
Join us for an evening with the AI developer community at GitHub San Francisco. The AI Builders Meetup brings together engineers, researchers, and builders who are shaping the next generation of AI applications.
This event will feature deep-dive sessions on two cutting-edge topics in AI development:
Session-level evaluation and observability for multi-turn LLM interactions
Scaling AI through multi-agent collaboration with live demonstrations
We’ll also spotlight three community-led demos and leave time for open networking. Whether you’re building LLM apps, experimenting with multi-agent frameworks, or simply curious about practical evaluation techniques, you’ll walk away with new insights and connections.
Agenda
5:30 PM – 6:00 PM | Check-in & Networking
6:00 PM – 6:30 PM | Session-Level Evaluations with Arize
Speaker: Sri Chavali, Arize AI
Evaluate AI systems the way users experience them: across entire sessions. Learn how to use the Arize Python SDK to:
Attach session and user IDs to spans with OpenInference instrumentation
Collapse traces into structured session DataFrames for analysis
Run LLM-as-a-judge evaluations (correctness, goal achievement, frustration) on multi-turn interactions
Log results into Arize AX or Phoenix for visualization and iterative improvement
Walk away with a practical framework for evaluating and improving the real-world performance of your AI systems.
6:30 PM – 7:00 PM | MassGen: Scaling AI Through Multi-Agent Collaboration
Speaker: Chi Wang, Founder of Autogen (Now AG2)
Discover how multi-agent systems can outperform single-model approaches by enabling diverse AI models (Claude, Gemini, GPT, Grok) to collaborate in real-time. This session will feature:
Architectures that enable cross-model synergy and iterative refinement
Live demos including creative writing consensus, travel planning, and complex problem-solving
Insights into recursive agent bootstrapping and the future of agent collaboration
7:00 PM – 7:30 PM | Community Demos
Three 5-minute demos from members of the AI builder community. Please email [email protected] if you would like to present your demo.
7:30 PM – 8:30 PM | Networking & Refreshments
**Space is limited and entry is strictly first come, first served—arriving early gives you the best chance of getting in. There will be no entry once the event reaches capacity.