📊 AI BENCHMARK CLUB
tl;dr Come join us for technical talks focusing on AI benchmarks. We're a group of AI engineers, researchers, and academics.
Get ready for three incredible speakers / industry leaders:
Mark Saroufim from GPU Mode / Meta
Daniel Kim from Cerebras
This event is part of Open Source AI Week, right next to the PyTorch Conference venue.
Details
📆 Date & Time: Wednesday, October 22 @ 5:30 PM
📍 Location: SF, near Moscone West.
(Food and drinks provided. 🍕🍺)
Talk 1 by Mark will showcase BackendBench, for evaluating LLM-generated kernels with a focus on correctness
Talk 2 by George will cover Artificial Analysis's industry-shaping work in evaluating frontier models across many dimensions
Talk 3 by Daniel will discuss work his team did evaluating REAP, a new pruning method for sparse Mixture-of-Experts models on the agentic SWE-Bench benchmark.
Agenda:
5:30 - 6: Arrival, Food, Networking
6 - 7: Technical Talks, Q&A
7 - 7:30: Post Talk Networking
About the Meetup
Are you an engineer or researcher interested in measuring AI systems? This is a technical meetup where we dive into the guts of AI benchmarks of all stripes.
Interested in sharing AI benchmarking work you've done at a future session? Tell us about it.