

๐ AI BENCHMARK CLUB
โโtl;dr Come join us for technical talks focusing on AI benchmarks. We're a group of AI engineers, researchers, and academics.
โGet ready for three incredible speakers / industry leaders:
โMark Saroufim from GPU Mode / Meta
โGeorge Cameron from Artificial Analysis
โDaniel Kim from Cerebras
โThis event is part of Open Source AI Week, right next to the PyTorch Conference venue.
โDetails
โโ๐ Date & Time: Wednesday, October 22 @ 5:30 PM
๐ Location: SF, near Moscone West.
(Food and drinks provided. ๐๐บ)
โTalk 1 by Mark will showcase BackendBench, for evaluating LLM-generated kernels with a focus on correctness
โTalk 2 by George will cover Artificial Analysis's industry-shaping work in evaluating frontier models across many dimensions
โTalk 3 by Daniel will discuss work his team did evaluating REAP, a new pruning method for sparse Mixture-of-Experts models on the agentic SWE-Bench benchmark.
โAgenda:
โโ5:30 - 6: Arrival, Food, Networking
โโ6 - 7: Technical Talks, Q&A
โโ7 - 7:30: Post Talk Networking
โAbout the Meetup
โAre you an engineer or researcher interested in measuring AI systems? This is a technical meetup where we dive into the guts of AI benchmarks of all stripes.
โโInterested in sharing AI benchmarking work you've done at a future session? Tell us about it.