BioML Seminar 3.2 - Benchmarking Model Performance on Pandemic-Threat Viruses
[IN PERSON EVENT IN BERKELEY]
Join us for a new seminar from the BioML group in Machine Learning at Berkeley, sponsored by Amplify Partners. We're excited to host Sarah Gurev, an AI-for-Science Postdoctoral Fellow at FutureHouse.
Talk details: Viruses pose a significant threat to global health due to their rapid evolution, adaptability, and increasing potential for cross-species transmission. While advances in machine learning and the growing availability of sequence and structure data offer promise for large-scale mutation effect prediction, viruses present unique biological and informational constraints that may challenge these models. To quantify this, we introduce EVEREST—a framework for Evolutionary Variant Effect prediction with Reliability ESTimation—which assesses model performance on viral mutational fitness prediction using a curated benchmark of 45 viral deep mutational scanning datasets and quantifies model reliability in the absence of experimental data. This large-scale evaluation revealed wide differences in prediction accuracy across models and viral families. Protein language models have reached state-of-the-art performance at many mutation effect tasks, yet their effectiveness for viruses has been unclear despite their increasing deployment. We apply this framework across 40 WHO-prioritized pandemic-threat viruses, discovering that current models fail to reliably predict mutations in over half of these viruses. Our findings uncover key factors leading to underperformance, offer actionable recommendations for improving viral mutation effect prediction, and provide an objective framework for analyzing dual-use biosecurity risk.
Speaker details: Sarah Gurev is beginning a FutureHouse AI-for-Science Independent Postdoctoral Fellowship, co-advised by Sergey Ovchinnikov (MIT) and Aaron Schmidt (Ragon). She recently completed her PhD in Electrical Engineering and Computer Science at MIT (and a short postdoc) in Debora Marks lab at Harvard. She works on deep generative models of proteins--with a focus on modeling viral and immune protein sequences and structures--applied to viral evolution prediction, virus-host interactions, and future-proofed vaccine/antibody design.