Feindel Seminar Series: From Generalist to Specialist: Integrating Clinical Knowledge into Large Vision-Language Models
From Generalist to Specialist: Integrating Clinical Knowledge into Large Vision-Language Models
Foundational vision–language models (VLMs) are impacting computer vision and increasingly being explored in medical imaging as a promising route toward robust generalization. However, directly applying these generalist models, largely trained on natural images, has shown limited success in clinical settings due to significant domain shifts and the specialized expertise required for medical interpretation. These challenges have motivated substantial research into building domain-specific foundation models that incorporate medical knowledge. In this presentation, I will provide an overview of state-of-the-art methods and recent advances in this rapidly evolving area. I will discuss strategies that embed clinical expertise during both the pre-training and fine-tuning (adaptation) stages of medical VLMs. Such language-based clinical knowledge, derived from medical literature, encodes fine-grained pathological features, relevant biomarkers, and the relationships between them, thereby enabling models to better capture the structure of medical image interpretation. I will illustrate these concepts with practical examples and empirical findings, emphasizing both the benefits of integrating domain knowledge and the limitations of generalist models when applied to medical imaging tasks.
Ismail Ben Ayed
Ismail Ben Ayed is a Full Professor at ÉTS Montréal, where he holds a Research Chair in Artificial Intelligence for Medical Imaging. He is also a Principal Investigator at the University of Montreal Hospital Research Centre. His research spans machine learning, computer vision, optimization, and medical image computing. Ismail has authored more than 200 peer-reviewed publications in leading venues across these fields, along with two books and seven U.S. patents. Over the past five years, he has delivered more than 30 invited talks, including tutorials on foundation models at flagship conferences such as MICCAI. His work has been featured in major media outlets, including CTV News, Radio-Canada (CBC) and Québec Science Magazine. His team has received several international distinctions, including the Best Paper Award at the Medical Imaging with Deep Learning (MIDL) conference in 2021, as well as multiple top-ranking results in international challenges. Ismail serves as Executive Editor of the MELBA journal and as Associate Editor for Frontiers in Computer Science and the CMIG journal. He has been a regular Program Committee member for MICCAI and has served as Program Chair for IPMI 2025 and MIDL 2020.
Titles and affiliation
Full Professor, ETS Montreal; Researcher, CRCHUM; Chief AI Scientist, Diagnos Inc.
For more information, visit the web page
To watch online, click here