Cover Image for Arnav Chavan - Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Cover Image for Arnav Chavan - Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Led by Ahmad Mustafa Anis and Kanwal Mehreen. Part of the Cohere Labs Open Science initiative https://cohere.com/research/open-science
Hosted By

Arnav Chavan - Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

Google Meet
Registration
Welcome! To join the event, please register below.
About Event

​Abstract: Fine-tuning teaches language models new skills, but often degrades capabilities they already have. In this talk, we present Subspace Tuning (SST), a method that improves this learning–forgetting tradeoff by protecting the principal singular directions of weight matrices and restricting updates to the remaining subspace. We delve into why it matters to project the optimizer’s actual update, and how refreshing the protected directions balances adaptation and retention. Experiments across reasoning domains, model sizes, and successive fine-tuning stages show that SST substantially reduces forgetting while maintaining strong performance on new tasks.

Speaker bio: Arnav Chavan is a PhD student at UC Berkeley and a Senior Applied Scientist at Amazon Lab126. His research focuses on continual learning and efficient generative models. At Amazon, he works on hardware–software co-design for on-device generative AI. Previously, he co-founded Nyun AI, serving as CTO before Amazon acqui-hired the company.

Led by Ahmad Mustafa Anis and Kanwal Mehreen. Part of the Cohere Labs Open Science initiative https://cohere.com/research/open-science
Hosted By