

Arnav Chavan - Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Abstract: Fine-tuning teaches language models new skills, but often degrades capabilities they already have. In this talk, we present Subspace Tuning (SST), a method that improves this learning–forgetting tradeoff by protecting the principal singular directions of weight matrices and restricting updates to the remaining subspace. We delve into why it matters to project the optimizer’s actual update, and how refreshing the protected directions balances adaptation and retention. Experiments across reasoning domains, model sizes, and successive fine-tuning stages show that SST substantially reduces forgetting while maintaining strong performance on new tasks.
Speaker bio: Arnav Chavan is a PhD student at UC Berkeley and a Senior Applied Scientist at Amazon Lab126. His research focuses on continual learning and efficient generative models. At Amazon, he works on hardware–software co-design for on-device generative AI. Previously, he co-founded Nyun AI, serving as CTO before Amazon acqui-hired the company.