Anupam Nayak - Late-Stage Patching Full-Attention Transformers to Gated DeltaNets

​Description: This talk explores how model architecture can be changed late in training rather than remaining fixed after pretraining. In particular, pretrained full-attention transformers can be converted into more efficient hybrid architectures without retraining from scratch. Prior conversion approaches largely recover short-context performance but struggle to recover long-context reasoning and retrieval. We show that a pipeline of representation alignment, off-policy distillation, and prolonged on-policy distillation can recover these capabilities using only a small fraction of the original pretraining budget, suggesting that architectural choices can remain flexible much later in a model’s training lifecycle.

Bio: Anupam Nayak (https://hunchings.github.io) is a Ph.D. student in Electrical and Computer Engineering at Carnegie Mellon University, advised by Prof. Gauri Joshi. His research focuses on post-training and RL scaling, continual learning, and AI alignment. He was a Student Researcher at Google Research NYC, where he worked on mid training and continual learning for language models. He previously was an Electrical Engineering undergrad at IIT Bombay.