Oliver Sieberling - Dynamic Short Convolutions Improve Transformers

​This talk introduces dynamic short convolutions as a scalable neural network primitive for improving Transformer-based language models. In contrast to static short convolutions, dynamic short convolutions use input-dependent filters, which allows them to adaptively aggregate local context. I will discuss how this simple modification improves expressivity and leads to stronger performance in Transformer-based language models, as well as how we implemented dynamic short convolutions efficiently using custom Triton kernels.

​

​Oliver Sieberling is a PhD student at MIT, advised by Yoon Kim. His research focuses on the pretraining of large neural networks, with particular interests in efficient sequence modeling, scalable architectures, and hardware-algorithm co-design. He received his BSc in Computer Science from ETH Zurich in 2025 and has worked on LLM efficiency and evolutionary algorithms.