Where Does the Slash in Attention Matrices Come From?

​Description:
Discover the origin of the “slash pattern” in LLM attention heatmaps. This talk reveals how Slash-Dominant Heads (SDH) arise from the interaction of RoPE positional encoding and model structure—and how they generalize to out-of-distribution inputs. Join us for insights from theory, mechanism, and empirical evidence.

​Speaker: Fengzhuo Zhang, Postdoctoral Researcher at Yale CADMY, PhD from National University of Singapore. Research focus: statistical learning and optimization theory for LLMs.

Paper: Demystifying the Slash Pattern in Attention: The Role of RoPE

​Host: Mingyu Jin, PhD Student at Rutgers University, specializing in language model interpretability and explainable reasoning. Homepage: https://mingyuj666.github.io/