Attention May Be All You Need — But How Does It Actually Work?
Please join Charlottesville Data Science for our December event, Attention May Be All You Need — But How Does It Actually Work?
In this talk, Joel Fernandes, a Principal Engineer at NVIDIA, will demystify the attention mechanism foundational to modern AI systems, then survey a variety of tricks and techniques researchers are using to speed it up and scale it to ever-larger context sizes.
About the talk
Attention is at the heart of modern language models, but what actually happens when a model processes a prompt or generates its next token? This talk builds up the mechanics of attention through diagrams and small, concrete examples. We'll follow queries, keys, and values through attention scores, causal masking, and weighted sums, then connect those operations to the KV cache that makes token-by-token generation practical.
From that foundation, we'll survey techniques used to make attention more efficient, including grouped-query attention, sliding-window attention, sparse attention, and linear attention. We'll look at what each changes, why it helps, with examples.
Familiarity with machine learning is helpful, but no prior knowledge of attention internals is assumed.
About the speaker
Joel Fernandes is a Principal Engineer at NVIDIA based in Charlottesville, Virginia, with interests spanning systems software, machine learning, and efficient LLM inference. Previously, he spent over eight years at Google working on Linux kernel development and performance for Android and ChromeOS. He is a longtime Linux kernel contributor and has presented at conferences including Linux Plumbers and Kernel Recipes. He also organizes an informal LLM Internals Club at NVIDIA and writes about model and systems internals at https://www.joelfernandes.org.