Paper Club #5: Transformers, Embeddings & RoPE

​Welcome to DEVx's next AI Paper Club!

​Ever wonder how a transformer knows which token came first, which came later, and how far apart they are?

​This month, Tom McTavish is taking us into transformers, embeddings, and RoPE with the paper RoFormer: Enhanced Transformer with Rotary Position Embedding.

​RoPE is a clever way to encode position directly into attention, and it has become a foundational piece of most of today’s LLMs.

​Paper: https://arxiv.org/abs/2104.09864

​All are welcome to come and learn. Questions and curiosity encouraged!

​📍 LOCATION CHANGE

​AI Paper Club will be at a new location this time:

​11988 El Camino Real
San Diego, CA 92130

​We’ll be in the seminar space right off the lobby.

​Please double-check the address before heading over, especially if you’ve attended a previous Paper Club at our old location. See you there!