​May I have your Attention, please?

​Lossfunk Research Mixers are intimate, topic-first roundtables for people in Bengaluru with a real knack for research, whether they are researchers, engineers, writers, philosophers, artists, interdisciplinary thinkers, or independent tinkerers.

Topic on 22nd July: Transformers from the First Principles

​You Should Apply If

• ​You have a strong research instinct, whether or not you hold a formal research role.

• ​You enjoy going unusually deep into ideas across science, technology, philosophy, art, cognition, or culture.

• ​You’re actively exploring, building, reading, or investigating something you’re excited to discuss.

• ​You want a rigorous peer group, to battle-test ideas, and to meet unusually curious people from different disciplines who feel like they’re in the same orbit.

​We believe great researchers can come from anywhere. We especially encourage students, early-career professionals, motivated women, and people taking unconventional paths to apply.

​​A glimpse of the minds we mix:

• ​A top-30 world-ranked chess grandmaster who hangs out with researchers for fun

• ​A 20 y/o IIT Bombay student collaborating with a researcher at Yann LeCun's AMI Labs

• ​Cracked founders who've already sold their companies and are now chasing research for its own sake

• ​A transhumanist researcher who has worked at two of MIT's top labs (cough cough)

• ​Ex applied and foundational researchers from Microsoft

• ​Ex YC-backed founding engineer, still deeply curious about research

• ​Researchers with publications at ICML, ICLR, AAAI, and ACL

• ​Neuroscientists from top labs at IISc (one’s a solid artist, btw)

• ​A JEE All-India Rank 239, GTM dominator turned researcher

• ​A CMU graduate who's currently solving in robotics

• ​Young students and professionals who fly in from other cities to be in the room

• ​Bunch of our dear Lossfunk Researchers and Residents

​Transformers from the First Principles

​Transformers are everywhere. But how much of the architecture can we actually derive from first principles?

​We’ll start with the problem of representing language, briefly tracing the path from bag-of-words to Word2Vec and why static embeddings eventually fall short.

​From there, we’ll build attention from first principles, understanding what a single attention head computes, how softmax turns scores into information flow, and why we scale by √d. We’ll then build toward multi-head attention and explore questions around complexity and rank collapse.

​As the architecture comes together, we’ll look at normalization, encoder/decoder architectures, causal language modelling, and information flow through transformers, while introducing ideas like permutation invariance and equivariance along the way.

​The session will be highly interactive: slides for the foundations, followed by whiteboarding, derivations, intuition, and simulations.

​Presented by Srinitish Srinivasan

​Organized by Lossfunk, your friendly neighborhood AI lab.