Name: MLn Club (ML Reading Group) #5: A Mathematical Framework for Transformer Circuits
Start: 2026-06-15T19:00:00.000-04:00
End: 2026-06-15T21:00:00.000-04:00
Location: Pittsburgh, PA

We meet weekly on Sundays in person to talk about ML papers! Optional quiet reading from 7 pm, discussion starts 8 pm. Come and join the discussion! Open to all. Website: https://czhs.github.io/mln/

MLn Reading Club

Week 5: A Mathematical Framework for Transformer Circuits

What does it mean to "fully understand" a model — and is a one-layer transformer just a lookup table?

If the residual stream is only a communication channel, where does the computation actually live?

A Mathematical Framework for Transformer Circuits

 Elhage, Nanda, Olsson, et al. (Anthropic)

This is a founding document of transformer mechanistic interpretability. The authors rewrite attention-only transformers in a mathematically equivalent form where the weights become directly readable: the residual stream is a passive communication channel, each head is an independent additive unit, and every head splits into a QK circuit (where to attend) and an OV circuit (what to write). Under this lens, a one-layer model is just an ensemble of bigram and "skip-trigram" tables — bugs included. The payoff comes at two layers, where heads compose through the residual stream: K-composition with a previous-token head produces induction heads, which find earlier occurrences of the current token and copy what came next — a real in-context algorithm rather than a lookup table, and the paper's candidate mechanism for in-context learning in large models.

 for discussion at 8 pm, and (optional) quiet reading from 7 pm.

Reading Recommendations, Questions, or Comments? Contact us 