AI Papers Roundtable (Loop Transformers) [members only]

​Hey all! We're going to be digging into Loop Transformers tonight with Ben as our host 🎉 .

​Papers for the night:

• ​Looped Transformers as Programmable Computers

  • ​Don't read the whole thing it's long! - I'd suggest just the parts on the overall approach, and not how they implement specific things.

• ​Looped Transformers for Length Generalization

​Then these are nice supplements. No need to read the whole thing, even the abstracts are worth it for discussion:

1. ​Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning

2. ​Universal Length Generalization with Turing Programs

3. ​Simulation of Graph Algorithms with Looped Transformers

​Finally, I'm going to assume the belief that transformers (or any other architecture) are limited / poor at length generalization (or most true generalization). If you want context on that, there are lots of possible papers, but this is a nice one: What Algorithms can Transformers Learn? A Study in Length Generalization

​Also for context, my internal narrative for why Looped Transformer are interesting goes like this:

1. ​Compositionality is key to intelligence, but ML is bad at it. Length Generalization is one key form of compositionality.

2. ​Transformers are provably limited at Composition. but... only under the context of fixed depth or without scaling inference.

3. ​Therefore, architectures designed for scaled depth/inference are interesting! (and complementary to just more RL).

4. ​Loop transformers are a simple example of this (and other approaches, such as just scaling inference via CoT are similar).  Intriguingly, they can be hand-crafted to length-generalize. But can they also learn this?

​In case you need a paper-reading assistant ahead of the discussion, try out Open Paper.