Scaling Long Context Transformers by Bowen Yang (Cohere)

​Bowen Yang from Cohere will give a lecture on the frontier of long contexts and transformers. The lecture will cover a range of approaches from data preparation, modeling methods, architecture innovations and framework-level optimizations, followed by Q&A.

This is an advanced lecture: Understanding of attention and ROPE is expected.

Some good reading materials:
1. Ring Attention: https://arxiv.org/abs/2310.01889
2. Rope to Nope and Back Again: https://arxiv.org/abs/2501.18795

​There will be snacks and drinks.

​The 535 AI Lecture Series is brought to you thanks to Cohere.