📔 Deep Learning Classics [public]

​We’ll explore the technical details of the most significant advances in AI from the past decade. Every week, we’ll read and discuss two or three classic AI papers on a related theme.

​We want the class to work for a wide range of levels of experience. If you're relatively new to AI, by the end of the class, you’ll be comfortable reading and understanding AI papers, and you’ll have an understanding of how AI has developed over time. If you're already very experienced, we think you'll still get something new out of reading and discussing these papers again today, especially in light of the transformer-related advances of the last few years.

​To make the class more interactive, each paper will be presented by one of you! You should come to class having read (and understood as best you can) the papers assigned that week, and bring any questions you have so we can talk about them together.

​If you feel up for it, there’s also an optional project during the last few weeks of the class. You’ll work in a 2-3 person group to reproduce one of these papers, and possibly extend it to a new domain, or to some variant that you invent.

​

​~~~ Tentative Curriculum ~~~

​Week 1: Image Recognition

• ​Imagenet

• ​AlexNet

• ​(Optional) VGG, ResNet

​Week 2: Early NLP

• ​Seq2seq

• ​Word2vec

• ​(Optional) LSTM

​Week 3: Reinforcement Learning

• ​DQN

• ​A2C

• ​(Optional) REINFORCE, TD-Gammon

​Week 4: Solving Games

• ​AlphaGo

• ​PPO

• ​OpenAI Five

• ​(Optional) AlphaGo Zero / Alpha Zero

​Week 5: Training Techniques

• ​Adam

• ​Batch Norm

• ​Dropout

• ​(Optional) Layer Norm

​Week 6: Image Generation

• ​GANs

• ​Diffusion models

• ​CLIP

• ​(Optional) Latent diffusion

​Week 7: LM Pretraining

• ​Attention Is All You Need

• ​BERT

• ​GPT-1

• ​(Optional) ViT

​Week 8: LM Scaling

• ​GPT-2

• ​GPT-3

• ​Scaling Laws

• ​(Optional) Chinchilla, Gato