PRETRAIN #1: Build a GPT

​ChatGPT, Claude and Gemini all rest on the same core: an autoregressive transformer that predicts the next token. In this hands-on workshop, you’ll build one yourself, line by line, in PyTorch. You’ll go from raw text to a working model that generates text of its own.

​By the end, attention, tokens and “next-token prediction” won’t be buzzwords anymore. You’ll know exactly how a GPT works, because you wrote one.

​What you’ll do
• Tokenize a text dataset and set up the training data
• Implement self-attention and a transformer block
• Train a small GPT on a laptop or free GPU
• Sample from your model and see what it learned

​Who it’s for: Students and builders who are comfortable with:
• Python, ideally with some PyTorch basics (tensors, a training loop, .backward())
• Basic linear algebra and analysis
• Basic probability (distributions, sampling, Gaussians)
• ML fundamentals: training vs. test set, overfitting, what a neural network is, and gradient descent at an intuitive level

​Bring: A laptop. We’ll share a notebook and starter repo.