90/30 Club (ML reading) #56: Kernel Krazy p2
Week 56: ThunderKittens + Kernel Review
Additional Review
Extra Resources
This week, we'll be discussing ThunderKittens: A Framework for Fast GPU Kernels. The paper introduces a programming model designed to make high-performance GPU kernel development significantly more accessible while maintaining performance competitive with hand-optimized CUDA implementations. Through a set of abstractions built around tiles, warps, and asynchronous execution, ThunderKittens demonstrates how complex kernels for matrix multiplication, attention, state-space models, and other AI workloads can be expressed with dramatically less code while still achieving near-peak hardware utilization. As modern AI systems increasingly rely on custom kernels to unlock performance beyond general-purpose libraries, ThunderKittens raises an important question: can we make writing highly optimized GPU code substantially easier without sacrificing speed? The paper offers a compelling perspective on the future of GPU programming and serves as an excellent bridge between low-level CUDA optimization and emerging domain-specific kernel frameworks.
Join us at Mox to explore:
What design choices allow ThunderKittens to achieve performance competitive with highly tuned CUDA kernels while requiring significantly less implementation complexity.
As AI models continue to introduce new operators, attention mechanisms, and state-space architectures, will future GPU development rely more on domain-specific kernel frameworks than hand-written CUDA?
Discussion at 20:00, (optional) quiet reading from 19:00.