90/30 Club (ML reading) #54: TPU Performance

​Week 54: In-Datacenter Performance Analysis of a Tensor Processing Unit

link: https://arxiv.org/pdf/1704.04760

​Google’s TPU paper shows why modern AI is not just a model problem, but a datacenter systems problem. Instead of treating neural network inference as something CPUs or GPUs can “also” do, Google built a domain-specific chip around the actual bottlenecks of production inference: latency, power, memory movement, and predictable response times. The result was a 65,536 MAC systolic array with large on-chip memory, designed to serve real user-facing models under strict 99th percentile latency constraints. The key lesson is that inference hardware is not about peak FLOPs alone; it is about matching architecture to deployment reality.

Join us at Mox to explore:

• ​How did Google’s TPU design reveal the difference between training hardware and inference hardware?

• ​Why do latency guarantees matter more than average throughput in user-facing AI systems?

​🔎Analyzed Papers
✍​Google Drive for sharing Comments

​Discussion at 20:00, (optional) quiet reading from 19:00.