Scaling Model Inference
AI is rapidly shifting toward scaling inference from test-time reasoning and long contexts to multi-turn agents and, now, multi-agent swarms. As a result, the inference frontier has become the model frontier. It dictates both what architectures we train and how much intelligence we can deploy in practice.
Drawing on Google DeepMind's "How To Scale Your Model" and modern serving systems literature, this talk derives distributed LLM inference from first principles.
Starting from hardware rooflines and Transformer math, we'll build up why modern serving stacks look the way they do, covering prefill/decode disaggregation, chunked prefill, prefix cache, and speculative decoding and explore the open research problems shaping the next generation of models.
About the speaker:
Alhaad Gokhale is a Principal Engineer at Google DeepMind in Bengaluru, where he leads LLM inference and serving efficiency efforts across frontier models.
Over 13+ years at Google, he has built low-latency, high-throughput distributed systems spanning web-scale Search serving infrastructure, operating systems, and real-time Search verticals.
Pre-read:
Looking forward to seeing you!