90/30 Club (ML reading) #48: Inside vLLM: Anatomy of a High-Throughput LLM Inference System

​Week 48: Inside vLLM: Anatomy of a High-Throughput LLM Inference System

​Paper Link

​Modern LLM progress isn’t bottlenecked by model quality anymore, it’s bottlenecked by serving. This deep dive breaks down how systems like vLLM actually achieve high-throughput, low-latency inference at scale, from kernel-level memory tricks to distributed serving architecture.

​This post builds a full-stack mental model of inference: starting from a single-GPU offline engine and scaling all the way to multi-node production systems, with a focus on the mechanisms that actually matter in practice—scheduling, KV-cache management, and decoding strategies.

​Join us at Mox to explore:

• ​How paged attention reframes KV-cache as a memory virtualization problem (and why it’s the core unlock for throughput)

• ​Why continuous batching dominates naive batching—and how schedulers actually decide what runs next

• ​Tradeoffs between latency (TTFT, ITL) vs throughput, and how real systems optimize both simultaneously

​🔎Analyzed Papers

​​Discussion at 20:00, (optional) quiet reading from 19:00.