Name: MLn Club (ML Reading Group) #14: Inside vLLM: Anatomy of a High-Throughput LLM Inference System
Start: 2026-08-24T19:00:00.000-04:00
End: 2026-08-24T21:00:00.000-04:00
Location: Pittsburgh, PA

We meet weekly on Sundays in person to talk about ML papers! Optional quiet reading from 7 pm, discussion starts 8 pm. Come and join the discussion! Open to all. Website: https://czhs.github.io/mln/

MLn Reading Club

Welcome to Week 14: Inside vLLM: Anatomy of a High-Throughput LLM Inference System

When hyperscalers are pouring hundreds of billions into AI infrastructure, where does the money actually go — and how much of it ends up as useful tokens per second?

If prefill is compute-bound and decode is memory-bound, how much of the world's GPU fleet is sitting idle by design, and who captures the value when the serving layer that fixes it is open source?

The AI buildout is an inference buildout. Training a frontier model is a one-time cost; serving it to hundreds of millions of users, forever, is the bill that justifies the datacenters. Between the chip and the model sits the inference engine — the software that decides how many requests a GPU can hold at once, how much of its memory is wasted, and whether its compute is busy or waiting on HBM. Aleksa Gordić's 

 is an anatomy of the most widely deployed such engine, built up in layers: the scheduler and its waiting/running queues, paged attention (the KV cache treated like virtual memory, with block tables, reference counts, and eviction), continuous batching that flattens every request into one sequence, and the features on top — chunked prefill, prefix caching, grammar-constrained decoding, speculative decoding, disaggregated prefill/decode — then the multi-GPU and multi-node serving layer, closing with the roofline model that explains why latency and throughput are fundamentally in tension.

Each of those subsystems is a lever on capex efficiency: paged attention reclaims the memory that once limited batch size, prefix caching avoids recomputing shared prompts, speculative decoding fills idle compute during memory-bound decode, and disaggregation lets operators buy different hardware for prefill and decode.

 for discussion at 8 pm, and (optional) quiet reading from 7 pm.

📖 Reading Recommendations, Questions, or Comments? Contact us 

People who've been wanting to read up on the latest papers in ML and other fields but just haven't been able to find the time/motivation.