MLn Club (ML Reading Group) #14: Inside vLLM: Anatomy of a High-Throughput LLM Inference System
Welcome to Week 14: Inside vLLM: Anatomy of a High-Throughput LLM Inference System
When hyperscalers are pouring hundreds of billions into AI infrastructure, where does the money actually go — and how much of it ends up as useful tokens per second?
If prefill is compute-bound and decode is memory-bound, how much of the world's GPU fleet is sitting idle by design, and who captures the value when the serving layer that fixes it is open source?
The AI buildout is an inference buildout. Training a frontier model is a one-time cost; serving it to hundreds of millions of users, forever, is the bill that justifies the datacenters. Between the chip and the model sits the inference engine — the software that decides how many requests a GPU can hold at once, how much of its memory is wasted, and whether its compute is busy or waiting on HBM. Aleksa Gordić's Inside vLLM is an anatomy of the most widely deployed such engine, built up in layers: the scheduler and its waiting/running queues, paged attention (the KV cache treated like virtual memory, with block tables, reference counts, and eviction), continuous batching that flattens every request into one sequence, and the features on top — chunked prefill, prefix caching, grammar-constrained decoding, speculative decoding, disaggregated prefill/decode — then the multi-GPU and multi-node serving layer, closing with the roofline model that explains why latency and throughput are fundamentally in tension.
Each of those subsystems is a lever on capex efficiency: paged attention reclaims the memory that once limited batch size, prefix caching avoids recomputing shared prompts, speculative decoding fills idle compute during memory-bound decode, and disaggregation lets operators buy different hardware for prefill and decode.
Join us at CASI for discussion at 8 pm, and (optional) quiet reading from 7 pm.
📖 Reading Recommendations, Questions, or Comments? Contact us here!
🔎 View past meeting notes here.
What's this?
A super warm group of folks discussing their favorite topics!
In the first half, we host an optional quiet reading space
In the second half, we have a discussion where people can talk about what they found interesting about the reading and ask questions about things they didn't understand
When/Where:
CMU AI Safety Initiative's Office, 201 Craig Street, right across the PNC bank. Look for the open door up the stairs.
8pm discussion, 7pm optional quiet reading time.
Here's how it usually goes:
7:00 PM — arrival and settling in
8:00 PM — introductions
8:10 PM — discussion time
9:00 PM — wrap up then open discussion
Who's it for?
People who've been wanting to read up on the latest papers in ML and other fields but just haven't been able to find the time/motivation.
Why:
We've been procrastinating too much on our readings, even though we have so much fun doing them. We know we're not alone in this and want to keep others accountable for learning more about what they're passionate about!
We've also met a ton of really fun friends by discussing what we care about!
Rules/guidelines on how to act:
Act like a host, include people in conversations, talk to people even if they're strangers, offer to explain what you know, and keep an open mind! come to read stuff and find super fun friends :)
Bring snacks if you're feeling kind!