MLn Club (ML Reading Group) #17: Speculative Decoding
Welcome to Week 17: Fast Inference from Transformers via Speculative Decoding
Well-written blog post (you might want to read this first)
This week, we’re diving into speculative decoding, one of the key techniques behind modern high-throughput LLM inference. The core problem is simple: autoregressive decoding is inherently sequential—generating (K) tokens normally requires (K) serial runs of the model. Speculative decoding instead uses a smaller, faster draft model to propose several tokens ahead, then verifies those proposals in parallel with the larger target model. Crucially, the algorithm can accelerate generation without changing the target model’s output distribution.
This also gives us a chance to go deeper on speculative decoding after it came up during our vLLM week. Rather than treating it as another inference optimization in the stack, we’ll look at the actual algorithm: how draft tokens are proposed and accepted, why rejected tokens require a corrected distribution, and how the speedup depends on the draft model’s acceptance rate and relative cost.
In particular, the paper is a nice bridge between the systems questions we discussed with vLLM—latency, memory bandwidth, and parallelism—and the probabilistic details required to make speculative execution exact.
Join us at CASI for discussion at 8 pm, and (optional) quiet reading from 7 pm.
📖 Reading Recommendations, Questions, or Comments? Contact us here!
🔎 View past meeting notes here.
What's this?
A super warm group of folks discussing their favorite topics!
In the first half, we host an optional quiet reading space
In the second half, we have a discussion where people can talk about what they found interesting about the reading and ask questions about things they didn't understand
When/Where:
CMU AI Safety Initiative's Office, 201 Craig Street, right across the PNC bank. Look for the open door up the stairs.
8pm discussion, 7pm optional quiet reading time.
Here's how it usually goes:
7:00 PM — arrival and settling in
8:00 PM — introductions
8:10 PM — discussion time
9:00 PM — wrap up then open discussion
Who's it for?
People who've been wanting to read up on the latest papers in ML and other fields but just haven't been able to find the time/motivation.
Why:
We've been procrastinating too much on our readings, even though we have so much fun doing them. We know we're not alone in this and want to keep others accountable for learning more about what they're passionate about!
We've also met a ton of really fun friends by discussing what we care about!
Rules/guidelines on how to act:
Act like a host, include people in conversations, talk to people even if they're strangers, offer to explain what you know, and keep an open mind! come to read stuff and find super fun friends :)
Bring snacks if you're feeling kind!