Name: MLn Club (ML Reading Group) #17: Speculative Decoding
Start: 2026-09-13T19:00:00.000-04:00
End: 2026-09-13T21:00:00.000-04:00
Location: Pittsburgh, PA

We meet weekly on Sundays in person to talk about ML papers! Optional quiet reading from 7 pm, discussion starts 8 pm. Come and join the discussion! Open to all. Website: https://czhs.github.io/mln/

MLn Reading Club

Welcome to Week 17: Fast Inference from Transformers via Speculative Decoding

Well-written blog post (you might want to read this first)

, one of the key techniques behind modern high-throughput LLM inference. The core problem is simple: autoregressive decoding is inherently sequential—generating (K) tokens normally requires (K) serial runs of the model. Speculative decoding instead uses a smaller, faster 

 to propose several tokens ahead, then verifies those proposals in parallel with the larger target model. Crucially, the algorithm can accelerate generation 

without changing the target model’s output distribution

This also gives us a chance to go deeper on 

speculative decoding after it came up during our vLLM week

. Rather than treating it as another inference optimization in the stack, we’ll look at the actual algorithm: how draft tokens are proposed and accepted, why rejected tokens require a corrected distribution, and how the speedup depends on the draft model’s acceptance rate and relative cost.

In particular, the paper is a nice bridge between the systems questions we discussed with vLLM—

latency, memory bandwidth, and parallelism

—and the probabilistic details required to make speculative execution exact.

 for discussion at 8 pm, and (optional) quiet reading from 7 pm.

📖 Reading Recommendations, Questions, or Comments? Contact us 

People who've been wanting to read up on the latest papers in ML and other fields but just haven't been able to find the time/motivation.