Podcast · Past conversation
Why is Speculative Decoding Fast?
37 min · Watch on YouTube ↗
Joe Barrow and I work through why speculative decoding can make an LLM faster even though it does more computation. A smaller model drafts tokens, a larger model checks them in parallel, and the speedup depends on how much of the GPU is waiting on memory.
We use the GPU roofline model to explain when speculation helps, what changes as server load increases, and how dynamic speculation adjusts the draft length. We also discuss diffusion models as drafters and the tradeoff between latency and throughput.
A diagram from the discussion
Joe’s diagram shows a larger model checking several draft tokens in one pass.
What we discussed
- How a small model drafts tokens and a larger model verifies them in parallel.
- Why memory bandwidth and spare GPU compute matter.
- How batch size and draft acceptance change the speedup.
- Dynamic speculation and using diffusion models to draft tokens.