Why is Speculative Decoding Fast?
With Isaac Flath and Joe Barrow
Why speculative decoding can make an LLM faster while doing more computation, and how GPU memory bandwidth, batch size, and draft acceptance affect the speedup.
The next conversation hasn’t been announced yet.
With Isaac Flath and Joe Barrow
Why speculative decoding can make an LLM faster while doing more computation, and how GPU memory bandwidth, batch size, and draft acceptance affect the speedup.
Unit tests can make strong benchmarks. Joe Barrow and I will discuss why they are useful, when they are a good fit, what their results can and cannot tell you, and how we would design them.
What decision models are, how to call them, and how Perplexity Decider works and was built.
A recorded reading and discussion of the RT-DocLayout paper with Isaac Flath and Joe Barrow, exploring PP-DocLayoutV3’s boxes, reading order, and segmentation.
Joining tables and paragraphs across pages, organizing headings, and deciding when the extra processing helps.