Learn · Speed Hacks · Speculative Decoding
The Draft & Verify Trick
In plain English
Imagine a fast intern drafting an email, and the boss quickly approving it. If the intern makes a mistake, the boss corrects just that part! 👔
What is actually going on
Speculative decoding uses a tiny, fast 'draft' model to predict multiple future tokens. The large 'target' model then evaluates these tokens in parallel. Accepted tokens are emitted instantly, bypassing auto-regressive bottlenecks.
Why it matters
It can double or triple tokens/second without any loss in generation quality, as the large model strictly guarantees output correctness.
Another way to think about it
Autocorrect predicting your next 5 words, and you just hitting 'Accept'. ✅
Check yourself
What are the two models used in speculative decoding?
- A cloud model and a local model
- A draft model and a target model
- An image model and a text model
- A prefill model and a decode model
Show answer
A draft model and a target model — The draft model predicts quickly, and the target model verifies.
Speculative decoding reduces the intelligence/quality of the final output.
- True
- False
Show answer
False — The target (large) model has the final say, meaning the output quality is mathematically identical to running the large model alone.
Why is verifying multiple tokens faster than generating them normally?
- Because the draft model is smarter
- Verification can happen in parallel
- It skips the attention mechanism
- It runs on the CPU
Show answer
Verification can happen in parallel — The large model can verify multiple drafted tokens in a single forward pass, bypassing the auto-regressive bottleneck.