Learn · The Inference Pipeline · The Two Phases
Decode — Writing the Answer
In plain English
The Decode phase is the AI talking back to you, one word at a time. 🗣️ It has to think about everything it just said before saying the next word.
What is actually going on
Decoding is an auto-regressive process. The model takes the previously generated token, appends it to the context, and runs a forward pass to predict the next token. This is fundamentally memory-bandwidth bound.
Why it matters
Since every token requires loading the entire model from memory, decode speed is strictly limited by how fast your RAM is.
Another way to think about it
Laying down train tracks right in front of a moving train, piece by piece. 🚂
Check yourself
How does the decode phase generate text?
- All at once
- Auto-regressively, one token at a time
- In batches of 100
- By searching Wikipedia
Show answer
Auto-regressively, one token at a time — Each new token is predicted based on the sequence of all previous tokens.
The decode phase is bottlenecked by compute power (FLOPS).
- True
- False
Show answer
False — Decode is memory-bandwidth bound, because the entire model weight matrix must be loaded for just a single token.
Why can't decode happen in parallel like prefill?
- It uses too much electricity
- The next token depends on the current token
- The GPU isn't fast enough
- The context window is too small
Show answer
The next token depends on the current token — You can't predict token 5 until you know what token 4 is.