Decode
The memory-bandwidth-bound phase generating tokens one-by-one
Fundamentals
In plain English
The AI writing the answer word by word on the page.
Technically
The autoregressive generation phase where tokens are produced sequentially. Each step requires streaming every model parameter from memory to registers for 2 FLOPs of arithmetic, yielding arithmetic intensity ≤ 2 FLOPs/byte and making throughput strictly proportional to memory bandwidth.