FlashAttention
IO-aware exact attention algorithm eliminating memory bottlenecks
Performance
In plain English
A smart calculation technique that processes attention in small chunks in fast SRAM memory so the GPU never has to write huge intermediate matrices to slow VRAM.
Technically
An exact attention algorithm (Tri Dao et al., 2022/2023) that uses tiling and online softmax recomputation to compute attention in GPU SRAM (fast on-chip memory). Reduces memory reads/writes from O(N^2) to O(N), yielding 2x-4x speedups.