Grouped-Query Attention (GQA)
Sharing Key-Value heads across multiple Query heads
Architecture
In plain English
Having a group of 8 detectives share one assistant instead of each having their own, cutting the paperwork memory by 8x.
Technically
An attention architecture (Ainslie et al., 2023) that groups multiple query heads to share a single key-value head pair (e.g. 8 KV heads for 64 query heads in Llama 3 70B). Achieves an 8x reduction in KV cache size and memory bandwidth pressure with virtually identical perplexity to Multi-Head Attention.