Quantization
Compressing model weights from FP16 to 4-bit/8-bit precision
Formats
In plain English
Turning a 4K photo into an efficient JPEG. It looks almost identical to the human eye, but takes up 75% less room in your pocket.
Technically
The process of reducing numerical precision of weights and activations (e.g. FP16 16-bit floats to INT8/INT4/FP4 integers or micro-block floats). Lowers memory footprint and memory transfer pressure with minimal loss in model perplexity.