Learn · Making Models Fit · Quantization Basics
Quality vs Size Tradeoff
In plain English
If you compress a photo too much, it gets blurry. If you compress an AI too much, it gets a little bit stupid! 😵💫
What is actually going on
Aggressive quantization (e.g., 2-bit or 3-bit) increases perplexity, meaning the model's predictive accuracy degrades. The sweet spot for modern LLMs is usually 4-bit or 5-bit, which retains ~99% of FP16 capability.
Why it matters
Finding the balance ensures the model runs fast (due to lower bandwidth needs) without outputting gibberish.
Another way to think about it
A photocopy of a photocopy. Eventually, you can't read the text. 📄
Check yourself
What happens if you quantize a model too aggressively (e.g., to 2-bit)?
- It gets smarter
- It runs slower
- Its reasoning quality degrades severely
- It requires internet
Show answer
Its reasoning quality degrades severely — Too much compression loses crucial information in the weights.
What is generally considered the 'sweet spot' for quantization?
- 1-bit
- 4-bit
- 16-bit
- 32-bit
Show answer
4-bit — 4-bit quantization reduces size by 75% with barely noticeable quality loss.
Quantization only affects file size, not output quality.
- True
- False
Show answer
False — Quantization alters the weights slightly, which can affect the model's accuracy and output quality.