ailocalinference

Making Models Fit

Learn how huge models run on small devices via Quantization

2 modules4 lessons12 questions

01 Quantization Basics

What is Quantization?

Quantization is like saving a huge 4K photo as a JPEG. It looks almost the same, but it takes up way less space on your phone! 📸

Quality vs Size Tradeoff

If you compress a photo too much, it gets blurry. If you compress an AI too much, it gets a little bit stupid! 😵‍💫

02 Quantization Formats

GGUF — The Universal Format

GGUF is like a universal zip file for AI models! 🗂️ It holds everything your computer needs to run the AI in one clean package.

GPTQ & AWQ — GPU Quantization

While GGUF is great for CPUs and Macs, GPTQ and AWQ are special shrinking rays built specifically for powerful PC gaming graphics cards! 🎮