Learn · Foundations · The Three Axes
The Software Stack
In plain English
You wouldn't use a screwdriver to hammer a nail! 🛠️ Different AI tasks need different software tools to run smoothly on your computer.
What is actually going on
The inference stack involves high-level APIs (Ollama), inference engines (llama.cpp, MLX), and hardware-specific backends (Metal, CUDA). Matching the right software to your hardware ensures you utilize accelerators efficiently.
Why it matters
Using unoptimized software can result in 10x slower performance, even on high-end hardware.
Another way to think about it
Software is the translator between the AI's ideas and the computer's native language. 🗣️
Check yourself
What is the purpose of an inference engine like llama.cpp?
- To train models from scratch
- To execute models efficiently on local hardware
- To design UI
- To download data
Show answer
To execute models efficiently on local hardware — Inference engines are optimized to run the math required for model generation.
Which backend is optimized for Apple Silicon?
- CUDA
- Vulkan
- Metal
- ROCm
Show answer
Metal — Metal is Apple's native hardware-accelerated graphics and compute API.
Any software stack will perform exactly the same on the same hardware.
- True
- False
Show answer
False — Optimized software (using correct backends and kernels) is critical for max performance.