ExLlamaV2 (EXL2)
Variable bit-rate quantization engine for fastest GPU inference
Tools
In plain English
A custom racecar engine for NVIDIA GPUs that squeezes maximum tokens per second out of quantized weights.
Technically
An ultra-fast inference library and format (by Turboderp) written from scratch in C++/CUDA for NVIDIA GPUs. Uses mixed-precision EXL2 quantization (varying bits per layer based on error sensitivity) and optimized fused GEMM kernels for industry-leading decode tok/s.