LoRA
Low-Rank Adaptation for lightweight model fine-tuning
Training
In plain English
Adding a lightweight pair of glasses to the AI rather than rewiring its entire brain, letting you customize it with just 0.1% of the storage.
Technically
Low-Rank Adaptation (Hu et al., 2021). Freezes the pre-trained model weights W_0 and injects trainable rank-decomposition matrices (ΔW = B × A, where r ≪ d) into transformer layers, cutting trainable parameters by 10,000x and VRAM overhead by 3x.