LoRA
LoRA, short for low-rank adaptation, fine-tunes a model by training a small set of added weights instead of changing all of its parameters.
A LoRA adapter is typically tens or hundreds of megabytes, while the base model stays unchanged. Because only the small adapter is trained, fine-tuning needs much less GPU memory, and one base model can be paired with many adapters.
Image generators use LoRAs a lot: a handful of pictures can teach a style or a character, and the adapter is loaded next to the base model. Variants that train on top of a quantized base model, such as QLoRA, cut the memory needed even further.
Related terms
- Fine-tuning Fine-tuning continues a pretrained model's training on your own examples so that it behaves better for a specific task, style or domain.
- Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.