LoRA

LoRA, short for low-rank adaptation, fine-tunes a model by training a small set of added weights instead of changing all of its parameters.

  • Using models
  • Updated

A LoRA adapter is typically tens or hundreds of megabytes, while the base model stays unchanged. Because only the small adapter is trained, fine-tuning needs much less GPU memory, and one base model can be paired with many adapters.

Image generators use LoRAs a lot: a handful of pictures can teach a style or a character, and the adapter is loaded next to the base model. Variants that train on top of a quantized base model, such as QLoRA, cut the memory needed even further.

Related terms

  • Fine-tuning Fine-tuning continues a pretrained model's training on your own examples so that it behaves better for a specific task, style or domain.
  • Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.

Keep reading

All terms