Fine-tuning an open model for beginners: LoRA, QLoRA and when you need it
Learn when to fine-tune an open AI model instead of prompting or using RAG, how LoRA and QLoRA cut the cost, what data you need and what to check in the license.
Short answer. Try better prompts first, then retrieval from your own documents, and fine-tune only when you need consistent style, format or behavior that prompts cannot deliver. LoRA and QLoRA make fine-tuning possible on a single GPU by training a small add-on instead of the whole model. Check that the license of your base model allows it.
Three ways to adapt a model
| Approach | What you change | Best for | What it costs |
|---|---|---|---|
| Prompting | The instructions and examples you send | The first attempt, and most tasks | Nothing extra |
| Retrieval (RAG) | The documents the model can see for each question | Private or fast-changing knowledge, and answers that cite sources | An embedding model and a search index |
| Fine-tuning | The model’s weights | Consistent tone, format, domain jargon or tool-use habits | Data, GPU time and testing |
A useful rule: use retrieval to teach a model what to know, and fine-tuning to teach it how to behave. They also combine well: a fine-tuned model can answer in your format from documents that retrieval supplies.
Full fine-tuning, LoRA and QLoRA
Full fine-tuning updates every parameter. Training needs memory for the weights, for the gradients and for the optimizer’s running statistics, which commonly adds up to around 12 to 16 bytes per parameter. For a 7B model that is roughly 100 GB, so full fine-tuning is mostly a data-center activity.
LoRA freezes the base model and trains a small set of added weights, usually a small fraction of the parameters. The result is an adapter of tens or hundreds of megabytes that you load next to the unchanged base model, and one base can serve many adapters.
QLoRA trains a LoRA adapter on top of a base model that is quantized to 4 bits, which cuts the memory needed even further. The researchers who introduced it reported fine-tuning a 65B model on a single 48 GB GPU. A 7B to 8B model often fits on a 16 to 24 GB card, depending on sequence length and batch size.
What you need
- A clear task and a way to measure it. “Reply to support emails in our tone, with these sections” is a task. “Make the model better” is not.
- Examples. Pairs of input and ideal output, in the same chat format the model uses. A few hundred to a few thousand clean examples are often enough for style and format.
- A base model whose license allows it. A dense instruction-tuned model of 3B to 32B is the easiest place to start, such as Qwen3.8-27B, Gemma 4 or Olmo 3.1, all under Apache 2.0.
- A GPU, or a Mac. The model page shows the memory a model needs to run. Training needs more, as the QLoRA figures above suggest.
- A tool. Unsloth offers a desktop app, a web interface and a code package, and can export GGUF files. Transformers is the reference library that most training frameworks build on. MLX LM supports LoRA fine-tuning on a Mac.
A step-by-step path
- Write down the task and the success test. Decide how you will judge an answer as good.
- Try prompting and retrieval first. If they already work, stop here.
- Collect and clean examples. Remove personal data you have no right to use, fix errors and keep the style consistent. Set aside 10 to 20 percent as a test set that the model never trains on.
- Pick a base model and check its license allows fine-tuning and your use.
- Train a LoRA or QLoRA adapter with a small learning rate and a few passes over the data. Start small, and look at sample outputs as training goes.
- Compare with a baseline. Run the held-out examples through the base model with a good prompt and through your fine-tune, and judge both the same way.
- Deploy. Load the adapter next to the base model, merge it into the weights, or export a GGUF file for local tools. How to run AI models locally shows how to use the result.
Fine-tuning image and video models
Image and video generators are fine-tuned with LoRAs too. A handful of pictures can teach a style or a character, and the adapter loads next to the base model. Two details from our catalog:
- FLUX 3 Image Klein ships a separate Klein Base checkpoint meant for fine-tuning, next to the distilled 16-step version.
- LTX-2.5 comes with a trainer and a library of task LoRAs, and Wan 2.2 has a large community ecosystem of LoRAs.
Pitfalls
- Overfitting. Too many passes over too little data makes the model repeat your examples and fail on new inputs. Watch the held-out results, not the training loss.
- Forgetting. A fine-tune can lose general abilities. Test some tasks you did not train on.
- Template mismatch. Use the chat format that the model was built for, or results degrade in odd ways.
- Judging on training data. Always score on examples the model has never seen.
- Dirty data. Mistakes and inconsistencies in your examples become the model’s habits.
- License surprises. Conditions follow the derivative, as described below.
Check the license first
Fine-tuning usually creates a derivative of the base model, and the base license decides what you may do with it. Apache 2.0 and MIT allow it with light conditions. Other licenses add conditions that carry over to your result:
- The FLUX Kommunity license allows derivative models such as LoRA fine-tunes, to be distributed with the license and notice attached. It does not allow using the model or its outputs to improve another model that does a similar job.
- The LTX-2.x Community license says that fine-tunes and distilled models count as derivatives and stay under the same agreement.
- The LFM Open license lets you redistribute derivative works under the same license, and its US$10 million revenue limit still applies to you.
AI licenses explained walks through the rest.
Frequently asked questions
- How much data do I need to fine-tune a model?
- To teach a style, a format or a narrow task, a few hundred to a few thousand good examples are often enough. Teaching broader skills takes more. Quality matters more than quantity: a small, clean, consistent set usually beats a large messy one.
- Will fine-tuning teach the model new facts?
- Not reliably. Fine-tuning is better at changing behavior, such as tone, format and habits, than at adding knowledge. For facts that are private or change often, retrieval-augmented generation is usually the better tool.
- How much GPU memory do I need for QLoRA?
- A 7B to 8B model can often be fine-tuned with QLoRA on a single 16 to 24 GB GPU, depending on sequence length and batch size. The QLoRA authors reported fine-tuning a 65B model on one 48 GB GPU.
- Can I fine-tune a mixture-of-experts model?
- It is possible but harder and more memory-hungry, because all the experts must be loaded. Dense models are simpler to train, which is why beginners usually start with a dense model of 3B to 32B parameters.
- Can I sell or publish a fine-tuned model?
- Only if the base model's license allows derivatives, and its conditions usually follow your derivative: a license copy, a credit line, a revenue cap or output rules. Apache 2.0 and MIT allow it with light conditions. Check the license before you publish.
Key terms in this guide
- Embedding model An embedding model turns text, images or other data into a list of numbers that captures its meaning, so that similar items end up close together.
- Fine-tuning Fine-tuning continues a pretrained model's training on your own examples so that it behaves better for a specific task, style or domain.
- LoRA LoRA, short for low-rank adaptation, fine-tunes a model by training a small set of added weights instead of changing all of its parameters.
- Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
- Retrieval-augmented generation (RAG) Retrieval-augmented generation lets a model answer from your own documents by retrieving the relevant passages and adding them to the prompt.
- VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.
Models mentioned in this guide
| Model | Released | Parameters | Context | License | Smallest variant needs |
|---|---|---|---|---|---|
| Qwen3.8-27B Alibaba Qwen | 27.8B | 256K | Apache 2.0 (Permissive) | 19 GB Gaming GPU | |
| Gemma 4 Google DeepMind | 5.1B – 30.7B | 256K | Apache 2.0 (Permissive) | 3.52 GB Laptop or phone | |
| Olmo 3.1 Allen Institute for AI (Ai2) | 7.3B – 32.2B | 64K | Apache 2.0 (Permissive) | 5.04 GB Laptop or phone | |
| Granite 4.2 IBM | 3.7B – 29.3B | 128K | Apache 2.0 (Permissive) | 2.55 GB Laptop or phone | |
| FLUX 3 Image Klein Black Forest Labs | 5.6B | – | FLUX Kommunity (Conditional) | 8 GB Laptop or phone | |
| LTX-2.5 LTX (Lightricks) | 22B | – | LTX-2.x Community (Conditional) | 16 GB Laptop or phone |