Distillation
Distillation trains a smaller student model to imitate a larger teacher model, keeping much of its ability at a fraction of the size.
The student learns from the teacher's outputs instead of only from raw data. This is how many compact models reach strong results, and it is also used to make image models faster: the distilled FLUX Klein needs far fewer sampling steps than its undistilled Base version.
Some licenses restrict using a model's outputs to train another model, so check the terms before you distill from an open model into your own.
Related terms
- Fine-tuning Fine-tuning continues a pretrained model's training on your own examples so that it behaves better for a specific task, style or domain.
- Parameters Parameters are the numbers inside a model that are adjusted during training; their count, such as 7B or 70B, is the usual measure of a model's size.