Base model

A base model is the raw result of pretraining, before it has been taught to follow instructions or chat; it is the usual starting point for fine-tuning.

  • Models and architecture
  • Updated

Pretraining on huge amounts of text produces a base model that continues text but does not reliably answer questions or follow instructions. Post-training, such as instruction tuning and often reinforcement learning, turns it into the chat or 'instruct' model most people use. A developer who also releases the base model lets you do your own post-training.

The word is used loosely. For language models it means the checkpoint before instruction tuning, as with Thinking Machines' Inkling, which is released as a base for customization. For image models such as Z-Image, 'base' means the undistilled model, which is slower but gives more varied results and is a better starting point for fine-tuning than the fast distilled Turbo version.

Related terms

  • Fine-tuning Fine-tuning continues a pretrained model's training on your own examples so that it behaves better for a specific task, style or domain.
  • Distillation Distillation trains a smaller student model to imitate a larger teacher model, keeping much of its ability at a fraction of the size.
  • LoRA LoRA, short for low-rank adaptation, fine-tunes a model by training a small set of added weights instead of changing all of its parameters.
  • Reinforcement learning (RL) Reinforcement learning trains a model by rewarding good outcomes, such as a correct answer or passing tests, instead of copying example text.

All terms