Inference

Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.

  • The basics
  • Updated

Training teaches a model by adjusting its parameters over huge amounts of data, and it needs clusters of GPUs. Inference only reads the finished weights and produces answers, so it is far cheaper and can often run on a single machine.

When you run an open model with a tool such as Ollama, LM Studio, llama.cpp or vLLM, you are doing inference. How fast it runs depends on the memory capacity, the memory bandwidth and the engine you use.

Related terms

  • Fine-tuning Fine-tuning continues a pretrained model's training on your own examples so that it behaves better for a specific task, style or domain.
  • Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
  • VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.

Keep reading

All terms