Quantization

Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.

  • Hardware and memory
  • Updated

A 16-bit weight takes 2 bytes, and a 4-bit weight takes about half a byte. Quantizing a model from 16-bit to 4-bit therefore shrinks it to roughly a quarter of its size, which is often the difference between needing a server and fitting on a gaming GPU.

Common formats include FP8, INT8 and INT4, plus newer 4-bit floating-point formats such as MXFP4 and NVFP4. The GGUF format used by llama.cpp supports many quantization levels. The fewer the bits, the greater the risk of losing quality, especially in small models, so test at the level you plan to use.

Related terms

  • Precision (BF16, FP8, INT4) Precision is how many bits are used to store each number in a model, such as 16-bit BF16, 8-bit FP8 or 4-bit INT4; fewer bits mean a smaller, faster model.
  • GGUF GGUF is a single-file model format used by llama.cpp and the tools built on it, which bundles quantized weights with the information needed to run them.
  • VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.

Keep reading

All terms