Quantization
Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
A 16-bit weight takes 2 bytes, and a 4-bit weight takes about half a byte. Quantizing a model from 16-bit to 4-bit therefore shrinks it to roughly a quarter of its size, which is often the difference between needing a server and fitting on a gaming GPU.
Common formats include FP8, INT8 and INT4, plus newer 4-bit floating-point formats such as MXFP4 and NVFP4. The GGUF format used by llama.cpp supports many quantization levels. The fewer the bits, the greater the risk of losing quality, especially in small models, so test at the level you plan to use.
Related terms
- Precision (BF16, FP8, INT4) Precision is how many bits are used to store each number in a model, such as 16-bit BF16, 8-bit FP8 or 4-bit INT4; fewer bits mean a smaller, faster model.
- GGUF GGUF is a single-file model format used by llama.cpp and the tools built on it, which bundles quantized weights with the information needed to run them.
- VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.
Keep reading
- Fine-tuning an open model for beginners: LoRA, QLoRA and when you need it
- How much memory do I need to run an AI model? A simple guide
- How to choose an open-source translation model
- How to evaluate AI models yourself: look past the benchmark charts
- How to run an AI model on your own computer
- Quantization explained: Q4, FP8, GGUF and what you give up
- Running image and video AI models locally: ComfyUI, memory and licenses