Quantization explained: Q4, FP8, GGUF and what you give up

Quantization shrinks a model by storing its weights with fewer bits. Learn what FP8, INT4, Q4_K_M, GGUF, MXFP4 and NVFP4 mean, how much they save and how to pick one.

  • Intermediate
  • 5 min read
  • Published
  • Updated

Short answer. Quantization stores each weight with fewer bits, for example 8 or 4 instead of 16, which cuts memory by 2 to 4 times and usually speeds up generation, at the cost of a small loss of quality. For language models on consumer hardware, 4-bit is the usual sweet spot. Choose 8-bit when you have memory to spare, and test on your own tasks.

Why quantize at all

A model’s weights are numbers. Stored in 16-bit format (BF16), each one takes 2 bytes. A 30-billion-parameter model therefore weighs about 60 GB, which no ordinary graphics card can hold. Store the same numbers in 4 bits and the weights shrink to roughly a quarter of that.

Smaller weights also help speed. When you chat with a model on your own machine, generating each token requires reading all the active weights from memory, so the limit is usually how fast memory can deliver them, not how fast the processor can calculate. Fewer bytes to read means more tokens per second.

The cost is accuracy. Each weight is rounded to one of fewer possible values, so the model’s behavior drifts a little from the original. Good quantization methods keep this drift small.

The formats you will meet

Name Bits per weight Where you meet it Notes
BF16, FP16 16 Original releases The reference quality and the largest size
FP8, INT8 8 Server engines such as vLLM and SGLang, and many official releases About half the size of BF16, usually close to the original; FP8 needs recent GPUs for full speed
GPTQ, AWQ about 4 GPU servers and some local tools Methods that quantize an already-trained model to 4 bits
GGUF levels (Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M and others) about 3 to 8 llama.cpp, LM Studio, Ollama One file with many levels; Q4_K_M is about 4.8 bits
MXFP4 about 4.25 gpt-oss, Granite 4.2 30B MXFP4, Kimi K3.1 Pro An open 4-bit floating-point format in which blocks of values share a scale
NVFP4 about 4.5 NVFP4 builds of Mistral Large 4, Muse Spark Open, Reflection Beam NVIDIA’s 4-bit floating-point format; needs Blackwell-class GPUs for native speed
Ternary about 1.7 Ternary Bonsai 2 Every weight is -1, 0 or +1 with a shared scale per 128 weights; needs a special runtime and loses some quality, more at small sizes (see ternary weights)

The precision of every variant of every model is listed on its model page.

Reading a GGUF name

GGUF is the single-file format used by llama.cpp and the tools built on it. Names such as Q4_K_M follow a pattern:

  • Q and a number: roughly how many bits each weight uses, so Q8 is about 8 bits and Q4 about 4.
  • K: the k-quant method, which quantizes in blocks and gives more precision to the tensors that matter most.
  • S, M or L: small, medium or large within that level. Larger means a little more precision and a slightly bigger file.

As a rule of thumb, Q8_0 is close to lossless, Q6_K and Q5_K_M are very close, Q4_K_M is the usual balance of size and quality, and below that the quality drops faster.

Native low precision and after-the-fact quantization

You can quantize a model after it is trained, or train it with low precision in mind. Models trained with quantization in mind tend to lose less.

  • gpt-oss was post-trained with MXFP4 quantization of its expert weights. That is why the 21B model is only about 14 GB.
  • MiniMax M3.1 Flash has experts and attention paths trained to be quantization-friendly and ships at about 252 GB for 428 billion parameters.
  • DeepSeek V4.1 Flash has 552 billion parameters in about 510 GB, which is less than one byte per parameter.
  • Nanbeige4.3-33B offers an INT4 build of about 23 GB made with quantization-aware training, distillation from the BF16 model and reinforcement learning on the INT4 model itself, although its card gives no scores for that build.

For models like these, the shipped checkpoint is already small, and re-quantizing it further gives less benefit and more risk.

How much quality do you lose

It depends on the model, the method and the task.

  • Eight bits: almost always indistinguishable in practice.
  • Four bits: a small loss for large and mid-size models. Small models lose more, because they have fewer weights to spare.
  • Below four bits: the loss grows quickly. Only use it when nothing else fits, and test carefully.
  • Sensitive tasks: math, code, structured output and long reasoning chains are the first to show damage. Check non-English text as well if you rely on it.

The honest answer is to test. Run 10 to 20 of your real prompts on two builds and compare the results side by side. How to evaluate models yourself shows how.

Quantizing the context cache

The weights are not the only memory user. The KV cache grows with the length of the conversation, and some engines can store it in 8 bits too, which saves memory on long contexts, again with a small quality risk. How much memory do I need explains how big the cache gets.

How to choose

Situation Suggestion
Laptop or gaming GPU with limited memory 4-bit, for example GGUF Q4_K_M
Memory to spare, or work that is sensitive to errors 8-bit (Q8_0 or FP8), or Q6_K
A very small model, under about 3B parameters 6 to 8 bits, because small models suffer more
Data-center GPUs FP8, or NVFP4 on Blackwell-class hardware
Fine-tuning on a single GPU LoRA on a 4-bit base model, called QLoRA
Image and video models INT8 or FP8 builds where the developer or community offers them, tested on your own prompts

Who makes the quantized files

Developers publish some of them, such as the FP8 and NVFP4 builds in our catalog. Community members publish many more, especially GGUF files, and tools such as Ollama repackage models into their own libraries. Prefer builds from the developer or from well-known publishers, check the file name and size match what you expect, and remember that the license of the original model still applies. Fine-tuning for beginners covers QLoRA.

Frequently asked questions

What does Q4_K_M mean?
It is a quantization level in the GGUF format. Q4 means roughly 4 bits per weight, K means the k-quant method, which uses block-wise scales and keeps some of the more important tensors at higher precision, and M means the medium variant of that level. It averages about 4.8 bits per weight.
Is 4-bit quantization noticeably worse than the original?
For everyday chat, a good 4-bit build of a mid-size or large model usually feels close to the original. The loss is larger for small models and for tasks that are sensitive to small errors, such as math, code and structured output, so test on your own examples.
Does quantization change the license?
No. A quantized copy is a derivative of the original model, and the license of the original still applies to it.
Can I quantize a model myself?
Yes. llama.cpp can quantize to GGUF, MLX LM can quantize models for Apple silicon and Unsloth can export GGUF and other formats. For popular models, ready-made quantized builds are usually published already.
Is a larger model at 4-bit better than a smaller model at 8-bit?
In the same amount of memory, a larger model at 4-bit often does better than a smaller one at 8-bit, which is one reason 4-bit is so popular. Results vary by model and task, so test both.

Key terms in this guide

  • GGUF GGUF is a single-file model format used by llama.cpp and the tools built on it, which bundles quantized weights with the information needed to run them.
  • KV cache The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.
  • LoRA LoRA, short for low-rank adaptation, fine-tunes a model by training a small set of added weights instead of changing all of its parameters.
  • Precision (BF16, FP8, INT4) Precision is how many bits are used to store each number in a model, such as 16-bit BF16, 8-bit FP8 or 4-bit INT4; fewer bits mean a smaller, faster model.
  • Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
  • Quantization-aware training (QAT) Quantization-aware training teaches a model to work with low-precision weights while it is still being trained, so it loses less quality at 4 bits than a model squeezed afterwards.
  • Ternary weights (1.58-bit) Ternary weights limit every model weight to one of three values, -1, 0 or +1, so a model needs under 2 bits per weight instead of 16.
  • VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.

Models mentioned in this guide

Model Released Parameters Context License Smallest variant needs
gpt-oss OpenAI 21B – 117B 5.1B active 128K Apache 2.0 (Permissive) 15 GB Laptop or phone
Granite 4.2 IBM 3.7B – 29.3B 128K Apache 2.0 (Permissive) 2.55 GB Laptop or phone
MiniMax M3.1 Flash MiniMax 428B 23B active 1M MiniMax Community (Conditional) 277 GB Big-memory desktop
Mistral Large 4 Mistral AI 1.05T 52B active 1M Mistral Open-Weight (Conditional) 671 GB GPU server
DeepSeek V4.1 Flash DeepSeek 552B 16B active 1M MIT (Permissive) 561 GB GPU server
Qwen3.8-27B Alibaba Qwen 27.8B 256K Apache 2.0 (Permissive) 19 GB Gaming GPU
Nanbeige4.3-33B Nanbeige LLM Lab 33.1B 512K Apache 2.0 (Permissive) 23 GB Gaming GPU