Quantization-aware training (QAT)

Quantization-aware training teaches a model to work with low-precision weights while it is still being trained, so it loses less quality at 4 bits than a model squeezed afterwards.

  • Hardware and memory
  • Updated

Ordinary quantization converts a finished 16-bit model to fewer bits, and the quality loss grows as the bits shrink. In QAT the low precision is part of training, so the weights adapt to it. Nanbeige4.3-33B-INT4 was built this way, with distillation from the BF16 model and reinforcement learning on the INT4 model itself, and the result is about 23 GB, although the model card gives no scores for it.

Some developers now publish low-precision weights as the main release. Moonshot ships Kimi K2.7 Code natively in INT4, at about 595 GB, and MiniMax M3.1 Flash is released already compressed to roughly 4-bit. It is usually safest to start with the developer's own low-precision files, and to test them on your own tasks before you trust a third-party conversion.

Related terms

  • Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
  • Precision (BF16, FP8, INT4) Precision is how many bits are used to store each number in a model, such as 16-bit BF16, 8-bit FP8 or 4-bit INT4; fewer bits mean a smaller, faster model.
  • Distillation Distillation trains a smaller student model to imitate a larger teacher model, keeping much of its ability at a fraction of the size.
  • Reinforcement learning (RL) Reinforcement learning trains a model by rewarding good outcomes, such as a correct answer or passing tests, instead of copying example text.

Keep reading

All terms