Precision (BF16, FP8, INT4)
Precision is how many bits are used to store each number in a model, such as 16-bit BF16, 8-bit FP8 or 4-bit INT4; fewer bits mean a smaller, faster model.
Models are usually trained in 16-bit formats such as BF16. A release may also be offered at lower precision: FP8 and INT8 use one byte per weight, and INT4 and FP4 use half a byte. A 30B model is roughly 60 GB in BF16, 30 GB in FP8 and 15 GB in 4-bit, before overhead.
Each model page on this site lists the precision of every shipped variant, and its memory table estimates the needs at BF16, FP8 and 4-bit.
Related terms
- Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
- VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.