Ternary weights (1.58-bit)

Ternary weights limit every model weight to one of three values, -1, 0 or +1, so a model needs under 2 bits per weight instead of 16.

  • Hardware and memory
  • Updated

Three values carry log2(3), or about 1.58, bits of information, so ternary models are often called 1.58-bit models. Real files need a little more, because each group of weights also stores a scale factor. PrismML's Ternary Bonsai 2 uses one 16-bit scale for every 128 weights, which comes to about 1.7 bits per weight and shrinks a model to roughly a ninth of its 16-bit size.

Ternary weights are an extreme form of quantization. Quality drops, more for small models than for large ones, and the files usually need a special runtime: Ternary Bonsai 2's most compact packings run on PrismML's llama.cpp fork. Our VRAM calculator counts ternary files at 0.22 bytes per parameter.

Related terms

  • Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
  • Precision (BF16, FP8, INT4) Precision is how many bits are used to store each number in a model, such as 16-bit BF16, 8-bit FP8 or 4-bit INT4; fewer bits mean a smaller, faster model.
  • GGUF GGUF is a single-file model format used by llama.cpp and the tools built on it, which bundles quantized weights with the information needed to run them.

Keep reading

All terms