VRAM

VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.

  • Hardware and memory
  • Updated

A model's weights must be loaded into memory before it can answer. On a GPU that memory is VRAM. If the model does not fit, the software either refuses to load it or spills into slower system memory, and generation becomes much slower.

As a rule of thumb, the memory needed in gigabytes is about the parameter count in billions multiplied by the bytes per parameter, plus roughly 15% for overhead, and more for long contexts. At 4-bit precision a 30B model needs around 20 GB. The VRAM calculator on this site estimates it for a specific model at several precisions.

Related terms

  • Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
  • Unified memory Unified memory is a design, used by Apple silicon and some other chips, in which the CPU and GPU share one pool of memory, so a large model can use most of it.
  • KV cache The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.

Keep reading

All terms