VRAM
VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.
A model's weights must be loaded into memory before it can answer. On a GPU that memory is VRAM. If the model does not fit, the software either refuses to load it or spills into slower system memory, and generation becomes much slower.
As a rule of thumb, the memory needed in gigabytes is about the parameter count in billions multiplied by the bytes per parameter, plus roughly 15% for overhead, and more for long contexts. At 4-bit precision a 30B model needs around 20 GB. The VRAM calculator on this site estimates it for a specific model at several precisions.
Related terms
- Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
- Unified memory Unified memory is a design, used by Apple silicon and some other chips, in which the CPU and GPU share one pool of memory, so a large model can use most of it.
- KV cache The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.
Keep reading
- How to choose an open AI model: a practical decision guide
- Fine-tuning an open model for beginners: LoRA, QLoRA and when you need it
- How much memory do I need to run an AI model? A simple guide
- How to run an AI model on your own computer
- Quantization explained: Q4, FP8, GGUF and what you give up
- Running image and video AI models locally: ComfyUI, memory and licenses