How much memory do I need to run an AI model? A simple guide

Estimate the VRAM or RAM an open AI model needs: the parameters-times-bytes rule, what quantization saves, why context length and mixture of experts matter, with worked tables.

  • Beginner
  • 6 min read
  • Published
  • Updated

Short answer. Memory in gigabytes is roughly the model’s parameters in billions, times the bytes per parameter, plus about 15% for overhead. A common 4-bit build uses about 0.6 bytes per parameter, so a 30B model needs around 20 GB and an 8B model about 5.5 GB. Long contexts need extra memory on top. For mixture-of-experts models, count the total parameters, not the active ones.

The rule of thumb

The weights are the bulk of what a model needs to hold in memory. Each parameter takes a fixed number of bytes, which depends on the precision of the file you download:

Format Bits per parameter Bytes per parameter
BF16 or FP16 16 2
FP8 or INT8 8 1
4-bit (INT4, FP4) 4 0.5
GGUF Q4_K_M, a common 4-bit build about 4.8 about 0.6
Ternary, as in Ternary Bonsai 2 about 1.7 about 0.22

The memory needed is then:

GB ≈ parameters in billions × bytes per parameter × 1.15

The 1.15 adds about 15% for runtime buffers and a modest amount of context. It is the same rule the VRAM calculator and the memory estimates on every model page use. Here is what it gives:

Parameters BF16 (16-bit) FP8 (8-bit) 4-bit (Q4_K_M style)
3B 6.9 GB 3.5 GB 2.1 GB
8B 18.4 GB 9.2 GB 5.5 GB
14B 32.2 GB 16.1 GB 9.7 GB
32B 73.6 GB 36.8 GB 22.1 GB
70B 161 GB 80.5 GB 48.3 GB
120B 276 GB 138 GB 82.8 GB

A worked example: Qwen3.8-27B has 27.78 billion parameters. At 4-bit the rule gives 27.78 × 0.6 × 1.15 ≈ 19 GB, which suits a 24 GB graphics card. The BF16 weights are about 56 GB to download, and Alibaba also publishes an FP8 version of about 31 GB.

These are estimates. When a developer states a minimum or ships a smaller build, the model page uses that figure instead.

Which memory counts

  • A graphics card with its own VRAM. The model runs fastest when it fits entirely in VRAM. If it does not fit, the software either refuses to load it or spills into slower system memory.
  • Apple silicon and other unified-memory machines. On a Mac, or on systems such as AMD Strix Halo and NVIDIA DGX Spark, the processor and graphics share one pool of unified memory, so a machine with 128 GB can hold a model that no 24 GB graphics card could. The operating system and your other apps also need part of that pool, and the trade-off is speed: memory is usually slower than on high-end GPUs.
  • System RAM only. Models can run on the processor alone, but slowly.
  • A mix. Tools such as llama.cpp can place some layers on the GPU and the rest in RAM. Capacity decides whether the model fits, and memory bandwidth decides how fast it generates text.

Quantization is the biggest lever

Quantization stores each weight with fewer bits. Going from 16-bit to 4-bit shrinks the model to roughly a quarter of its size, which is often the difference between needing a server and fitting on a gaming GPU. The price is a small loss of quality that grows as you use fewer bits and smaller models. Quantization explained covers the formats and the trade-offs.

Many large releases now ship in low precision from the start. gpt-oss was post-trained with MXFP4 quantization of its expert weights, which is why its 21B model is only about 14 GB, and DeepSeek V4.1 Flash ships its 552 billion parameters in about 510 GB.

Context length and the KV cache

To produce each new token, a model looks back at all the earlier ones. It keeps intermediate results in the KV cache so that it does not recompute them, and that cache lives in memory too. It grows with the length of the conversation and multiplies with the number of users served at once.

As an illustration, an 8B-class model with 32 layers and 8 key-value heads of size 128 stores about 128 KB of 16-bit cache per token. A 128,000-token context therefore needs roughly 16 GB of cache on top of the weights. That is more than the 4-bit weights of the same model.

Model designs differ a lot here. DeepSeek V4.1 Flash is built to save memory on long inputs, and Ling-3.1-flash uses 63 linear-attention layers out of 72 to keep a 1-million-token context affordable. If you plan to use long contexts, check the model page and leave generous headroom.

Mixture of experts: memory follows the total

A mixture-of-experts model has many specialist sub-networks and uses only a few of them for each token. That makes it fast, but all the experts must still be in memory.

gpt-oss 20B has 21 billion parameters in total and 3.6 billion active. You need memory for about 21 billion, which OpenAI says fits within 16 GB, but it generates text at roughly the speed of a much smaller model. GLM-5.3-Flash has 320 billion in total and 18 billion active, so it needs a very large memory even though each token is cheap to compute. Hubs and model pages list both numbers.

Image, video and speech models

These models ship as pipelines of several parts: the main generator, a text encoder and a decoder. Plan around the total download size, not just the headline parameter count.

  • FLUX 3 Image Klein has 5.6 billion parameters, but its BF16 download is about 20 GB. An INT8 build is meant for 8 GB of VRAM, with the text encoder as extra weight.
  • Qwen-Image-2.1 is 7.1 billion parameters, yet the full BF16 pipeline is about 33 GB because of its 8B text encoder. An INT8 repack is about 17 GB.
  • Wan 2.2 has a 5B model that runs on a 24 GB GPU, while its 27B text-to-video and image-to-video models need about 80 GB by Alibaba’s instructions.
  • LTX-2.5 asks for 80 GB or more in its full form. A distilled INT8 build gets it to about 16 GB.

ComfyUI can offload weights to system memory to fit larger models on a smaller card, at the cost of speed. Running image and video models locally goes into this.

What fits on typical hardware

The site groups models into six memory tiers, based on the smallest variant of each model. The examples below use our estimates for that smallest variant.

Tier Memory Typical machine Examples from our catalog
Laptop or phone up to 16 GB A typical laptop, phone or 8 to 16 GB graphics card MiniCPM5-2B (0.7 GB), Gemma 4 E2B (3.5 GB), Olmo 3.1 7B (5 GB), gpt-oss 20B (15.4 GB)
Gaming GPU 16 to 32 GB RTX 3090, 4090 or 5090, or a 32 GB Mac Qwen3.8-27B (19 GB), Muse Glimmer (20 GB), Wan 2.2 5B (24 GB)
Workstation 32 to 96 GB A 96 GB workstation card or several GPUs Kolibri-1 (54 GB)
Big-memory desktop 96 to 512 GB Mac Studio, AMD Strix Halo, NVIDIA DGX Spark Qwen3.8-Flash-Next (124 GB), GLM-5.3-Flash (221 GB), Ling-3.1-flash (386 GB)
GPU server 512 GB to 1.5 TB An 8-GPU H100, H200 or B200 node GLM-5.3 (520 GB), DeepSeek V4.1 Flash (561 GB), Mistral Large 4 (671 GB)
Multi-node cluster over 1.5 TB Several GPU servers working together Qwen3.8-Max (1.7 TB), Kimi K3.1 Pro (1.7 TB)

The models by VRAM page lists every model in each tier.

A five-step checklist

  1. Find the model’s total parameter count on its model page.
  2. Pick a precision: 4-bit for language models on consumer hardware, 8-bit if you have memory to spare.
  3. Apply the rule: parameters × bytes per parameter × 1.15.
  4. Add room for the context cache if you need long conversations or several users.
  5. Compare with your VRAM or unified memory, leave some for the operating system, and test it.

Frequently asked questions

How much VRAM do I need to run an 8B model?
About 5.5 GB at 4-bit precision, about 9 GB at 8-bit and about 18 GB at 16-bit by the rule used on this site, which adds 15% for overhead. Long conversations need more for the context cache.
Can I run a model that is bigger than my VRAM?
Often yes. Tools such as llama.cpp can keep some layers on the GPU and the rest in system memory, and ComfyUI can offload weights for image and video models. It works, but generation becomes much slower.
Do mixture-of-experts models need less memory?
No. Every expert must be loaded, so memory follows the total parameter count. What gets cheaper is speed, because only the active parameters are used for each token.
Does a longer context window need more memory?
Yes. The model stores a key-value cache for every token of the conversation, so a long context or many simultaneous users can take as much memory as the weights themselves.
Is RAM or VRAM more important?
VRAM, if you have a dedicated graphics card, because the model runs fastest when it fits entirely there. On Apple silicon and other unified-memory machines the CPU and GPU share one pool, so total memory is what counts.

Key terms in this guide

  • KV cache The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.
  • Mixture of experts (MoE) A mixture-of-experts model contains many specialist sub-networks and uses only a few of them for each token, so it can be very large yet fast.
  • Precision (BF16, FP8, INT4) Precision is how many bits are used to store each number in a model, such as 16-bit BF16, 8-bit FP8 or 4-bit INT4; fewer bits mean a smaller, faster model.
  • Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
  • Ternary weights (1.58-bit) Ternary weights limit every model weight to one of three values, -1, 0 or +1, so a model needs under 2 bits per weight instead of 16.
  • Unified memory Unified memory is a design, used by Apple silicon and some other chips, in which the CPU and GPU share one pool of memory, so a large model can use most of it.
  • VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.

Models mentioned in this guide

Model Released Parameters Context License Smallest variant needs
Gemma 4 Google DeepMind 5.1B – 30.7B 256K Apache 2.0 (Permissive) 3.52 GB Laptop or phone
Qwen3.8-27B Alibaba Qwen 27.8B 256K Apache 2.0 (Permissive) 19 GB Gaming GPU
gpt-oss OpenAI 21B – 117B 5.1B active 128K Apache 2.0 (Permissive) 15 GB Laptop or phone
GLM-5.3-Flash Z.ai (Zhipu AI) 320B 18B active 1M MIT (Permissive) 221 GB Big-memory desktop
FLUX 3 Image Klein Black Forest Labs 5.6B – FLUX Kommunity (Conditional) 8 GB Laptop or phone
Wan 2.2 Alibaba Wan 5B – 27B 14B active – Apache 2.0 (Permissive) 24 GB Gaming GPU