KV cache
The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.
To produce each new token, the model looks back at all the earlier tokens. Instead of recomputing them every time, it stores intermediate results, called keys and values, in the KV cache. That makes generation fast, but it uses memory.
The cache grows with the context length and with the number of simultaneous users, so a long context or many users can take as much memory as the weights themselves. Engines such as vLLM manage the cache with techniques like PagedAttention, and some model designs are built to keep it small.
Related terms
- Context window The context window is the maximum amount of text, measured in tokens, that a model can take into account at one time, including your prompt and its own answer.
- VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.
- Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.