Context window
The context window is the maximum amount of text, measured in tokens, that a model can take into account at one time, including your prompt and its own answer.
A 128K window holds about 128,000 tokens, roughly a 300-page book, and a 1M window is eight times that. Everything you want the model to use, such as documents, chat history and instructions, has to fit inside, together with the answer it writes.
Longer windows help with large documents and codebases, but they cost memory. The KV cache that stores the conversation grows with the number of tokens, so using a very long context on your own machine needs far more memory than the weights alone. The advertised window is also a maximum: quality can drop on very long inputs, so test with your own material.
Related terms
- Token A token is a small chunk of text, often a word or part of a word, that a language model reads and writes; models measure length and speed in tokens.
- KV cache The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.
- VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.