Efficient attention (sparse, linear and hybrid)

Efficient attention designs let a model read very long inputs with far less memory and compute than standard attention, by looking at fewer tokens or compressing the past.

  • Models and architecture
  • Updated

Standard attention compares every token with every earlier token, so the work grows with the square of the input length and the KV cache grows with every token. At a 1M-token context that is very expensive. Sparse attention lets each token look at a chosen subset of earlier tokens. Linear attention and state-space layers such as Mamba compress the past into a fixed-size state instead. Hybrid designs mix a few full-attention layers with many cheaper ones.

Recent open models use these ideas. DeepSeek V4, GLM 5.3 and MiniMax M3 use sparse attention, Kimi K3.1 Pro combines linear Kimi Delta Attention with sparse attention, Qwen3.5 and Ling 3.1 Flash use linear-attention designs, and NVIDIA's Nemotron 3 Ultra interleaves Mamba-2 layers with a few attention layers. The payoff is a smaller KV cache and cheaper long-context serving. The cost is support, since these models often need specific vLLM or SGLang builds, and quality on very long inputs should be tested on your own material.

Related terms

  • KV cache The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.
  • Context window The context window is the maximum amount of text, measured in tokens, that a model can take into account at one time, including your prompt and its own answer.
  • Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.
  • Mixture of experts (MoE) A mixture-of-experts model contains many specialist sub-networks and uses only a few of them for each token, so it can be very large yet fast.

All terms