Speculative decoding and multi-token prediction
Speculative decoding speeds up text generation by letting a cheap draft guess several tokens ahead and having the full model check them all at once.
A language model normally writes one token at a time, and each token needs a full pass through the model. In speculative decoding a small draft model, or an extra prediction layer inside the model, proposes several next tokens. The main model then checks them in a single pass and keeps the ones it agrees with, so in standard implementations the output is what it would have written alone, only faster.
Many recent models ship a multi-token-prediction (MTP) layer for this, for example Hy3 and the DeepSeek V4 and GLM-5 families, and inference servers such as vLLM and SGLang can use it. It helps most on predictable text such as code and costs a little extra memory. Check whether your server supports a model's MTP layer, because support and settings differ between tools.
Related terms
- Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.
- KV cache The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.
- Token A token is a small chunk of text, often a word or part of a word, that a language model reads and writes; models measure length and speed in tokens.