Speculative Decoding
A small model drafts several tokens and a large model verifies them in parallel to speed up generation.
In plain language
Large models generate one token at a time. Speculative decoding has a cheaper draft model guess several tokens, then the large model verifies them in one forward pass; accepted tokens stay and rejected ones restart at the divergence. It reduces latency without changing the output distribution, making it useful for high-concurrency, latency-sensitive serving. Performance depends on draft-target agreement; a poor draft causes frequent rejection and extra overhead.
Example
A support bot uses a 1B model to draft and a 70B model to verify in batches, reducing time-to-first-token while preserving quality.
Recommended tools
Related terms
Inference
The computation a trained model performs to produce output from an input.
Latency
The time from sending a request to receiving the model response.
Model Distillation
Training a smaller student model on a larger teacher model's outputs to compress capability.
Agent Memory
A mechanism for an agent to store and retrieve user preferences, past tasks and key facts across sessions.