Speculative decoding is a technique used to speed up the generation of text by Large Language Models (LLMs). In a standard decoding process, a model generates one token at a time, which is a 'memory-bound' operation. This means the GPU spends more time waiting for model parameters to load from global memory into local memory than it does actually performing the mathematical calculations (floating point operations, or FLOPs). Because the GPU is not fully utilized during these steps, there is 'wasted' compute capacity available.
Speculative decoding addresses this by using a small, fast 'draft' model to propose a sequence of several tokens at once. A larger, more capable model then verifies this entire sequence in a single forward pass. While this actually increases the total number of FLOPs executed—since the large model must process the draft tokens—it changes the nature of the work. By processing multiple tokens simultaneously, the operation becomes 'compute-bound,' where the GPU's processing power is fully engaged. At low batch sizes (where the system is not already saturated with multiple simultaneous requests), this allows the model to complete a single sequence much faster by utilizing the 'free' compute that would otherwise sit idle.
This figure illustrates the core mechanism of speculative decoding where a large model verifies multiple draft tokens simultaneously.
However, this efficiency is not absolute and depends heavily on the 'acceptance rate' of the draft tokens. Every rejected token represents wasted work, as the large model spent compute cycles on a proposal that was ultimately discarded. This creates a trade-off between 'breadth' (processing more sequences at once) and 'depth' (speculating more tokens for a single sequence). At high throughput, if the batch size is large enough to saturate the GPU's compute, speculative decoding can become counterproductive. This is why some serving frameworks, such as vLLM, provide options to disable speculative decoding automatically when the batch size reaches a certain threshold. Research into dynamic speculation aims to solve this by adjusting the 'depth' of speculation per sequence based on the current available compute, rather than using a fixed number of tokens for every step. (Source: https://jbarrow.ai/2026-10-08-why-is-speculative-decoding-fast/)
This image demonstrates dynamic speculation, where the depth of speculation is adjusted per sequence to optimize throughput.
The underlying principle of trading memory-bound latency for compute-bound throughput is also observed in other fields, such as robotics. In Vision-Language-Action (VLA) models, 'action chunking' is used to group multiple future actions into a single inference step. This helps overcome the slow latency of high-dimensional models, ensuring that the robot can execute a coherent sequence of movements rather than reacting to single, isolated points in time. (Source: https://news.ycombinator.com/item?id=50152539)
Ultimately, the speed of speculative decoding is a matter of hardware orchestration. By aligning the workload with the GPU's strengths—performing more calculations while the data is already loaded—the system achieves faster results even though the total mathematical effort required to produce the final text has increased.