Researchers have identified a fundamental flaw in standard Multi-Token Prediction (MTP) training where a 'residual' component of the gradient interferes with both next-token and multi-token learning, a problem that worsens as models scale. By introducing Soft-MTP, a one-line code change, they can isolate a useful KL self-distillation signal from this noise, resulting in lower perplexity and faster speculative decoding across various model sizes.
The Flaw in Standard MTP Training
Multi-Token Prediction (MTP) is a common technique used to improve downstream performance and enable faster inference through speculative decoding. However, standard MTP training suffers from a gradient decomposition issue: the cross-entropy gradient contains a useful KL self-distillation term and a residual with a low signal-to-noise ratio.
This noisy residual buries the useful KL signal, hindering the model's ability to learn effectively. The interference becomes increasingly problematic as the model size increases, creating a bottleneck for scaling MTP capabilities.
“Soft-MTP is, in effect, a free lunch for MTP optimization.”
— Multi-Token Prediction Done Right
The KL Gradient-Recovery Trick
Soft-MTP addresses this by implementing a 'KL gradient-recovery trick.' When two distributions share a linear output head, the gradient of their KL divergence can be recovered exactly by combining their cross-entropy gradients through straight-through gradient routing.
This figure illustrates the core mechanism of Soft-MTP, specifically how it handles gradient routing to improve multi-token prediction training.
This approach is computationally efficient because it does not require the massive vocabulary-sized logit matrices typically needed for direct KL divergence minimization. It is compatible with standard fused cross-entropy kernels, making it a 'free lunch' for optimization.
The implementation is a single function change that supports both DeepSeek- and Meta-style MTP architectures. It reuses existing NTP and MTP cross-entropy computations to backpropagate only the desired KL self-distillation signal.
Performance Gains and Trade-offs
Testing across models ranging from 167M to 1.2B parameters shows that Soft-MTP consistently outperforms standard MTP. Key improvements include lower next-token prediction (NTP) perplexity and higher zero-shot accuracy across all tested scales.
For larger models (689M parameters and above), Soft-MTP also achieves lower MTP perplexity. Furthermore, the method demonstrates superior math and code reasoning capabilities following fine-tuning.
While Soft-MTP matches or exceeds direct KL divergence minimization on most metrics, the specific magnitude of the 'noisy residual' interference and the exact values for all metrics are not fully detailed in the primary report.