Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a f
Soft-MTP removes a noisy gradient residual from multi-token prediction via a straight-through KL recovery trick, improving perplexity, speculative decoding, and reasoning without added cost.
Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard MTP training, in which a residual component of the MTP gradient interferes with both next-token and multi-token learning, increasingly so with scale. Specifically, the MTP cross-entropy gradient can be decomposed exactly into a KL self-distillation term that pulls each MTP head toward the model's own next-token prediction (NTP) distribution and improves speculative decoding, and a residual that has a low signal-to-noise ratio and buries the useful KL signal. We introduce the KL…
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Multi-Token Prediction Done Right was first seen on Hugging Face · Daily papers on Oct 30, 2026. Since then it has appeared in 38 checks on 4 different days. Its best position was #10 of 30, reached on Nov 2, 13:28 UTC. It dropped off the list after Nov 2, 17:28 UTC.
Position on Hugging Face · Daily papers at every check (lower is better).
Best rank per day (UTC).
Platform metric as reported by Hugging Face.
Archived daily, weekly and monthly lists that include this entry.
Not at the moment. It was last on the list on Nov 2, 17:28 UTC, after 4 days of trending.
#10 on Hugging Face · Daily papers, on Nov 2, 13:28 UTC. Trendoday checks this list every 120 minutes.
It was first seen on Oct 30, 2026 and has been on the list on 4 days (38 checks) since.