Better and Faster Large Language Models via Multi-Token Prediction
Researchers are experimenting with “multi-token prediction” for large language models, training them to predict several future tokens at once and optionally using extra output heads at inference time to speed up generation without sacrificing quality. Commenters probe how this interacts with grammar, sampling, and joint probability over sequences, relate it to speculative decoding and hierarchical or semantic representations, and note trade‑offs where auxiliary multi-token objectives can help smaller models but sometimes hurt larger ones. The exchange also surfaces a broader need for clearer explanations of modern ML concepts, as rapid advances in architectures and decoding strategies outpace accessible, high-level documentation.
Multi-token prediction and grammaticality
- Clarification that during training the model predicts several future tokens, but at inference it usually uses only the standard next-token head.
- Optional use of extra heads for speculative decoding: predict multiple tokens, then verify and discard mismatches so quality matches ordinary decoding.
- Concern that independently predicting token n+1 and n+2 could break grammar; response is that verification/rejection and standard sampling keep outputs consistent.
Sentence-level / semantic vectors vs token prediction
- Several commenters wonder why we don’t predict sentence- or “concept”-level vectors and then decode to text.
- Issues raised: need a second model to map back to text; embeddings are lossy; inverse mapping is many-to-one and may produce invalid language.
- Some link this to encoder–decoder or autoencoder ideas and note that similar architectures have been tried; main difficulty is focusing on semantics when supervision is in token space.
- Related idea: hierarchical prediction (paragraph style → sentence form → words), but interactions across levels are hard.
Planning vs pure next-token prediction
- Debate over whether LLMs “plan” ahead or just locally optimize next tokens.
- Some argue that if transformers did not implicitly encode likely future structure, they’d behave like Markov models and perform far worse.
- Others insist that mathematically the model only computes P(next token | past tokens); any appearance of long-term planning is emergent from sequential conditioning and training data statistics.
- Examples discussed: grammar like “a/an”, rhyming, and the ability to end a paragraph on a specified word.
Speculative decoding and self-speculation
- Multi-token prediction here is viewed as a built-in draft model for self-speculative decoding, reducing latency by up to ~3× without degrading quality.
- Emphasis that speculative decoding, when implemented correctly, samples from the same distribution as standard autoregressive decoding; differences are due only to randomness.
Joint vs single-token probabilities
- One subthread analyzes how predicting only the next token can diverge from the most likely multi-token sequence.
- Suggestions include training to predict further-ahead tokens (nth token) to better approximate joint distributions, though practical methods remain unclear.
Meta: scaling and trade-offs
- Reports that multi-token prediction helps at some scales but can hurt at larger scales or higher numbers of predicted tokens.
- Noted general confusion in the ecosystem (terminology like “heads,” rapidly changing practices) and recommendations for educational resources to understand the broader pipeline.