Better and Faster Large Language Models via Multi-Token Prediction

Multi-token prediction and grammaticality

• Clarification that during training the model predicts several future tokens, but at inference it usually uses only the standard next-token head.
• Optional use of extra heads for speculative decoding: predict multiple tokens, then verify and discard mismatches so quality matches ordinary decoding.
• Concern that independently predicting token n+1 and n+2 could break grammar; response is that verification/rejection and standard sampling keep outputs consistent.

Sentence-level / semantic vectors vs token prediction

• Several commenters wonder why we don’t predict sentence- or “concept”-level vectors and then decode to text.
• Issues raised: need a second model to map back to text; embeddings are lossy; inverse mapping is many-to-one and may produce invalid language.
• Some link this to encoder–decoder or autoencoder ideas and note that similar architectures have been tried; main difficulty is focusing on semantics when supervision is in token space.
• Related idea: hierarchical prediction (paragraph style → sentence form → words), but interactions across levels are hard.

Planning vs pure next-token prediction

• Debate over whether LLMs “plan” ahead or just locally optimize next tokens.
• Some argue that if transformers did not implicitly encode likely future structure, they’d behave like Markov models and perform far worse.
• Others insist that mathematically the model only computes P(next token | past tokens); any appearance of long-term planning is emergent from sequential conditioning and training data statistics.
• Examples discussed: grammar like “a/an”, rhyming, and the ability to end a paragraph on a specified word.

Speculative decoding and self-speculation

• Multi-token prediction here is viewed as a built-in draft model for self-speculative decoding, reducing latency by up to ~3× without degrading quality.
• Emphasis that speculative decoding, when implemented correctly, samples from the same distribution as standard autoregressive decoding; differences are due only to randomness.

Joint vs single-token probabilities

• One subthread analyzes how predicting only the next token can diverge from the most likely multi-token sequence.
• Suggestions include training to predict further-ahead tokens (nth token) to better approximate joint distributions, though practical methods remain unclear.

Meta: scaling and trade-offs

• Reports that multi-token prediction helps at some scales but can hurt at larger scales or higher numbers of predicted tokens.
• Noted general confusion in the ecosystem (terminology like “heads,” rapidly changing practices) and recommendations for educational resources to understand the broader pipeline.