This story on HackerNoon has a decentralized backup on Sia.
Transaction ID: DdYIwf73VqMZ5rl48lww7-gFEBAiJMubQTkY8n_2Vjc
Cover

Unleashing LLM Training Efficiency: Multi-Token Prediction's Near-Zero Overhead

Written by @cosmological | Published on 2025/7/22
TL;DR —
Explore Table S5 revealing multi-token prediction's remarkable training efficiency across LLM sizes (0.3B-13B), showing minimal overhead relative to next-token prediction—a solvable issue for even faster future training.

Abstract and 1. Introduction

2. Method

3. Experiments on real data

4. Ablations on synthetic data

5. Why does it work? Some speculation

6. Related work

7. Conclusion, Impact statement, Environmental impact, Acknowledgements and References

A. Additional results on self-speculative decoding

B. Alternative architectures

C. Training speeds

D. Finetuning

E. Additional results on model scaling behavior

F. Details on CodeContests finetuning

G. Additional results on natural language benchmarks

H. Additional results on abstractive text summarization

I. Additional results on mathematical reasoning in natural language

J. Additional results on induction learning

K. Additional results on algorithmic reasoning

L. Additional intuitions on multi-token prediction

M. Training hyperparameters

C. Training speeds

Table S5: Training time relative to next-token prediction training. The slight overhead when using multi-token prediction here is explained by a suboptimal use of Fully Sharded Data Parallel. In our implementation, when doing separate backward passes for each head, we lose the overlap of layer weight communication and computation, therefore it incurs a very slight overhead that can be removed if reimplemented correctly.

Authors:

(1) Fabian Gloeckle, FAIR at Meta, CERMICS Ecole des Ponts ParisTech and Equal contribution;

(2) Badr Youbi Idrissi, FAIR at Meta, LISN Université Paris-Saclayand and Equal contribution;

(3) Baptiste Rozière, FAIR at Meta;

(4) David Lopez-Paz, FAIR at Meta and a last author;

(5) Gabriel Synnaeve, FAIR at Meta and a last author.

This paper is available on arxiv under CC BY 4.0 DEED license.

[story continues]

Written by
@cosmological
From Big Bang's singularity to galaxies' cosmic dance the universe unfolds its majestic tapestry of space and time.
Topics and
tags
multi-token-prediction|llm-training|training-efficiency|computational-overhead|next-token-prediction|model-scalability|fsdp|deep-learning-optimization
This story on HackerNoon has a decentralized backup on Sia.
Transaction ID: DdYIwf73VqMZ5rl48lww7-gFEBAiJMubQTkY8n_2Vjc