This story on HackerNoon has a decentralized backup on Sia.
Transaction ID: NeWTQHyWGjzVUVfYJTM9e2E-P4iP7GffUNADnqrizY8
Cover

Redefining Induction: Multi-Token vs. Next-Token on High-Quality LLM Data

Written by @cosmological | Published on 2025/7/23
TL;DR —
This figure showcases how training on higher-quality data enforces early induction capability, demonstrating that multi-token prediction's advantage in this task diminishes for larger models as feature learning transforms it into a next-token problem.

Abstract and 1. Introduction

2. Method

3. Experiments on real data

4. Ablations on synthetic data

5. Why does it work? Some speculation

6. Related work

7. Conclusion, Impact statement, Environmental impact, Acknowledgements and References

A. Additional results on self-speculative decoding

B. Alternative architectures

C. Training speeds

D. Finetuning

E. Additional results on model scaling behavior

F. Details on CodeContests finetuning

G. Additional results on natural language benchmarks

H. Additional results on abstractive text summarization

I. Additional results on mathematical reasoning in natural language

J. Additional results on induction learning

K. Additional results on algorithmic reasoning

L. Additional intuitions on multi-token prediction

M. Training hyperparameters

J. Additional results on induction learning

Figure S14: Induction capability of n-token prediction models trained on higher-quality data. Shown is accuracy on the second token of two token names that have already been mentioned previously. Training on a 9:1 mix of a books dataset and the children storiy dataset, we observe that induction capability forms significantly earlier in training (not shown here) and to a higher degree. We believe that this is explained both because our evaluation dataset no longer contains out-of-distribution tokens (Section 4.1) and because the higher-quality data contained in the books dataset makes induction necessary earlier on (especially for small models, cf. Singh et al. (2023)). In particular, by enforcing the formation of induction capability in the model by means of the dataset – instead of the loss – the advantage of 2-token prediction models on this task disappears except for the smallest models: feature learning converts the task into a pure next-token prediction task.

Authors:

(1) Fabian Gloeckle, FAIR at Meta, CERMICS Ecole des Ponts ParisTech and Equal contribution;

(2) Badr Youbi Idrissi, FAIR at Meta, LISN Université Paris-Saclayand and Equal contribution;

(3) Baptiste Rozière, FAIR at Meta;

(4) David Lopez-Paz, FAIR at Meta and a last author;

(5) Gabriel Synnaeve, FAIR at Meta and a last author.

This paper is available on arxiv under CC BY 4.0 DEED license.

[story continues]

Written by
@cosmological
From Big Bang's singularity to galaxies' cosmic dance the universe unfolds its majestic tapestry of space and time.
Topics and
tags
llm-generalization|next-token-task|ai-training|deep-learning-insights|data-impact|induction-capability|high-quality-data|multi-token-prediction
This story on HackerNoon has a decentralized backup on Sia.
Transaction ID: NeWTQHyWGjzVUVfYJTM9e2E-P4iP7GffUNADnqrizY8