Stop Thinking of LLMs as Next-Token Predictors

Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete. It’s a fine zeroth-order approximation, and it is grounded in something real: transformer-based language models emit tokens autoregressively:

while not done : tokens . append ( model . sample_next_token ( tokens ))

This certainly has the shape of something you might call a next-token predictor. During pre-training, the model repeatedly takes some prior tokens, looks at the token that actually followed them, and makes that token more likely to be sampled next. Conceptually, the training loop looks something like this:

for tokens in training_data : for position in range ( 1 , len ( tokens )): prior_tokens = tokens [: position ] actual_next_token = tokens [ position ] model . make_more_likely ( actual_next_token , after = prior_tokens , )

make_more_likely is, of course, doing a heroic amount of work here. Under the hood are loss functions, gradients, and parameter updates, but for this post we only care about their combined effect: the token that actually came next becomes more likely.

Crucially, every actual_next_token comes from an existing sequence in training_data . It’s probably fair to say that the base model also behaves as a next-token predictor: it is trained to predict next tokens as they occur in its training data.

But the LLMs we use are not just base models. They are post-trained, and a key part of modern post-training is reinforcement learning with verifiable rewards (RLVR). During pre-training, the model learns only from sequences that already exist in the training data. During RLVR, the model explores by generating new sequences and learning from their outcomes. Conceptually, the RLVR training loop looks something like this:

for task in training_tasks : for explored_tokens in model . explore ( task ): reward = evaluate_outcome ( task , explored_tokens ) for position in range ( len ( explored_tokens )): prior_tokens = task + explored_tokens [: position ] explored_next_token = explored_tokens [ position ] model . make_more_likely ( explored_next_token , after = prior_tokens , according_to = reward , )

make_more_likely is doing the same kind of work in both loops, but for a fundamentally different reason. During pre-training, it makes an actual_next_token more likely because that token appeared in the training data. During RLVR, it makes an explored_next_token more likely because the explored sequence containing it earned a high reward.

... continue reading