GoKawiilAn introduction to diffusion language models and the research advances that underlie today's diffusion LLMs. We describe the building blocks of recent open-source models, starting from simple masking diffusion, and including techniques for iterative refinement, post-training, and variable-length generation. Material is adapted from workshop talks and lectures at ICLR 2026 and MLSS 2026 .
Introduction: Autoregressive and Diffusion Language Models
Two families of generative AI algorithms are widely used today. For continuous data such as images or video, the state-of-the-art approach is based on diffusion models. For discrete data such as text or code, the standard approach is instead autoregressive models. This article explores an alternative for discrete data, one built on the modern paradigm of diffusion.
Mainstream language models are autoregressive: they generate tokens left-to-right, one at a time, each conditioned on the tokens before it. This approach is powerful, but it also has inherent limitations:
No error correction : once a token is emitted it cannot be revised, so early mistakes compound.
: once a token is emitted it cannot be revised, so early mistakes compound. Generation is slow : producing a sequence takes as many steps as there are tokens, and does not naturally lend itself to fast, parallel generation.
: producing a sequence takes as many steps as there are tokens, and does not naturally lend itself to fast, parallel generation. Causal attention: generation only ever looks backward, never at future context.
Diffusion models take a different approach. Rather than producing text one token at a time, they generate the whole sequence at once, starting from an initial guess and iteratively refining it over a number of steps. This unlocks several advantages: generation can trade off speed and quality by using fewer or more steps, mistakes can be corrected along the way, and every step attends to bidirectional context.
Autoregressive LLMs generate one token at a time, left to right, taking as many steps as there are tokens (top). Diffusion LLMs -- such as Gemma Diffusion shown here -- instead start from a rough, full-length draft and refine every position in parallel over a few rounds (bottom), rewriting the whole sequence at each step rather than emitting a single token. Figure credit: M. Grootendorst & Gemma Diffusion.
Applying diffusion to language had long been an open problem. In 2024 the field reached a turning point, as diffusion models became competitive with autoregressive models on quality. By 2026, diffusion LLMs are a reality, with releases from leading industry labs — Mercury 2 (Inception Labs) , Gemma Diffusion (Google) , and Nemotron Diffusion (NVIDIA) . This article traces the ideas and papers that underlie these modern models.
... continue reading