gg2
Continuous Diffusion Language Models
Archived — this story has rotated out of today’s deck. It is kept here in full.
The gist
Diffusion language models generate text by refining full drafts, not token-by-token. They rival autoregressive LLMs, with releases from Google, NVIDIA, and Inception Labs.
Background
Diffusion language models (DLMs) are an alternative to autoregressive models like GPT. Instead of generating text left-to-right, they start with a rough draft and iteratively refine all positions in parallel, allowing error correction and faster generation. In 2024, they became competitive with autoregressive models, and by 2026, major labs like Google, NVIDIA, and Inception Labs have released diffusion LLMs.
How it unfolded
- 2024Diffusion models became competitive with autoregressive models on quality for language generation.
- May 2026ByteDance open-sourced the continuous latent space diffusion language model Cola DLM.
- Jun 2026Google DeepMind released the diffusion text model Diffusion Gemma with open source weights for the first time.
- Aug 2026By 2026, diffusion LLMs are a reality with releases from Mercury 2 (Inception Labs), Gemma Diffusion (Google), and Nemotron Diffusion (NVIDIA).
Who’s saying what
- Support
- Diffusion language models offer advantages like error correction, parallel generation, and bidirectional context, making them a promising alternative to autoregressive models.
- Caution
- The review notes that diffusion language models are still less mature than image diffusion, with unresolved questions about likelihood estimation and decoding, and sampling remains computationally expensive.