Contents
Introduction: Autoregressive and Diffusion Language Models
Two families of generative AI algorithms are widely used today. For continuous data such as images or video, the state-of-the-art approach is based on diffusion models. For discrete data such as text or code, the standard approach is instead autoregressive models. This article explores an alternative for discrete data, one built on the modern paradigm of diffusion.
Mainstream language models are autoregressive: they generate tokens left-to-right, one at a time, each conditioned on the tokens before it. This approach is powerful, but it also has inherent limitations:
- No error correction: once a token is emitted it cannot be revised, so early mistakes compound.
- Generation is slow: producing a sequence takes as many steps as there are tokens, and does not naturally lend itself to fast, parallel generation.
- Causal attention: generation only ever looks backward, never at future context.
Diffusion models take a different approach. Rather than producing text one token at a time, they generate the whole sequence at once, starting from an initial guess and iteratively refining it over a number of steps. This unlocks several advantages: generation can trade off speed and quality by using fewer or more steps, mistakes can be corrected along the way, and every step attends to bidirectional context.
Autoregressive LLMs generate one token at a time, left to right, taking as many steps as there are tokens (top). Diffusion LLMs -- such as Gemma Diffusion shown here -- instead start from a rough, full-length draft and refine every position in parallel over a few rounds (bottom), rewriting the whole sequence at each step rather than emitting a single token. Figure credit: M. Grootendorst & Gemma Diffusion.
Applying diffusion to language had long been an open problem. In 2024 the field reached a turning point, as diffusion models became competitive with autoregressive models on quality. By 2026, diffusion LLMs are a reality, with releases from leading industry labs — Mercury 2 (Inception Labs) , Gemma Diffusion (Google) , and Nemotron Diffusion (NVIDIA) . This article traces the ideas and papers that underlie these modern models.
Background: Gaussian Diffusion
Before introducing diffusion for language, we start with a brief overview of Gaussian diffusion for image generation. We will then build up discrete diffusion by analogy.
Generating by iterative denoising
The central concept underlying diffusion models is denoising. Instead of painting an image in one shot, a diffusion model produces images step by step, starting from pure random noise and removing a little of it at every step until a coherent image emerges. Generating an image through many small steps turns out to be far simpler than producing it all at once, and this is what makes diffusion models so effective.
How does a model learn to denoise? The trick is to teach it by showing examples of noise being gradually transformed into an image. Diffusion achieves this via two complementary processes. First, a forward process takes a clean source image and turns it into pure noise, one step at a time. Second, a reverse process learns to invert this transformation, turning pure noise back into an image; it is trained on the image-to-noise trajectories produced by the forward process.
Forward process
The forward process takes a clean training image and produces a sequence of increasingly noisy images that trace a path from clean data to pure noise. It does this by mixing in a growing amount of random Gaussian noise at each step, until the image dissolves into pure static. This step requires no learning at all — we are simply adding noise — yet it is enormously useful, because it manufactures an endless supply of training data: examples of images being transformed into noise, and vice versa.
A Gaussian diffusion trajectory on an image. Reading left to right, the forward process gradually adds noise until a clean photo of a dog dissolves into pure static. This trajectory will serve as training data for the reverse process.
Reverse process
The reverse process is where the actual learning happens. We train a model to transform noise into images by following the steps produced by the forward process in reverse.
Reverse path for Gaussian diffusion. Again going left to right, the generative reverse process is trained to reproduce the trajectory from the forward process in reverse, starting with noise, and reconstructing the original image.
Concretely, given a noisy image, we train a machine learning model to separate the noise from the underlying image or, equivalently, to predict either the noise that was added or the clean image itself, since given the noisy input, knowing one determines the other. Once the model can do this, generation is simple: start from pure noise, ask the model to estimate and strip away a bit of it, and repeat. Each pass nudges the sample a little closer to something that looks like real data, until a clean image remains.
The two processes that define diffusion. The forward process (top, left to right) turns clean data $x_0$ into complete noise $x_T$ by adding a little Gaussian noise at each step; the generative reverse process (bottom, right to left) is trained on this data to denoise $x_T$ back to $x_0$ using the same sequence of steps. After training on a sufficiently large set of trajectories, the model learns to generalize and generates new images starting from random samples of white noise.
This forward/reverse recipe — corrupt data with noise, then learn to reverse the corruption one step at a time — is the blueprint for every diffusion model.
Simple Masked Diffusion Models
The main obstacle in bringing diffusion to language is deciding what "noise" should mean for discrete tokens. For example, the noise used in classical diffusion is Gaussian, and adding continuous Gaussian noise to categorical variables is not well-defined. Below we introduce one simple yet effective approach that defines noise via masking. Our group popularized this approach, and it now forms the basis of most open-source diffusion language models.
Masked Diffusion in a Nutshell
The easiest way to understand masked diffusion is as an unmasking transformer. We train the model by taking clean sequences, masking a random fraction of their tokens, and asking a bidirectional transformer to fill in the blanks. If you know BERT, this is essentially BERT with a randomized masking rate — but unlike BERT, the resulting model is generative. You can think of masked diffusion as a generative BERT.
Training masked diffusion as an unmasking transformer. The forward masking process samples a random noise level $0 < t < 1$ and hides that fraction of the tokens in a clean datapoint $x$, producing a partially masked $z_t$; the model $x_\theta$ is then trained to reconstruct the original tokens. Figure credit: Sasha Rush.
Once we trained the unmasking transformer, we can generate text by starting from a fully masked sequence and repeating two steps many times:
- Infilling: Ask the model to fill in every blank in the current sequence, yielding a rough guess of the clean tokens.
- Remasking: Randomly re-noise the infilled sequence by replacing tokens with masks, but keep a few more tokens unmasked than in the previous round.
One sampling step. A denoising model fills in the masked positions of the current sequence $z_t$, then a random subset of those fresh predictions is re-masked to form $z_s$, the slightly-less-masked sequence at the next step. Iterating this fills the sequence in an arbitrary order. Figure credit: Sasha Rush.
Each round leaves fewer positions masked, until the sequence converges to a clean sample from the model. Generation thus amounts to starting from a sequence full of blanks and gradually filling in words in an arbitrary order.
Sampling in action on the opening of One Hundred Years of Solitude. Early in generation only some tokens have been filled in; the many gaps (and the masked cells in the bar below) mark positions that are still blank.
A few rounds later, most positions have been filled and only a handful of masked tokens remain; the passage is already largely legible.
At convergence every position is unmasked, yielding a clean sample — the complete, coherent opening passage.
Understanding Masked Diffusion: A Probabilistic Perspective
We can also understand a bit better why this process works by framing it as an analog of the Gaussian diffusion model we saw earlier. Just like Gaussian diffusion, masked diffusion can be described as a model consisting of a forward and a reverse process.
Forward process
The goal of the forward process is to generate training data for the reverse process. Its output is a trajectory that starts from a datapoint and ends at a sequence of pure noise; the reverse process will then be trained to produce this trajectory in reverse.
The key challenge is deciding what "noisy" should mean. In Gaussian diffusion, we added varying amounts of white noise to an image. In masked diffusion, we instead randomly mask a fraction of the tokens in a discrete sequence. The amount of masking is governed by a schedule $\alpha_t$ — the probability that a given token remains unmasked — which plays the role of the signal-to-noise ratio in Gaussian diffusion. It starts at $1$ when $t = 0$ (a clean sequence) and decreases to $0$ when $t = 1$ (a fully masked sequence). The time variable $t$ indexes a path from clean to noisy data, and at time $t$ a partially masked sequence $z_t$ has, in expectation, a fraction $\alpha_t$ of its tokens unmasked.
The masked-diffusion forward process. As the signal level $\alpha_t$ decreases from $1$ to $0$, each token of the clean sequence $x$ is masked with probability $1 - \alpha_t$, interpolating from the clean datapoint to a partially masked $z_t$ and finally to a fully masked sequence. Figure credit: Sasha Rush.
We implement this process as a Markov chain over a sequence of variables $z_t$ indexed by $t$, with $z_0$ being the clean, unmasked sequence. For $s < t$, the chain defines $q(z_t \mid z_s)$ by masking each still-unmasked token of $z_s$ with probability $(\alpha_s - \alpha_t)/\alpha_s$. Running this Markov chain for a number of steps produces a trajectory going from clean data to fully masked noise.
The forward process as a Markov chain over increasingly masked latents. Moving from $z_s$ to $z_t$ (with $s < t$), each still-unmasked token is masked with a schedule-dependent probability, defining the transition $q(z_t \mid z_s)$ that carries the sequence toward a fully masked state. Figure credit: Sasha Rush.
Reverse process
Next, as in Gaussian diffusion, we train the reverse process to walk the sequence of increasingly masked latents in reverse — starting from a fully masked sequence and ultimately generating outputs similar to clean data.
Using Bayes' rule, we can derive the mathematically optimal reverse process $q(z_s \mid z_t, x)$ when the clean sequence $x$ is known . This optimal process has two steps: (1) given a partially masked $z_t$, we peek at $x$ to find the true clean tokens; (2) form $z_s$ by replacing each masked position of $z_t$ with its value in $x$ with probability $(\alpha_s - \alpha_t) / (1 - \alpha_t)$, and otherwise leaving it masked.
In practice, the final output $x$ is obviously unknown when we generate it. We therefore train a model $x_\theta(z_t)$ to predict the final clean sequence given the current state $z_t$ and apply the ideal reverse process $q(z_s \mid z_t, x)$ using the estimate $x_\theta(z_t)$ in place of the real $x$. More formally, we define the reverse process as a probability $p(z_s \mid z_t) = q\big(z_s \mid z_t, x_\theta(z_t)\big)$. This definition recovers the sampling algorithm we described earlier: at each step, we use the model $x_\theta(z_t)$ to fill in the blanks of $z_t$, and we keep a subset of these filled-in tokens in $z_s$.
The reverse process. Using the denoising model's prediction $x_\theta(z_t)$ of the clean sequence, each masked token of $z_t$ is unmasked with probability $(\alpha_s - \alpha_t)/(1 - \alpha_t)$. This mirrors the optimal unmasking posterior $q(z_s \mid z_t, x)$ that one would use if the true clean sequence $x$ were known. Figure credit: Sasha Rush.
A probabilistic model for masked diffusion
Putting these pieces together gives us the mathematical definition of a masked diffusion language model (MDLM). The forward process $q(z_t \mid z_s)$ produces a trajectory from clean to fully masked data, and the reverse process $p(z_s \mid z_t)$ learns to undo it. Moreover, the reverse process defines a latent variable model $p(x, z_1, \dots, z_T)$ in which $T$ intermediate partially masked samples $z_1,...,z_T$ are latent variables. Generating from the reverse process $p(z_s \mid z_t)$ is the same as performing ancestral sampling from this model.
The forward masking process $q(z_t \mid z_s)$ and the learned reverse process $p(z_s \mid z_t, x_\theta)$ jointly define a latent-variable model over $(x, z_1, \dots, z_T)$ — a path of increasingly masked latents connecting the clean sequence $x$ to a fully masked sequence. Figure credit: Sasha Rush.
We can also look at the likelihood $\log p(x)$ of the model $p$ to assess its quality. In latent variable models this is intractable, so we resort to approximations via variational inference. For a masked diffusion language model, the evidence lower bound (ELBO) used to approximate the likelihood has a surprisingly simple form (assuming for simplicity $\alpha_t = 1-t$) :
\[\mathcal{L}_\text{ELBO} = \mathbb{E}_{t \sim \mathcal{U}[0,1]}\; \mathbb{E}_{q(z_t \mid x)} \left[ \frac{1}{t}\, \log p_\theta\!\left(x \mid z_t\right) \right].\]
Let's unpack this formula. The inner term $\log p_\theta(x \mid z_t)$ is the likelihood of a clean sequence $x$ given a partially masked sequence $z_t$ sampled from the forward process. In other words, it is the cross-entropy loss between the predictions of our unmasking transformer and the true tokens — this is exactly the BERT loss!
Differently from BERT, this loss is averaged over all $t$, and hence over all possible masking rates, rather than a single fixed one. It is also normalized by $t$, the expected fraction of tokens that are masked (since $\alpha_t = 1-t$); this factor ensures that each BERT loss is normalized for the number of tokens over which the loss is taken.
Summarizing and Evaluating Masked Diffusion
In summary, MDLM is very similar to BERT, with two key differences:
- It admits principled sampling algorithms, corresponding to ancestral sampling in a latent variable model.
- Training uses a randomized masking rate, which corresponds to a principled variational lower bound on the log-likelihood.
Most interestingly, the evidence lower bound enables a principled comparison between autoregressive and diffusion language models using log-likelihood (or, equivalently, perplexity) — the standard metric for evaluating language models. While for a long time there was a substantial gap in perplexity between diffusion and autoregressive language models, simplified masked diffusion models were among the first to close much of this gap .
Test perplexity (lower is better) on the LM1B benchmark. Simplified masked diffusion (MDLM) narrows the gap to autoregressive models (dashed line), improving over earlier discrete-diffusion methods such as Diffusion-LM, D3PM, DiffusionBERT, and SEDD. Results taken from the MDLM paper.
Building a Real-World Diffusion Language Model
As defined above, masked diffusion models are helpful for building intuition, but they are not production-ready: they generate only fixed-length sequences, they do not support iterative refinement (error correction) out of the box, and they are not especially fast without additional post-training. The rest of this article explores extensions that address these limitations, in the context of modern open-weights diffusion models.
Block Diffusion for Flexible-Length Generation
The first issue that arises with standard MDLMs is their limitation to generating fixed-length sequences. Block diffusion addresses this limitation by performing diffusion over blocks, conditioned on previously generated tokens . These blocks can be of arbitrary size, ranging from a dozen to thousands of tokens, and should ideally depend on the application domain.
Block diffusion, as illustrated by Gemma Diffusion. Conditioned on the input prompt, the model diffuses one block ("canvas") of tokens at a time (here 256 tokens per block), appending blocks left to right until the full 1024-token sequence is complete. Earlier blocks are cached and conditioned on, much like autoregressive KV caching. Figure credit: M. Grootendorst.
For example, in biological applications, we might have prior knowledge about the length of the interactions we want to capture, and set the block size to the minimum length needed to capture them. In language modeling, we may instead be interested in maximizing GPU utilization; in that case we would choose the block size so that the arithmetic intensity of our forward pass (which also depends on the batch size) matches that of the underlying hardware.
Additionally, block diffusion naturally supports KV caching, a technique that accelerates sequence generation in autoregressive models. Once a block has been generated using a transformer architecture, its keys and values can be cached and reused when generating future blocks.
Other approaches to variable-length generation rely on connections between masked diffusion models and any-order autoregressive models. For instance, Set Diffusion extends block diffusion to operate over arbitrary sets of positions rather than left-to-right blocks. Other approaches, such as Edit Flows or FlexMDM instead model generation as a sequence of insertion, deletion, and substitution operations, which lets the model grow or shrink the sequence as it refines it.
Architectures: Encoder, Decoder, and Encoder–Decoder
Standard masked diffusion models are effectively encoder-only (like BERT), in contrast to decoder-only autoregressive models (like GPT). Using an encoder-only architecture requires sampling algorithms that invoke the full network at every denoising step, which can incur a relatively high computational cost.
A key insight is that diffusion performs two kinds of computation: (1) computing a representation of the tokens that have been generated so far, and (2) denoising the corrupted tokens. This observation suggests using separate modules for each task. The result is an encoder–decoder architecture, which relies on an encoder to represent clean tokens and a lightweight decoder to iteratively refine a noised sequence. Encoder–decoder architectures are at the core of state-of-the-art open-source diffusion LLMs, such as Gemma Diffusion and the recent Nemotron Diffusion models .
An encoder–decoder diffusion architecture, as used in Gemma Diffusion. A heavy encoder processes the clean input query (the already-generated context) once, while a denoiser iteratively de-noises the masked "canvas", conditioning on the encoder's representation, to produce the final tokens. Figure credit: M. Grootendorst.
In addition to accelerating discrete diffusion inference, this architecture enables faster training of block diffusion models: after partitioning a sequence into blocks, we pass the blocks into a smaller decoder at training time, which reduces the number of FLOPs needed for training.
Iterative Refinement and Built-In Error Correction
Part of the appeal of diffusion is iterative refinement. Standard MDLMs lack this: once a token is unmasked it can never be updated, because the forward masking process never remasks an unmasked token, so the model never learns to correct itself. Modern diffusion LMs modify the forward and reverse processes to restore this capability.
Standard masked diffusion cannot correct itself. Because the forward masking path never re-masks a token once it is unmasked, the reverse model $p(z_s \mid z_t, x_\theta)$ is never shown examples of revising a committed token — so a mistake made early in generation can never be fixed.
Remasking Diffusion
The simplest fix is remasking: at each step we keep some newly unmasked tokens but also re-mask a small subset of previously unmasked tokens, letting them be regenerated. Concretely, consider the example below, in which a masked diffusion model introduces a grammatical error. With remasking, this token flips to a mask and then gets corrected when the model receives additional context.
Remasking enables error correction. Reading from $t=1$ (fully masked) up to $t=0$, the model first commits an ungrammatical token ("sell", red) because it is a plausible completion; a later remasking step flips it back to a mask and, with more surrounding context, corrects it to "sells" (green).
Remasking can be applied to standard pretrained MDLMs in a principled way as a plug-in sampler (formally, it can be seen as implementing a predictor–corrector Markov chain ). Most interestingly, it endows discrete diffusion with a form of inference-time compute scaling: increasing the number of sampling steps lets remasking approach autoregressive quality, while under a tight compute budget it better preserves quality than plain MDLM sampling .
Sample quality (MAUVE, higher is better) versus sampling compute. Remasking diffusion (ReMDM) with more sampling steps ($T = 1024 \to 4096$) raises quality from 0.40 to 0.66, closing much of the gap to autoregressive generation (dashed line, 0.76) and surpassing plain MDLM (0.04) and earlier variants. Results from the ReMDM paper.
Uniform State Diffusion
Alternatively, we may use an entirely different type of forward and reverse process than masking. Uniform state diffusion is perhaps the most common alternative discrete form of noise. Instead of masking, its forward process replaces tokens with random ones in the vocabulary. The reverse process starts with a random sequence and flips tokens until the result looks like data.
Two discrete noise processes, both corrupting "The cat sat on the mat". In absorbing-state (masked) diffusion (top), tokens are progressively replaced by [MASK]. In uniform-state diffusion (bottom), tokens are instead replaced by random vocabulary tokens, so intermediate sequences stay mask-free and can be edited repeatedly.
At generation time the model sees a sequence with no masks and decides whether to replace each token, which naturally supports error correction, since any token — not just masked ones — can be revised at any step. While masked diffusion models typically train faster (achieving better perplexities), uniform diffusion language models (UDLMs) facilitate faster sampling and better controllability , as we will discuss below. For instance, the open source Gemma diffusion model is a UDLM.
Uniform-state diffusion in action within Gemma Diffusion. Generation starts from an initial noisy "canvas" of random tokens ($t_0$) and denoises to a clean sequence ($t_7$). At each step, we predict the clean sequence and renoise part of the tokens to a new random token, letting the model revise and correct any position throughout sampling. Figure credit: M. Grootendorst.
Like MDLM, UDLM supports a simplified evidence lower bound objective that improves training . Earlier, more general frameworks such as D3PM also studied uniform and other structured noise processes, and methods such as generalized interpolating discrete diffusion (GIDD) combine masking and uniform noise into a single process that can recover either as a special case .
Accelerating Diffusion Sampling via Distillation
Because diffusion can generate or refine multiple tokens per step, it can be 5–10× faster than autoregressive generation. However, sampling many tokens at once introduces inconsistencies (e.g., two tokens that disagree in grammatical number, as in the remasking example above); if the underlying noise process cannot correct these errors, they accumulate. This is why most fast diffusion models today rely on error-correcting noise processes such as remasking or UDLM.
Sampling acceleration for these models is often inspired by progressive distillation : a model is trained on its own generations to skip a step, and this is repeated recursively, halving the number of sampling steps each round. In diffusion language models, this idea generalizes to techniques such as self-distillation through time and discrete consistency distillation .
Progressive distillation for faster sampling. A student model is trained to reproduce two of the teacher's denoising steps in one, and the procedure is applied recursively — halving the number of sampling steps each round while approximating the same mapping from noise to data. Figure credit: Lily Weng.
These can be interpreted as a form of on-policy training: standard training is off-policy, since the reverse process is trained on samples from the forward process rather than the samples it will actually see from itself at generation time. Training on the model's own samples in a post-processing step — too expensive to do during primary training — is what enables these methods to improve sampling speed.
Diffusion Enables Controllable Generation
Diffusion models excel at controllable generation: producing a sample $x$ that also satisfies a target property $y$, such as consistency with a prompt if $x$ is an image or binding affinity to a target site if $x$ is a molecule. Because they refine globally rather than committing to irreversible local edits, diffusion models navigate the sample space more effectively and produce better samples with the target property.
In practice, controllability manifests as a Pareto trade-off between naturalness (does the sample look like real data?) and property satisfaction (does it have the property we want?). For example, we could ask the model to produce molecules that look natural, but they might not have the binding affinity we seek. Conversely, we could optimize for binding affinity, but the outputs might not look like natural molecules and not be synthesizable. These two considerations induce a Pareto frontier on which diffusion improves over autoregression.
Controllable generation as a Pareto trade-off between sample naturalness (validity & novelty, x-axis) and how well a target property is satisfied (y-axis), traced out by varying the guidance strength. Discrete diffusion (yellow) attains a better frontier than autoregressive models (blue): at a given level of naturalness it reaches higher property satisfaction.
Algorithms for controllable generation
Diffusion models are especially effective at controllable generation via techniques such as classifier-based guidance (CBG) and classifier-free guidance (CFG). For example, in CBG, if we have a predictor model $p(y \mid x)$ of the target property $y$ given a sample $x$, we can use this model at each step to guide the generation process and provably yield a sample from the conditional distribution $p(x \mid y) \propto p(y \mid x)\, p(x)$.
Left: Why diffusion is well suited to guidance. Autoregressive models commit to "local", left-to-right predictions, whereas diffusion makes "global" refinements to the whole sequence that can be refined and error-corrected over multiple steps. Right: Classifier-based guidance for Gaussian diffusion. A classifier that predicts a property of interest $y$ can be used to steer each denoising step toward maintaining that desired property.
Both techniques extend naturally to MDLM and UDLM , and are especially effective when combined with UDLM or with MDLM plus remasking, since both support revising tokens repeatedly as guidance is applied.
Discrete classifier-based guidance (D-CBG): the math
By Bayes' rule, any conditional reverse process decomposes as follows:
$$ \underbrace{\log p(z_s \mid z_t, y)}_{\text{conditional distribution}} = \underbrace{\log p(y \mid z_t, z_s)}_{\text{predictive term}} + \underbrace{\log p(z_s \mid z_t)}_{\text{unconditional term}} + c, $$
where $c$ is a log-normalization constant. Now suppose we have a classifier $p(y \mid z_t)$ of the property $y$ from a noisy sequence $z_t$, as well as an unconditional diffusion model. We can plug them into the right hand side of the above equation to construct a conditional model from these two individual components.
$$ \underbrace{\log p^{(\gamma)}(z_s \mid z_t, y)}_{\text{new unnormalized distribution}} = \gamma\, \underbrace{\log p(y \mid z_s, z_t)}_{\text{guidance term}} + \underbrace{\log p(z_s \mid z_t)}_{\text{original diffusion model}}, $$
where $\gamma > 0$ is a guidance strength parameter that trades off the property against the model's own preferences. For a single token, the left-hand-side distribution is easy to normalize: we simply sum over the $N$ possible values of that token in the vocabulary. Extending this to a full sequence $z_t^{(1:L)}$ of length $L$ requires additional normalization techniques .
Discrete classifier-free guidance (D-CFG): the math
Instead of training a separate classifier, suppose we have a conditional model $p(z_s \mid z_t, y)$ and an unconditional model $p(z_s \mid z_t)$. Recall the CBG factorization from above,
$$ \log p^{(\gamma)}(z_s \mid z_t, y) = \gamma \cdot \underbrace{\log p(y \mid z_t, z_s)}_{\text{apply Bayes' rule}} + \log p(z_s \mid z_t) + c, $$
and apply Bayes' rule to the classifier term,
$$ \log p(y \mid z_t, z_s) = \log p(z_s \mid z_t, y) - \log p(z_s \mid z_t) + c. $$
Substituting this in and absorbing the $z_s$-independent factors into the normalization constant leaves a simple combination of the conditional and unconditional reverse models:
$$ \underbrace{\log p^{(\gamma)}(z_s \mid z_t, y)}_{\text{new unnormalized distribution}} = \gamma\, \underbrace{\log p(z_s \mid z_t, y)}_{\text{conditional model}} + (1 - \gamma)\, \underbrace{\log p(z_s \mid z_t)}_{\text{unconditional model}}, $$
where $\gamma > 0$ is a guidance strength parameter. This form is convenient because it avoids a separate classifier and, as before, is tractable to normalize: for each token we only sum over its $N$ possible values. In practice, the conditional $p(z_s \mid z_t, y)$ and unconditional $p(z_s \mid z_t)$ distributions are parameterized by the same model, trained by randomly dropping the conditioning signal $y$ so that the network learns both modes at once .
Post-Training Diffusion Language Models
Diffusion language models can also be post-trained with reinforcement learning to improve reasoning, following the same broad recipe that has driven recent gains in autoregressive LLMs. The main complication is that RL algorithms like policy gradient methods need the likelihood of a sampled trajectory, which is easy for autoregressive models (a simple product of next-token probabilities) but expensive for masked diffusion models, whose training objective averages over all masking orders. d1 introduces diffu-GRPO, a critic-free policy-gradient algorithm that estimates these trajectory log-probabilities with a mean-field approximation, combined with masked supervised fine-tuning to distill reasoning behavior from existing datasets . d2 improves on this by deriving more accurate likelihood estimators — an exact one-pass estimator for models that support any-order decoding, and an approximate estimator with a tunable compute–accuracy trade-off otherwise — yielding a new state of the art on logical and math reasoning benchmarks without relying on supervised fine-tuning at all .
Post-training with RL is also central to biological applications of diffusion language models, where the desired reward is often an experimentally measured property (e.g., binding affinity or gene expression) rather than a verifiable answer. Methods such as DRAKES back-propagate this kind of reward through the sampling trajectory of a discrete diffusion model to fine-tune it directly for DNA and protein design , and a broader tutorial surveys the space of RL-based fine-tuning algorithms for diffusion models more generally .
Diffusion Large Language Models Today
Over the last few years, diffusion language models have been scaled up to billions of parameters, showing improvements over autoregressive models at scale. We highlight work in science and language, and we describe how these models are built by combining the basic building blocks introduced above.
Biological and Scientific Domains
One of the first success areas of diffusion language models has been scientific applications, particularly biological sequences. There are two reasons for this:
- Biological sequences are less likely to exhibit the left-to-right biases baked into autoregressive models
- Scientific applications often benefit from controllable generation, which is a natural strength of diffusion
Protein Sequence Modeling
Perhaps the first large scale application of discrete diffusion has been in protein modeling. The recent ESM3 model implements MDLM at up to 100B parameters and is trained on massive amino-acid databases. Both the training and sampling algorithms of ESM3 match the masking diffusion approach described above. The authors report that this approach outperformed autoregressive baselines and set a new state of the art in protein generation.
ESM3 casts protein modeling as masked diffusion over three interleaved tracks: sequence, structure, and function. Tokens across all tracks are masked and reconstructed with a cross-entropy loss, letting the 100B-parameter model generate and condition across modalities (Hayes et al., Science 2025).
In terms of impact, the ESM models were among the first to apply large-scale sequence modeling to biological sequences. These models are widely used across proteomics for tasks such as variant effect prediction, protein folding, and protein generation.
Nucleotide Sequence Modeling
Proteins are the building blocks of life, but their activity is heavily regulated by non-coding genomic sequences that fall outside the scope of protein models like ESM3. DNA language models generalize the approach of ESM3 to both coding and non-coding genomic sequences. In a collaboration between our research group, InstaDeep, and BioNTech, we trained the Nucleotide Transformer v3 (NT-v3) family of models, which scales MDLM to billions of parameters and over a trillion tokens of DNA. These models take as input multi-track data that includes gene-expression levels alongside raw sequence. At inference time, these tracks support discrete classifier-free guidance with remasking, enabling generation of DNA sequences with target properties.
Nucleotide Transformer v3 (NT-v3) applies masked diffusion to DNA: over a 4 kb sequence it iteratively unmasks tokens sampled from $P(\text{DNA} \mid \text{condition})$, here generating a masked enhancer conditioned on a fixed promoter.
As a demonstration of the conditional generative capabilities of these models, we used NT-v3 to produce regulatory DNA sequences to enhance or repress the expression of specific genes. We generated a range of sequences by varying the guidance strength parameter in discrete CFG and we tested their capabilities in the wetlab. These generated sequences modulate gene expression better than previous baselines, and demonstrate the effectiveness of guidance.
Across target expression bins, enhancers generated by guided NT-v3 (blue) achieve higher measured activity than native enhancers (gray) at the "High" and "Highest" levels, evidence that diffusion-designed sequences can modulate gene expression as intended using guidance-based methods.
Diffusion-Based Large Language Models
While discrete diffusion models have found early success in modeling biological sequences, the last 18 months have seen the rapid emergence of diffusion large language models.
LLaDA
LLaDA scaled MDLM to 8B parameters, emulating the LLaMA recipe, with an MDLM backbone, block diffusion at sampling time, and compatibility with remasking and post-training. It is open-weights and reports favorable scaling versus autoregressive models .
LLaDA, an 8B-parameter open-weights diffusion LLM, is competitive with similarly-sized autoregressive models (LLaMA2-7B, LLaMA3-8B) across general, math, and code benchmarks (left), and reports autoregressive-like scaling of accuracy with training FLOPs on GSM8K and MMLU (right).
The LLaDA models are open-weights and serve as the foundation for a large body of academic research.
Mercury
Mercury is the first commercial diffusion LLM, announced in 2025. Its differentiator is speed: leveraging parallel generation, it exceeds 1,000 tok/sec/user on standard GPUs while matching the quality of its class. Mercury 2 rivals speed-optimized frontier models (Claude Haiku, Gemini Flash-Lite/Flash) at 5–10× the speed .
Intelligence (Artificial Analysis agentic index, y-axis) versus output speed (tokens/second, x-axis) for comparable models. Mercury 2, a commercial diffusion LLM, sits far to the right at roughly 1,200 tokens/second — several times faster than autoregressive models of comparable quality (including GPT-5 mini and Claude 4.5 Haiku). Figure credit: Inception AI.
This level of speed was previously only achievable using specialized chips (e.g., Groq) purpose-built to accelerate autoregressive inference. In contrast, a diffusion model achieves comparable speeds on standard GPUs by modifying the algorithm to better fit the underlying hardware rather than the other way around.
Speed of a diffusion model (Mercury Mini, early 2025) compared with autoregressive Llama-8B served across specialized inference stacks. Running on commodity GPUs, diffusion rivals the speeds dedicated AI-chip and speculative-decoding providers such as SambaNova and Groq, while also achieving higher intelligence. Figure credit: Inception AI.
Gemma Diffusion
Gemma Diffusion is a modern open-weights diffusion model from Google, widely supported in popular frameworks including Unsloth and Hugging Face . It combines a UDLM backbone, block diffusion for generating variable-length sequences, an encoder–decoder architecture, and accelerated sampling — shifting the generation bottleneck from memory bandwidth to compute by denoising an entire block ("canvas") of tokens in parallel on each forward pass, rather than emitting a single token at a time.
Nemotron Diffusion
Nemotron Diffusion is a family of open-weights diffusion models from NVIDIA trained with a joint autoregressive–diffusion objective, implementing MDLM and block diffusion. Recent models add an encoder-decoder architecture and scale to 35B parameters. The models report roughly 2-8× the throughput of comparable AR models while retaining up to 99% of their quality, and a single checkpoint can still fall back to plain autoregressive decoding.
Conclusion & Parting Thoughts
The field of diffusion language models has exploded over the past two years, with production-grade dLLM releases from multiple frontier labs. Diffusion offers potential advantages over autoregressive models in speed (up to 10× via parallel generation), controllability (iterative refinement for property-targeted generation), multi-modality (a single algorithmic approach across images and text), and inference-time scaling. Diffusion has not yet been scaled to the same parameters, compute, and data as autoregressive models, but experiments at up to 100B parameters show significant promise.
Is Diffusion A Path Towards More Intelligent Models? A Scaling-Law Perspective
We would like to conclude this article with an interesting question that the authors have often received when giving presentations on this work. It can be paraphrased as follows: can diffusion models eventually yield fundamental improvements over autoregressive models in terms of pure intelligence?
To answer this, we take a perspective grounded in scaling laws. The most important source of progress in model intelligence from 2019 to 2024 has surely been the scaling of pre-training in large language models. What made this scaling possible? The development of the transformer architecture, together with the wider availability of compute. But what made the transformer special? Before the transformer, the ubiquitous architecture for language models was based on RNNs. RNNs, however, never led to pre-training scaling because they did not scale — specifically, because they were fundamentally sequential algorithms that could not take full advantage of GPUs, which are highly parallel computers. It took the transformer to introduce a fully parallel training algorithm that scaled to large GPUs and unlocked the LLM revolution.
Since 2024, the gains from pre-training have been plateauing, and most of the intelligence gains in models have instead come from scaling post-training and inference-time compute. Yet both post-training and inference are bottlenecked by the ability to generate quickly, which today is done via a sequential algorithm. If inference could be made parallel, just as training was, the result could accelerate gains in intelligence as dramatic as those we saw in pre-training.
We view diffusion as the approach that could make inference fully parallel and unlock these gains. In a nutshell, diffusion may be to inference-time and post-training scaling laws what the transformer was to RNNs for pre-training scaling laws. By being able to spend more FLOPs per second, diffusion can unlock better hardware utilization, which in turn opens up our ability to scale.
It is still early to say how quickly diffusion will improve, but it suffices to say that, in our opinion, the prize is large.
Instagram Engineering
Shopify Engineering