LLaCO: Continuous Diffusion for Large Language Models

TL;DR. We propose LLaCO-8B, a continuous diffusion language model that comprehensively outperforms existing continuous diffusion language models TESS-2 and DSL-LLaDA, achieving state-of-the-art performance among continuous diffusion language models.

Results

LLaCO comprehensively outperforms previous continuous diffusion baselines and rivals discrete diffusion models of comparable scale.

Model / configuration	ARC-E	ARC-C	MMLU	GPQA	BBH	GSM8K	MATH	HumanEval	MBPP
Discrete diffusion
LLaDA-8B	75.08	53.67	63.97	26.79	53.16	78.70	43.82	45.12	40.20
LLaDA-1.5-8B	79.46	55.20	64.19	27.01	52.72	79.38	44.26	46.34	43.20
Dream-7B	83.50	55.29	69.74	30.13	54.68	80.14	51.20	54.27	57.80
LLaCO-8B	76.68	54.78	69.89	31.25	51.30	80.74	62.12	48.17	46.20
Continuous diffusion
TESS-2-7B	66.58	38.05	47.98	30.13	37.05	23.96	4.70	1.22	8.60
DSL-LLaDA-8B	73.15	48.81	59.05	28.35	46.81	25.40	15.10	20.73	25.60
LLaCO-8B	76.68	54.78	69.89	31.25	51.30	80.74	62.12	48.17	46.20
\(P\) schedule	70.12	52.30	61.77	29.69	46.72	64.82	35.00	35.98	35.60
\(Q\) schedule	72.43	53.84	65.74	30.80	48.61	77.56	58.30	40.24	37.20
W/o mask-biased endpoint	74.83	54.01	67.28	30.58	48.16	79.23	60.02	30.49	36.60
W/o spherical noising	74.58	54.58	67.35	31.03	49.09	78.39	59.22	43.29	41.40

Overall performance comparison and ablations. Models are grouped by generation paradigm, with LLaCO-8B listed in both groups for reference. Indented rows under LLaCO-8B alter a single factor of the default configuration and are retrained. Bold values denote the best result within each primary group.

Abstract

Continuous diffusion language models suffer from a fundamental geometric misalignment between continuous trajectories and discrete vocabulary identities. This compresses identity decay into a narrow terminal window, forcing standard diffusion to misallocate optimization budgets. To resolve this, we establish the Competition Threshold Theory to model directional token competition, and propose Semantic Retention Scheduling for logarithmic rank decay. Based on these principles, we propose LLaCO, a framework equipped with an embedding directional adapter, timestep-conditioned modulation, and mask-biased spherical noising. Extensive evaluations demonstrate that LLaCO-8B comprehensively outperforms prior continuous baselines and rivals discrete models.

Method

LLaCO adapts a pretrained backbone to continuous representations and reallocates the denoising trajectory to align with the progression of identity decay.

**Figure 1. Overview of the LLaCO framework.** (a) The model architecture features an embedding directional adapter and timestep-conditioned modulation to produce token probabilities for cross-entropy training and clean-direction estimation. (b) Spherical noising proceeds along a geodesic trajectory toward a mask-biased endpoint. (c) Semantic Retention Scheduling leverages logarithmic rank retention to reallocate the optimization budget during both training and inference.

Model architecture. An embedding directional adapter and timestep-conditioned modulation adapt the pretrained backbone to continuous directional states, producing token probabilities for cross-entropy training and clean-direction estimation.
Spherical noising. Formulated as a geodesic interpolation toward a mask-biased endpoint, the spherical noising process eliminates norm decay while preserving accurate token competition.
Trajectory reallocation. Guided by Semantic Retention Scheduling, LLaCO reallocates the denoising trajectory proportionally to the logarithmic rank decay, dedicating maximum optimization budget to identity transitions during both training and inference.
Blockwise denoising. Text is generated block by block: each block starts from the mask-biased endpoint, is denoised along a uniform grid in semantic time via geodesic (SLERP) updates toward the predicted clean direction, and is then decoded and appended as clean context for subsequent blocks.

Theoretical Foundation

Standard continuous diffusion spends most of its optimization budget on noise levels where token identity remains intact. Two theoretical results explain why and guide how LLaCO allocates its denoising steps.

Animation: identity of the token language under a standard noise schedule versus LLaCO — Adding noise to the token “language”; the vertical axis shows the rank of the source token. Under a standard linear noise schedule (left), the entire rank decline is squeezed into the last ~9% of the steps. LLaCO allocates steps by \(Q_{\log}\), so the rank decline spans about 90% of the noising process (right).

Competition Threshold Theory (CTT)

Token competition must be evaluated directionally. Under Euclidean decoding, embedding norm decay alone causes spurious overtaking by smaller-norm candidates, even without noise, so the source token loses its top-1 rank far earlier than under cosine ranking (Fig. 2(a)).
High-dimensional overtaking requires extreme noise levels. On 4096-dimensional pretrained embeddings, source tokens largely hold their top-1 rank until the effective noise magnitude reaches \(\kappa = \text{noise magnitude} / \text{signal magnitude} \approx 11\), and even then only 10% have lost it (Fig. 2(d)).

Semantic Retention Scheduling (SRS)

Existing token identity metrics track the rank of the source token under noise: \(P\) is the probability that the source token remains top-1, and \(Q\) is the fraction of candidates yet to overtake the source.

Standard schedules spend over 90% of the optimization budget on stagnant retention phases. They reach the noise level at which identity decays only in the last few steps, concentrating identity decay into a narrow window near the noisy endpoint (Fig. 2(b, c)).
LLaCO allocates denoising steps by the logarithmic retention \(Q_{\log}\). \(P\) collapses entirely at the first overtake, ignoring all subsequent decay, while \(Q\) is diluted by the vast vocabulary tail. By measuring rank decay on a logarithmic scale, our logarithmic retention \(Q_{\log}\) captures the critical intermediate decay of leading semantic ranks (Fig. 2(b, c)).

Empirical Validation

Our theory also predicts these statistics directly: without parameter fitting, the theoretical predictions closely match measurements on pretrained embeddings (Fig. 3).

Identity decay along noising trajectories — **Figure 2. Identity decay along continuous noising trajectories.** In (a)–(c), the horizontal axis is trajectory time \(\gamma\), from the clean endpoint (\(\gamma = 0\)) to the noisy endpoint (\(\gamma = 1\)). (a) Top-1 rate of the source token under Euclidean decoding and cosine ranking along a linear trajectory. (b, c) \(P\), \(Q_{\log}\), and \(Q\) along the linear and spherical geodesic trajectories. (d) Fraction of source tokens that have lost their top-1 rank as the effective noise magnitude \(\kappa\) increases.

Predicted and measured competition statistics against the effective noise magnitude — **Figure 3. Predicted versus measured competition statistics on pretrained embeddings.** Top row: predicted and measured (a) expected rank \(\mathbb{E}[r]\), (b) \(P\), (c) \(Q_{\log}\), and (d) \(Q\) against the effective noise magnitude \(\kappa\). Bottom row: prediction errors against the measurements.

Citation

@article{jia2026llaco,
  title   = {LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling},
  author  = {Jia, Zijian and Zhang, Peixuan and Li, Genying and Wu, Di and Fu, Yujie and Li, Si and Shi, Boxin and Weng, Shuchen},
  journal = {arXiv preprint arXiv:2610.32604},
  year    = {2026}
}