Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models

Large Language Models (LLMs) have achieved state-of-the-art performance on a broad range of Natural Language Processing (NLP) tasks, including document processing and code generation. Autoregressive Language Models (ARMs), which generate tokens sequentially conditioned on all previous tokens, have been the predominant paradigm for LLMs. While these models have achieved high accuracy across a range of downstream tasks, they exhibit low arithmetic intensity due to the inherent sequential dependency in next-token prediction. Recently, Diffusion Language Models (DLMs) have emerged as a promising alternative architecture. DLMs generate output tokens in parallel, mitigating the limitations of sequential decoding. However, the performance implications of DLMs relative to commonly deployed ARMs are not fully understood. In this work, we present a comprehensive study of the performance characteristics of ARMs and DLMs, combining theoretical analysis with empirical profiling to characterize the trade-offs between these approaches. We show that although DLMs can achieve higher arithmetic intensity than ARMs by leveraging parallelism across token positions, they fail to scale effectively with longer contexts. We then explore block-wise decoding for DLMs, which decouples arithmetic intensity from sequence length and enables better scaling to long contexts (similar to ARMs). We also examine batched inference and find that ARMs exhibit superior throughput as they benefit more from parallelism across sequences in the batch. Finally, we highlight opportunities for accelerating DLM inference, emphasizing that reducing the number of sampling steps is key for open-source DLMs to achieve lower latency relative to ARMs.

💡 Research Summary

This paper presents a systematic performance comparison between Autoregressive Language Models (ARMs) and Diffusion Language Models (DLMs), focusing on inference latency, throughput, and arithmetic intensity (AI). The authors evaluate two representative 8‑billion‑parameter models—LLaMA‑3‑8B‑Instruct (ARM) and LLaDA‑8B‑Instruct (DLM)—on NVIDIA RTX A6000 and A100 GPUs using FP16 precision.

Theoretical framework
The authors first derive asymptotic formulas for FLOPs, memory operations (MOPs), and AI for four inference scenarios: ARM pre‑fill, ARM decode (with KV caching), naive DLM (no caching), and block‑wise DLM (partial KV reuse). They show that ARM pre‑fill is compute‑bound when the prompt length Lp is small relative to the hidden dimension d, but becomes memory‑bound for long prompts. ARM decode, thanks to KV caching, has AI that collapses to O(1) for large Lp, making it memory‑bandwidth limited. Naive DLM processes the full sequence at every diffusion step, yielding AI≈O(B·L) (or O(L) without batching), which turns compute‑bound for long sequences. Block‑wise DLM updates only a block of size G in parallel; its AI depends on G rather than the total length, decoupling performance from L.

Empirical profiling
The authors measure end‑to‑end latency as a function of generated token count Lg for short (Lp = 128) and long (Lp = 2 k) prompts. With batch size B = 1 and diffusion steps K = Lg (the common setting in current open‑source DLMs), DLMs are faster than ARMs for very short prompts but quickly become slower as Lg grows, especially with long prompts where each diffusion step recomputes the entire context. Roofline plots confirm that ARM pre‑fill sits in the compute‑bound region, ARM decode on the bandwidth roof, naive DLM moves from bandwidth‑bound (short sequences) to compute‑bound (long sequences), and block‑wise DLM remains near the bandwidth roof regardless of L.

Block‑wise decoding
Introducing block‑wise decoding with KV caching (prompt and inactive blocks cached, active block of size G updated in parallel) reduces latency by 2–3× compared with naive DLM. AI becomes a function of G only, making performance invariant to generation length. However, when K scales with Lg, the model still denoises roughly one token per step, limiting overall speed. The authors argue that reducing the number of diffusion steps—through multi‑token denoising or more efficient schedules—is essential for DLMs to compete with ARMs.

Batch scaling
Throughput (tokens / second) is evaluated for batch sizes up to 16 on a single A100. ARM throughput scales well up to B = 16 for short prompts and degrades gracefully for long prompts, thanks to lightweight, memory‑bound decode that benefits from batch parallelism. Block‑wise DLM throughput plateaus earlier (around B = 8) and suffers from higher compute load per block and from the need to keep the full KV cache (size B·L) in memory, causing out‑of‑memory errors for larger batches with long prompts. Naive DLM shows no batch benefit because its full‑sequence attention is always compute‑bound.

Discussion and future directions
The paper identifies three primary levers to close the performance gap: (1) decrease diffusion steps K, possibly by designing denoisers that output multiple tokens per step; (2) develop KV‑cache‑friendly attention mechanisms or hybrid architectures that retain bidirectional context while allowing reuse of previously computed keys/values; (3) create hardware accelerators optimized for the dense matrix operations and high bandwidth demands of DLMs. The authors also note that block‑wise decoding preserves task accuracy, suggesting that efficiency gains need not sacrifice quality.

Conclusion
While DLMs offer higher arithmetic intensity through parallel token updates, their current implementations suffer from poor scaling with long contexts and limited batch efficiency due to the absence of KV caching. Block‑wise decoding mitigates these issues by fixing AI to the block size, but ARMs still dominate in throughput and memory efficiency, especially in batched serving scenarios. The paper provides a clear roadmap—reduce diffusion steps, improve cache reuse, and tailor hardware—to make open‑source diffusion language models a viable alternative to autoregressive models.