Diffusion based approaches to long form text generation suffer from prohibitive computational cost and memory overhead as sequence length increases. We introduce SA-DiffuSeq, a diffusion framework that integrates sparse attention to fundamentally improve scalability for long document modeling. By selectively allocating attention within the diffusion process, SA-DiffuSeq significantly reduces computational complexity while maintaining semantic coherence and generation quality. A key component of our method is a soft absorbing state tailored to sparse attention dynamics, which stabilizes diffusion trajectories and accelerates sequence reconstruction. This design improves sampling efficiency and enhances precision in long range dependency modeling. Extensive experiments demonstrate that SA-DiffuSeq consistently surpasses state of the art diffusion baselines in both training efficiency and sampling speed, with especially strong gains on extended sequences. These properties make SA-DiffuSeq well suited for demanding long form applications such as scientific writing, large scale code generation, and multi turn long context dialogue. Overall, our results indicate that incorporating structured sparsity into diffusion models is a promising direction for efficient and expressive long text generation.
The rapid growth of long-form textual content-ranging from technical reports and scientific articles to large-scale code repositories and extended conversational logs-has exposed fundamental limitations in existing text generation paradigms (Beltagy, Peters, and Cohan 2020;Wang, Hamza, and Florian 2017;Zhang et al. 2025i,g;Fan et al. 2025a). Unlike short-context generation, long-text modeling requires maintaining global semantic consistency while remaining computationally feasible as sequence length increases (Li et al. 2024;Zhang et al. 2025h;Cai et al. 2025). This tension between expressiveness and scalability has become a central challenge in modern natural language generation.
Transformer-based architectures Vaswani et al. (2017); Zhang et al. (2025f) have been instrumental in advancing natural language processing due to their powerful selfattention mechanism. However, full self-attention incurs quadratic computational and memory complexity with respect to sequence length, making it increasingly impractical for long-document generation. When generation spans thousands of tokens, the cost of maintaining dense pairwise interactions quickly dominates both training and inference, resulting in substantial inefficiencies and limiting real-world applicability (Achiam et al. 2023;Zhang et al. 2025d,e).
To address these bottlenecks, sparse attention mechanisms have been proposed as a practical compromise. Models such as Longformer (Beltagy, Peters, and Cohan 2020) restrict attention patterns to reduce complexity and enable longer contexts to be processed. While effective in lowering computational overhead, sparse attention often introduces new challenges: aggressively limiting attention can weaken the model’s ability to capture global semantic structure, leading to degraded coherence and reduced generation quality as document length grows.
In parallel, diffusion-based models have recently been extended to text generation, offering a fundamentally different modeling perspective. DiffuSeq (Gong et al. 2023) formulates sequence generation as an iterative denoising process, which provides robustness and controllability through gradual refinement. Despite these advantages, diffusion-based text models face their own scalability issues. The iterative nature of denoising leads to slow convergence and high computational cost (Zhang et al. 2025b;Austin et al. 2021;Chen et al. 2023;Zhang et al. 2025a), particularly when attention operations are applied repeatedly over long sequences. As a result, naively scaling diffusion models to long documents remains challenging.
These difficulties are further compounded by limitations observed in large language models (LLMs). Most LLMs are trained on text segments capped at approximately 8,000 tokens and exhibit significant performance degradation when applied to substantially longer inputs (Li et al. 2024). This degradation highlights a broader issue: existing architectures are not inherently designed to allocate computation adaptively across extended contexts, but instead rely on uniform processing strategies that scale poorly with sequence length. Taken together, these observations reveal several unresolved gaps. First, current approaches struggle to jointly optimize efficiency and generation quality for long texts, often improving one at the expense of the other. Second, most models lack mechanisms for dynamically allocating computational arXiv:2512.20724v1 [cs.CL] 23 Dec 2025 resources based on the varying semantic complexity of different text segments. Finally, existing designs provide limited support for preserving long-range semantic dependencies under constrained attention and computational budgets.
To address these challenges, we introduce SA-DiffuSeq, a novel long-text generation framework that rethinks diffusion-based modeling through adaptive computation and structured sparsity. SA-DiffuSeq integrates the Mixture of Experts (MoE) paradigm into the DiffuSeq architecture and augments it with a diffusion-aware sparse attention mechanism, enabling scalable and high-quality generation for extended sequences. At a high level, SA-DiffuSeq dynamically routes different segments of a document to specialized experts, allowing computational resources to scale with local semantic complexity rather than sequence length alone. In parallel, a customized sparse attention mechanism tailored to diffusion-based generation substantially reduces attention computation while preserving access to global contextual information. To further stabilize and accelerate the denoising process, we introduce soft absorption states into the diffusion trajectory, improving reconstruction accuracy and convergence speed. Finally, SA-DiffuSeq incorporates advanced sampling techniques such as DPM-solver++ (Lu et al. 2022), which significantly reduce the number of diffusion steps required during generation without compromising output quality.
These design choices translate into consistent empirical improveme
This content is AI-processed based on open access ArXiv data.