Efficient Long-Sequence Diffusion Modeling for Symbolic Music Generation

Symbolic music generation is a challenging task in multimedia generation, involving long sequences with hierarchical temporal structures, long-range dependencies, and fine-grained local details. Though recent diffusion-based models produce high quality generations, they tend to suffer from high training and inference costs with long symbolic sequences due to iterative denoising and sequence-length-related costs. To deal with such problem, we put forth a diffusing strategy named SMDIM to combine efficient global structure construction and light local refinement. SMDIM uses structured state space models to capture long range musical context at near linear cost, and selectively refines local musical details via a hybrid refinement scheme. Experiments performed on a wide range of symbolic music datasets which encompass various Western classical music, popular music and traditional folk music show that the SMDIM model outperforms the other state-of-the-art approaches on both the generation quality and the computational efficiency, and it has robust generalization to underexplored musical styles. These results show that SMDIM offers a principled solution for long-sequence symbolic music generation, including associated attributes that accompany the sequences. We provide a project webpage with audio examples and supplementary materials at https://3328702107.github.io/smdim-music/.

💡 Research Summary

The paper introduces SMDIM (Structured Music Diffusion with Integrated Modeling), a novel framework designed to generate long‑sequence symbolic music efficiently while preserving both global musical structure and fine‑grained local details. Symbolic music, represented as discrete token sequences (pitch, duration, timing, etc.), poses two major challenges for diffusion‑based generative models: (1) the quadratic memory and compute cost of self‑attention when handling thousands of tokens, and (2) the need to maintain intricate local rhythmic and melodic relationships throughout the iterative denoising process.

Core Contributions

1. Linear‑time Global Modeling via Structured State‑Space Models (SSM).
   SMDIM adopts the Mamba‑style SSM, which models the evolution of a hidden state through linear recurrences ( xₜ = A xₜ₋₁ + ηₜ ) and projects it to observable token embeddings ( yₜ = C xₜ + εₜ ). Because the recurrence can be computed with O(L) complexity, the model captures long‑range dependencies such as thematic repetitions, key changes, and overall form without the O(L²) burden of full‑attention Transformers.

2. Hybrid MF‑A Block for Global‑Local Fusion.
   Each MF‑A block combines three sub‑components: (i) multi‑head self‑attention for short‑range token interactions, (ii) a Mamba SSM module for efficient long‑range context, and (iii) a feed‑forward network for non‑linear transformation. The block also implements a dynamic masking schedule that determines which positions are refined at each diffusion step, allowing the network to focus computation on the most ambiguous tokens while leaving already‑stable regions untouched.

3. Discrete Denoising Diffusion with Absorbing State (D3PM).
   The forward diffusion process progressively replaces tokens with a special absorbing (mask) state according to a predefined βₜ schedule. The reverse process is trained to directly predict the original token at each masked position (x‑prediction), rather than estimating noise. This formulation aligns naturally with the categorical nature of symbolic music and simplifies training on large datasets.

4. Hierarchical Architecture.
   Input tokens are first embedded, then down‑sampled by a 1‑D convolution to a shorter, richer representation. The compressed sequence passes through several MF‑A blocks, after which a transposed convolution upsamples back to the original length, followed by a linear head that outputs token probabilities. This hierarchy reduces the effective sequence length for the expensive SSM layers while still allowing the final reconstruction to retain the original resolution.

Experimental Validation
The authors evaluate SMDIM on four public datasets covering Western classical piano (MAESTRO), pop music (POP909), a newly curated Chinese folk music set (FolkDB), and a subset of the Lakh MIDI corpus. Sequence lengths range from 512 to 2048 tokens. Baselines include Transformer‑based diffusion models (MusicDiff), hierarchical diffusion (H‑Diff), and non‑diffusion Transformers (MuseNet).

• Quality Metrics: SMDIM achieves higher Long‑Term Structure Score (LTSS) and Groove‑Similarity, improving by 0.12–0.18 points over the best baseline. Human listening tests show a 15–22 % increase in perceived coherence and expressiveness.
• Efficiency: On an NVIDIA A100 GPU, training memory consumption drops by 30–45 % and inference speed roughly doubles. Notably, on the longest FolkDB sequences (≥1024 tokens) baseline models encounter out‑of‑memory errors, whereas SMDIM runs without issue.
• Ablation Studies: Removing the Mamba component degrades long‑range coherence, while omitting self‑attention harms local rhythmic accuracy. Varying the masking ratio reveals that a moderate 0.5 schedule balances quality and speed best.

Limitations and Future Work
SMDIM currently focuses on unconditional music generation; extending it to conditional settings (e.g., text‑to‑music, style transfer) will require additional control mechanisms. The performance of the Mamba module is sensitive to its state dimension and depth, suggesting a need for automated hyper‑parameter search. Moreover, the subjective evaluation relies on a limited pool of listeners, so broader cross‑cultural studies are warranted.

Conclusion
By integrating a linear‑time state‑space backbone with a hybrid attention‑SSM refinement block, SMDIM offers a principled solution to the long‑sequence symbolic music generation problem. It delivers state‑of‑the‑art generation quality while dramatically reducing computational overhead, especially for very long or under‑represented musical styles. The work opens avenues for scalable, high‑fidelity music generation and sets a new benchmark for future research in diffusion‑based symbolic audio modeling.