DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding

Autoregressive (AR) large audio language models (LALMs) such as Qwen-2.5-Omni have achieved strong performance on audio understanding and interaction, but scaling them remains costly in data and computation, and strictly sequential decoding limits inference efficiency. Diffusion large language models (dLLMs) have recently been shown to make effective use of limited training data, and prior work on DIFFA indicates that replacing an AR backbone with a diffusion counterpart can substantially improve audio understanding under matched settings, albeit at a proof-of-concept scale without large-scale instruction tuning, preference alignment, or practical decoding schemes. We introduce DIFFA-2, a practical diffusion-based LALM for general audio understanding. DIFFA-2 upgrades the speech encoder, employs dual semantic and acoustic adapters, and is trained with a four-stage curriculum that combines semantic and acoustic alignment, large-scale supervised fine-tuning, and variance-reduced preference optimization, using only fully open-source corpora. Experiments on MMSU, MMAU, and MMAR show that DIFFA-2 consistently improves over DIFFA and is competitive to strong AR LALMs under practical training budgets, supporting diffusion-based modeling is a viable backbone for large-scale audio understanding. Our code is available at https://github.com/NKU-HLT/DIFFA.git.

💡 Research Summary

DIFFA‑2 presents a practical diffusion‑based large audio language model (dLLM) that challenges the dominance of autoregressive (AR) backbones in large audio language modeling. Building on the proof‑of‑concept DIFFA work, the authors introduce several key innovations to make diffusion modeling competitive under realistic data, compute, and latency constraints.

The architecture consists of a frozen Whisper‑Large‑V3 encoder, a dual‑adapter interface, and a diffusion LLM backbone (LLaDA). The semantic adapter reduces the temporal resolution of encoder outputs (50 Hz → 12.5 Hz) via a two‑layer convolutional subsampling followed by a linear projection, aligning audio features with textual tokens. The acoustic adapter is a two‑layer Q‑former with 64 learnable queries that attend to intermediate encoder states, capturing prosodic, emotional, environmental, and musical cues that are not directly expressed in text. This dual‑stream design supplies the diffusion backbone with both content‑oriented and compact acoustic summaries.

Training follows a four‑stage progressive curriculum:

1. Semantic alignment (Stage 1) – Only the semantic adapter is trained on large ASR corpora (LibriSpeech, GigaSpeech) using mask‑prediction loss, aligning it with the textual semantic space while keeping the diffusion backbone frozen.

2. Joint semantic‑acoustic alignment (Stage 2) – Both adapters are trained on a synthetic supervised fine‑tuning (SFT) dataset that includes caption‑grounded QA, TTS‑converted QA, multiple‑choice QA, and an ASR subset. The same LLaDA mask‑reconstruction objective is applied, encouraging the model to learn fine‑grained acoustic cues across speech, sound, and music.

3. Backbone fine‑tuning with LoRA (Stage 3) – Low‑Rank Adaptation (LoRA) is introduced to update the diffusion backbone while still updating the adapters. LoRA modifies only ~1 % of parameters, providing sufficient capacity for audio understanding without catastrophic forgetting.

4. Variance‑Reduced Preference Optimization (Stage 4) – Preference triplets (preferred answer vs. a subtly wrong “rejected” answer) are generated using a strong LLM. The authors adopt VRPO, a DPO‑style objective that reduces variance in Monte‑Carlo ELBO estimates by sharing masking patterns between the policy and reference models (antithetic sampling). This stabilizes preference learning even for long, acoustically rich sequences.

During inference, the model initializes the response as fully masked and iteratively denoises over a fixed number of steps. To achieve practical latency, DIFFA‑2 employs factor‑based parallel decoding (Wu et al., 2025): tokens are processed in left‑to‑right blocks, but within each block predictions are made in parallel and low‑confidence tokens are re‑masked for subsequent refinement. This semi‑autoregressive strategy reduces the number of diffusion steps by 2–3× while preserving generation quality.

Experiments on three multimodal audio benchmarks—MMSU (multimodal speech understanding), MMAU (multimodal audio understanding), and MMAR (multimodal audio reasoning)—show that DIFFA‑2 (8 B parameters) consistently outperforms its predecessor DIFFA‑1 (by 4–6 percentage points) and reaches performance comparable to much larger AR models such as Qwen‑3‑Omni (30 B) and Qwen‑2.5‑Omni (7 B). Notably, DIFFA‑2 excels in the “paralinguistics” sub‑category, indicating that the acoustic adapter and VRPO alignment effectively capture prosody, emotion, and background sounds. The factor‑based parallel decoding variant (DIFFA‑2 w/ FPD) achieves similar scores with a substantial speedup, confirming the practicality of diffusion‑based inference.

In summary, DIFFA‑2 demonstrates that a diffusion backbone, when equipped with (1) dual semantic‑acoustic adapters, (2) a staged curriculum that leverages open‑source ASR, SFT, and preference data, (3) efficient LoRA fine‑tuning, and (4) parallel decoding, can match or exceed state‑of‑the‑art AR large audio language models under realistic training budgets. The authors release all code, data pipelines, and model checkpoints, paving the way for further research on diffusion‑based multimodal language models.