Addressing Gradient Misalignment in Data-Augmented Training for Robust Speech Deepfake Detection

In speech deepfake detection (SDD), data augmentation (DA) is commonly used to improve model generalization across varied speech conditions and spoofing attacks. However, during training, the backpropagated gradients from original and augmented inputs may misalign, which can result in conflicting parameter updates. These conflicts could hinder convergence and push the model toward suboptimal solutions, thereby reducing the benefits of DA. To investigate and address this issue, we design a dual-path data-augmented (DPDA) training framework with gradient alignment for SDD. In our framework, each training utterance is processed through two input paths: one using the original speech and the other with its augmented version. This design allows us to compare and align their backpropagated gradient directions to reduce optimization conflicts. Our analysis shows that approximately 25% of training iterations exhibit gradient conflicts between the original inputs and their augmented counterparts when using RawBoost augmentation. By resolving these conflicts with gradient alignment, our method accelerates convergence by reducing the number of training epochs and achieves up to an 18.69% relative reduction in Equal Error Rate on the In-the-Wild dataset compared to the baseline.

💡 Research Summary

This paper investigates a previously overlooked problem in speech deepfake detection (SDD) when data augmentation (DA) is employed: the gradients back‑propagated from the original utterance and its augmented counterpart often point in conflicting directions, which can impede convergence and degrade performance. The authors first quantify this phenomenon, showing that roughly 25 % of training iterations exhibit gradient conflicts (negative inner product) when using the RawBoost augmentation. Visualizing the loss landscapes reveals that the original input typically yields a smooth surface, whereas the augmented input produces a more rugged surface with multiple sharp valleys, leading to divergent descent directions.

To address this, the authors propose a Dual‑Path Data‑Augmented (DPDA) training framework. Each training sample is processed simultaneously through two parallel paths: one receiving the raw waveform and the other receiving an augmented version generated by a signal‑level DA method. Separate losses L(x) and L(˜x) and their gradients gₓ and g_{˜x} are computed. The core contribution lies in aligning these two gradients before the final parameter update. Three established gradient‑alignment techniques from multi‑task learning are evaluated: PCGrad, GradVac, and CAGrad. PCGrad detects conflict when ⟨gₓ, g_{˜x}⟩ < 0 and projects each gradient onto the normal plane of the other; GradVac maintains a dynamic target cosine similarity and linearly combines the gradients when the similarity falls below this target; CAGrad solves a constrained optimization problem that seeks an update close to the naïve combined gradient while maximizing the minimum inner‑product with both gₓ and g_{˜x}.

Experiments are conducted on the ASVspoof2019 Logical Access training set, with three challenging test sets: ASVspoof2021 DF, In‑the‑Wild (ITW), and Fake‑or‑Real (FoR). Three state‑of‑the‑art SDD backbones—XLSR‑AASIST, XLSR‑Conformer‑TCM, and XLSR‑Mamba—are evaluated. RawBoost (configuration 4) serves as the primary augmentation, while additional experiments combine MUSAN + RIR with or without RawBoost. Because DPDA doubles memory consumption, batch size is halved (from 20 to 10) to fit typical GPUs.

Results demonstrate consistent gains across all models and datasets. For the XLSR‑Conformer‑TCM backbone, DPDA alone slightly worsens EER on some sets, but adding PCGrad reduces EER from 7.97 % to 6.48 % on ITW and from 5.31 % to 4.47 % on the In‑the‑Wild set—a relative reduction of up to 18.69 % compared with the baseline single‑path training. Similar improvements are observed for XLSR‑AASIST (ITW EER = 5.42 %, FoR = 3.04 %) and XLSR‑Mamba (21DF = 1.74 %). Gradient‑conflict analysis shows that PCGrad cuts the proportion of conflicting iterations by roughly half and yields a smoother, faster‑decreasing validation loss curve, confirming enhanced training stability.

The paper’s contributions are threefold: (1) it provides the first systematic quantification and visualization of gradient conflicts induced by data augmentation in SDD; (2) it introduces a simple yet effective DPDA framework combined with gradient‑alignment methods to reconcile those conflicts; (3) it validates the approach across multiple architectures, augmentation strategies, and real‑world test sets, achieving state‑of‑the‑art performance. Limitations include the increased computational cost due to dual‑path processing and the empirical nature of the alignment choice—PCGrad performed best but the underlying reason remains unclear. Future work may explore more nuanced conflict‑detection criteria, memory‑efficient implementations, and extensions to other modalities such as video deepfake detection.